Tips on how to debug EFI code (iPXE) from within KVM after ipxe.efi has crashed with #GP?

[Konrad Rzeszutek Wilk <konrad@kernel.org>]

Hey,

I am hitting an interesting issue with the latest versions of iPXE,
edk2, and GRUB wherein I am trying to boot GRUB (via iPXE), and then
load Xen, I get this:

!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID -
00000000 !!!!
ExceptionData - 0000000000000000
RIP  - 00000000BEC2949C, CS  - 0000000000000038, RFLAGS - 0000000000210216
RAX  - 0000000000000000, RCX - 00000000BFA2DC18, RDX - 0000000000000000
RBX  - 00000000BFA2DC60, RSP - 00000000BFEDDFD8, RBP - 0000000000000008
RSI  - 0000000000000000, RDI - 0000000000000001
R8   - 0000000000000001, R9  - 0000000000000030, R10 - 000000000009E000
R11  - 00000000BFF1A000, R12 - 00000000BFEF7260, R13 - 00000000BFA2DC18
R14  - 00000000BF40BD18, R15 - 00000000BEB4E8A4
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFE7D000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000BFE6B698 0000000000000047, LDTR - 0000000000000000
IDTR - 00000000BF620018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 00000000BFEDDC30
!!!! Find image 808610ed.efidrv (ImageBase=00000000BEC27000,
EntryPoint=00000000BEC2E089) !!!!


And now I am trying to figure out how to troubleshoot this.
(and yes I am thinking it was related to the Tivoli work-around, but
disabling that didn't help).

I was somehow hoping I could use 'crash' tool and run it with with the guest
memory and the iPXE code:

#crash ipxe/src/bin/ipxe.lkrn.tmp ./guest-memory
..

WARNING: machine type mismatch:
         crash utility: X86_64
         ipxe/src/bin/ipxe.lkrn.tmp: X86

crash: ipxe/src/bin/ipxe.lkrn.tmp: not a supported file format
..

<sad trombone>

So how do folks troubleshoot things like this?




A bit background information:

- The /guest-memory I got by having an -qmp argument and telneting in
it and doing:

{"execute":"dump-guest-memory","arguments":{"paging":
false,"protocol":"file:/guest-memory"}}

- The 80810ed.efidrv is built using:

(cd ipxe/src;make bin-x86_64-efi/808610d3.efi  CONFIG=qemu
DEBUG=iscsi:4,scsi:4 EMBED=/ipxe.config -j8)
if [ $? -ne 0 ]; then exit 1; fi
(cd ipxe/src;EfiRom -f 0x8086 -i 0x1528 --pci23 -ec
bin-x86_64-efi/808610d3.efidrv -o ../../808610d3.rom)
if [ $? -ne 0 ]; then exit 1; fi

The ipxe script is simple:

!ipxe

dhcp && echo * DHCP worked || goto dhcp_retry
echo * IP address: ${net0/ip} ; echo * Subnet mask: ${net0/netmask}
ifstat
set url http://10.0.1.2/
echo ${url}
imgfetch ${url}/grub.efi
imgstat
boot grub.efi


- And grub.efi was created with this:

GRUB_MODULES="boot chain configfile echo efinet eval ext2 fat font
gettext gfxterm gzio help linux loadenv lsefi normal part_gpt par
t_msdos read regexp search search_fs_file search_fs_uuid search_label
terminal terminfo test tftp time multiboot multiboot2 net slee
p efifwsetup  lsefisystab lspci lsefi lsacpi lsefimmap acpi fat eval
elf file linux linux16 pcidump http serial efi_gop"

grub-install/usr/bin/grub-mkimage \
                --config ${GRUB_CFG} \
                --directory=`pwd`/grub-install/usr/lib64/grub/x86_64-efi \
                --output=grub.efi \
                --format=x86_64-efi \
                --prefix "(http)/" \
                $GRUB_MODULES

# more grub.config
set timeout=5

net_bootp
net_ls_addr
net_ls_cards
net_ls_routes
lspci
set net_default_server=10.0.1.2
serial --speed=115200 --unit=0 --word=8
terminal_output  serial console
terminal_input  serial console
multiboot2 (http,10.0.1.2)/xen.gz dom0_mem=max:1G loglvl=all
guest_loglvl=all console=com1 com1=115200,8n1
module2 (http,10.0.1.2)/vmlinuz console=hvc0 debug initcall_debug
module2 (http,10.0.1.2)/initrd.img
boot