Tips on how to debug EFI code (iPXE) from within KVM after ipxe.efi has crashed with #GP?

Tips on how to debug EFI code (iPXE) from within KVM after ipxe.efi has crashed with #GP?

[Konrad Rzeszutek Wilk]

Hey,

I am hitting an interesting issue with the latest versions of iPXE,
edk2, and GRUB wherein I am trying to boot GRUB (via iPXE), and then
load Xen, I get this:

!!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID -
00000000 !!!!
ExceptionData - 0000000000000000
RIP  - 00000000BEC2949C, CS  - 0000000000000038, RFLAGS - 0000000000210216
RAX  - 0000000000000000, RCX - 00000000BFA2DC18, RDX - 0000000000000000
RBX  - 00000000BFA2DC60, RSP - 00000000BFEDDFD8, RBP - 0000000000000008
RSI  - 0000000000000000, RDI - 0000000000000001
R8   - 0000000000000001, R9  - 0000000000000030, R10 - 000000000009E000
R11  - 00000000BFF1A000, R12 - 00000000BFEF7260, R13 - 00000000BFA2DC18
R14  - 00000000BF40BD18, R15 - 00000000BEB4E8A4
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 0000000000000000, CR3 - 00000000BFE7D000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000BFE6B698 0000000000000047, LDTR - 0000000000000000
IDTR - 00000000BF620018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 00000000BFEDDC30
!!!! Find image 808610ed.efidrv (ImageBase=00000000BEC27000,
EntryPoint=00000000BEC2E089) !!!!


And now I am trying to figure out how to troubleshoot this.
(and yes I am thinking it was related to the Tivoli work-around, but
disabling that didn't help).

I was somehow hoping I could use 'crash' tool and run it with with the guest
memory and the iPXE code:

#crash ipxe/src/bin/ipxe.lkrn.tmp ./guest-memory
..

WARNING: machine type mismatch:
         crash utility: X86_64
         ipxe/src/bin/ipxe.lkrn.tmp: X86

crash: ipxe/src/bin/ipxe.lkrn.tmp: not a supported file format
..

<sad trombone>

So how do folks troubleshoot things like this?




A bit background information:

- The /guest-memory I got by having an -qmp argument and telneting in
it and doing:

{"execute":"dump-guest-memory","arguments":{"paging":
false,"protocol":"file:/guest-memory"}}

- The 80810ed.efidrv is built using:

(cd ipxe/src;make bin-x86_64-efi/808610d3.efi  CONFIG=qemu
DEBUG=iscsi:4,scsi:4 EMBED=/ipxe.config -j8)
if [ $? -ne 0 ]; then exit 1; fi
(cd ipxe/src;EfiRom -f 0x8086 -i 0x1528 --pci23 -ec
bin-x86_64-efi/808610d3.efidrv -o ../../808610d3.rom)
if [ $? -ne 0 ]; then exit 1; fi

The ipxe script is simple:

!ipxe

dhcp && echo * DHCP worked || goto dhcp_retry
echo * IP address: ${net0/ip} ; echo * Subnet mask: ${net0/netmask}
ifstat
set url http://10.0.1.2/
echo ${url}
imgfetch ${url}/grub.efi
imgstat
boot grub.efi


- And grub.efi was created with this:

GRUB_MODULES="boot chain configfile echo efinet eval ext2 fat font
gettext gfxterm gzio help linux loadenv lsefi normal part_gpt par
t_msdos read regexp search search_fs_file search_fs_uuid search_label
terminal terminfo test tftp time multiboot multiboot2 net slee
p efifwsetup  lsefisystab lspci lsefi lsacpi lsefimmap acpi fat eval
elf file linux linux16 pcidump http serial efi_gop"

grub-install/usr/bin/grub-mkimage \
                --config ${GRUB_CFG} \
                --directory=`pwd`/grub-install/usr/lib64/grub/x86_64-efi \
                --output=grub.efi \
                --format=x86_64-efi \
                --prefix "(http)/" \
                $GRUB_MODULES

# more grub.config
set timeout=5

net_bootp
net_ls_addr
net_ls_cards
net_ls_routes
lspci
set net_default_server=10.0.1.2
serial --speed=115200 --unit=0 --word=8
terminal_output  serial console
terminal_input  serial console
multiboot2 (http,10.0.1.2)/xen.gz dom0_mem=max:1G loglvl=all
guest_loglvl=all console=com1 com1=115200,8n1
module2 (http,10.0.1.2)/vmlinuz console=hvc0 debug initcall_debug
module2 (http,10.0.1.2)/initrd.img
boot

Re: [ipxe-devel] Tips on how to debug EFI code (iPXE) from within KVM after ipxe.efi has crashed with #GP?

[Michael Brown]

On 28/09/17 18:37, Konrad Rzeszutek Wilk wrote:
> !!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID -
> 00000000 !!!!
> ExceptionData - 0000000000000000
> RIP  - 00000000BEC2949C, CS  - 0000000000000038, RFLAGS - 0000000000210216
 > ....
> !!!! Find image 808610ed.efidrv (ImageBase=00000000BEC27000,
> EntryPoint=00000000BEC2E089) !!!!
> 
> And now I am trying to figure out how to troubleshoot this.
> (and yes I am thinking it was related to the Tivoli work-around, but
> disabling that didn't help).

The Tivoli workaround is for legacy BIOS only; it doesn't apply to the 
UEFI build of iPXE.

You have the RIP and ImageBase, so you know that the exception happens 
at offset +0x249c within your iPXE binary.  You can use this in 
conjunction with the corresponding map file from the iPXE build (which 
will probably be named bin-x86_64-efi/808610d3.efidrv.tmp.map, but see 
below) to figure out exactly where the crash is occurring.

> #crash ipxe/src/bin/ipxe.lkrn.tmp ./guest-memory

That's a completely different iPXE binary from the one you are using.

> - The 80810ed.efidrv is built using:
> 
> (cd ipxe/src;make bin-x86_64-efi/808610d3.efi  CONFIG=qemu
> DEBUG=iscsi:4,scsi:4 EMBED=/ipxe.config -j8)
> if [ $? -ne 0 ]; then exit 1; fi
> (cd ipxe/src;EfiRom -f 0x8086 -i 0x1528 --pci23 -ec
> bin-x86_64-efi/808610d3.efidrv -o ../../808610d3.rom)
> if [ $? -ne 0 ]; then exit 1; fi

You can build this much more simply in a single command:

   make bin-x86_64-efi/808610d3.efirom EMBED=/ipxe.config

Also note that:

- debug levels in iPXE are bitmasks, so DEBUG=....:4 generally doesn't 
make much sense.  You probably want DEBUG=iscsi,scsi for minimal debug 
(i.e. detailed error messages for any problems), or DEBUG=iscsi:3,scsi:3 
for much more verbose logging.  That said, you don't seem to be using 
iSCSI anyway, so .....?

- In your build chain (using EfiRom) I think you have a typo: you build 
the EFI _application_ bin-x86_64-efi/808610d3.efi but then use the EFI 
_driver_ bin-x86_64-efi/808610d3.efidrv, which you have not explicitly 
built.  Your final binary may therefore not match what you think you 
have just built.  You should probably not worry about any of this, and 
just use the single-command build given above instead.

Michael

Re: [ipxe-devel] Tips on how to debug EFI code (iPXE) from within KVM after ipxe.efi has crashed with #GP?

[Laszlo Ersek]

On 09/28/17 20:04, Michael Brown wrote:
> On 28/09/17 18:37, Konrad Rzeszutek Wilk wrote:
>> !!! X64 Exception Type - 0D(#GP - General Protection)  CPU Apic ID -
>> 00000000 !!!!
>> ExceptionData - 0000000000000000
>> RIP  - 00000000BEC2949C, CS  - 0000000000000038, RFLAGS -
>> 0000000000210216
>> ....
>> !!!! Find image 808610ed.efidrv (ImageBase=00000000BEC27000,
>> EntryPoint=00000000BEC2E089) !!!!
>>
>> And now I am trying to figure out how to troubleshoot this.
>> (and yes I am thinking it was related to the Tivoli work-around, but
>> disabling that didn't help).
> 
> The Tivoli workaround is for legacy BIOS only; it doesn't apply to the
> UEFI build of iPXE.
> 
> You have the RIP and ImageBase, so you know that the exception happens
> at offset +0x249c within your iPXE binary.  You can use this in
> conjunction with the corresponding map file from the iPXE build (which
> will probably be named bin-x86_64-efi/808610d3.efidrv.tmp.map, but see
> below) to figure out exactly where the crash is occurring.

Or run "objdump -S 808610d3.efidrv.tmp", and look up the offset in the
output. (First, check if (EntryPoint - ImageBase), i.e., 0x7089, equals
"start address" in the "objdump -x" output.)

Laszlo