Hello all,

I am encountering an intermittent crash issue with AMD SEV-ES guests when using OVMF from edk2-stable202505 up to the latest master (commit d82e9b7). The issue does not occur with edk2-stable202502. 

Due to its intermittent nature, bisecting has been challenging, and I’m seeking assistance from the community to debug and resolve this issue.

Issue Description

The AMD SEV-ES guest crashes intermittently during a loop of 200 start/shutdown tests. The crash occurs at varying points: sometimes within 30 loops, sometimes around 100, or after 150+ loops. I’ve reproduced this in my local CI setup but have been unable to pinpoint a specific commit due to the issue’s inconsistency.

Reproduction Steps

1. Build upstream EDK2 OVMF (OVMF_X64/OVMF.fd).

2. Run the following QEMU command in a loop of 200 start/shutdown cycles:

qemu-system-x86_64 \
  -machine q35,confidential-guest-support=sev0,vmport=off \
  -object sev-guest,id=sev0,policy=0x5,cbitpos=51,reduced-phys-bits=1 \
  -name guest=vm,debug-threads=on \
  -drive if=pflash,format=raw,unit=0,file=/home/VT_BUILD/usr/local/OVMF_X64/OVMF.fd,readonly \
  -m 4096 \
  -object memory-backend-ram,size=4096M,id=mem-machine_mem \
  -smp 255,maxcpus=255,cores=255,threads=1,dies=1,sockets=1 \
  -cpu 'EPYC-Genoa-v1',+svm,+x2apic \
  -kernel /home/VT_BUILD/usr/local/bzImage \
  -append 'root=/dev/sda rw console=ttyS0,115200n8 earlyprintk=ttyS0,115200 net.ifnames=0 biosdevname=0 movable_node swiotlb=65536' \
  -drive id=disk0,file=/home/VT_BUILD/22.04-server_vm1.qcow2,format=qcow2,if=none \
  -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true \
  -device scsi-hd,drive=disk0 \
  --enable-kvm \
  --nographic \
  -monitor tcp:localhost:4444,server,nowait \
  -trace "kvm_.*|cpu_.*|exec_.*|translate_.*|sev_.*|memory_region_ops_.*|apic_.*"

Below is the crash reported in the guest serial log:

WakeupBufferStart = 87000, WakeupBufferSize = E5
AP Vector: 16-bit = 87000/39, ExchangeInfo = 87040/A5
WakeupBufferStart = 7F000, WakeupBufferSize = 7F80
Dxe: SevEsAPMemory = 7E36C000
CpuDxe: 5-Level Paging = 0
error: kvm run failed Invalid argument
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00error: kvm run failed Invalid argument

SS =0000 00000000 0000ffff 00009300
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 error: kvm run failed Invalid argument

DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
Aerror: kvm run failed Invalid argument
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
error: kvm run failed Invalid argument
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
error: kvm run failed Invalid argument
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00
..
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00
..
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 00800000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 <bf> bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00

Component Versions Tested

• QEMU: I have used both stable 9.x and 10.x, issue gets recreated with both.
  

• OVMF: edk2-stable202505 to master (commit d82e9b7); no issues with edk2-stable202502

• Host Kernel: Ubuntu Noble (6.8.0-60-generic) and latest RC kernel (v6.16-rc3), issue gets recreated with both.
  

• Guest Image: Ubuntu 22.04 server (qcow2 format) with distro kernel as well as latest RC kernel (v6.16-rc3), issue gets recreated with both.

I am attaching host traces captured by enabling kvm_amd.dump_invalid_vmcb=1 as well as the guest serial log to provide more context.

I can provide additional logs, traces, or test results if needed to aid debugging.

Best regards,

Srikanth Aithal <sraithal@amd.com>

You receive all messages sent to this group.

View/Reply Online (#121431) | | Mute This Topic | New Topic
Your Subscription | Contact Group Owner | Unsubscribe [rebecca@openfw.io]