Hello all, I am encountering an intermittent crash issue with AMD SEV-ES guests when using OVMF from edk2-stable202505 up to the latest master (commit d82e9b7). The issue does not occur with edk2-stable202502. Due to its intermittent nature, bisecting has been challenging, and I’m seeking assistance from the community to debug and resolve this issue. _Issue Description_ The AMD SEV-ES guest crashes intermittently during a loop of 200 start/shutdown tests. The crash occurs at varying points: sometimes within 30 loops, sometimes around 100, or after 150+ loops. I’ve reproduced this in my local CI setup but have been unable to pinpoint a specific commit due to the issue’s inconsistency. _Reproduction Steps_ 1. Build upstream EDK2 OVMF (OVMF_X64/OVMF.fd). 2. Run the following QEMU command in a loop of 200 start/shutdown cycles: |qemu-system-x86_64 \ -machine q35,confidential-guest-support=sev0,vmport=off \ -object sev-guest,id=sev0,policy=0x5,cbitpos=51,reduced-phys-bits=1 \ -name guest=vm,debug-threads=on \ -drive if=pflash,format=raw,unit=0,file=/home/VT_BUILD/usr/local/OVMF_X64/OVMF.fd,readonly \ -m 4096 \ -object memory-backend-ram,size=4096M,id=mem-machine_mem \ -smp 255,maxcpus=255,cores=255,threads=1,dies=1,sockets=1 \ -cpu 'EPYC-Genoa-v1',+svm,+x2apic \ -kernel /home/VT_BUILD/usr/local/bzImage \ -append 'root=/dev/sda rw console=ttyS0,115200n8 earlyprintk=ttyS0,115200 net.ifnames=0 biosdevname=0 movable_node swiotlb=65536' \ -drive id=disk0,file=/home/VT_BUILD/22.04-server_vm1.qcow2,format=qcow2,if=none \ -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true \ -device scsi-hd,drive=disk0 \ --enable-kvm \ --nographic \ -monitor tcp:localhost:4444,server,nowait \ -trace "kvm_.*|cpu_.*|exec_.*|translate_.*|sev_.*|memory_region_ops_.*|apic_.*" | Below is the crash reported in the guest serial log: WakeupBufferStart = 87000, WakeupBufferSize = E5 AP Vector: 16-bit = 87000/39, ExchangeInfo = 87040/A5 WakeupBufferStart = 7F000, WakeupBufferSize = 7F80 Dxe: SevEsAPMemory = 7E36C000 CpuDxe: 5-Level Paging = 0 error: kvm run failed Invalid argument EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00error: kvm run failed Invalid argument SS =0000 00000000 0000ffff 00009300 error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 error: kvm run failed Invalid argument DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument Aerror: kvm run failed Invalid argument EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument error: kvm run failed Invalid argument TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 error: kvm run failed Invalid argument EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 .. EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 .. EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a10f10 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000b004 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 00800000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=2d d9 26 77 a1 6a ae 2a 1f db a6 dc 82 47 d4 3d f1 a7 0f a7 bd 5f b3 01 71 5a c6 ff 55 03 59 98 b6 a2 ad 8e 27 85 79 65 38 81 c5 32 6b 3f ea 00 00 _Component Versions Tested_ * *QEMU*: I have used both stable 9.x and 10.x, issue gets recreated with both. * *OVMF*: edk2-stable202505 to master (commit d82e9b7); no issues with edk2-stable202502 * *Host Kernel*: Ubuntu Noble (6.8.0-60-generic) and latest RC kernel (v6.16-rc3), issue gets recreated with both. * *Guest Image*: Ubuntu 22.04 server (qcow2 format) with distro kernel as well as latest RC kernel (v6.16-rc3), issue gets recreated with both. I am attaching host traces captured by enabling kvm_amd.dump_invalid_vmcb=1 as well as the guest serial log to provide more context. I can provide additional logs, traces, or test results if needed to aid debugging. Best regards, Srikanth Aithal -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#121431): https://edk2.groups.io/g/devel/message/121431 Mute This Topic: https://groups.io/mt/113838839/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-