Re: [edk2-devel] [PATCH V6 1/1] OvmfPkg: Enable TDX in ResetVector

["Min Xu" <min.m.xu@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6074 bytes --]

Hi, Brijesh

On September 18, 2021 7:30 PM, Brijesh Singh wrote:
> Hi Min,
> 
> On 9/18/21 12:16 AM, Xu, Min M wrote:
> > Hi, Brijesh
> >
> > On September 17, 2021 11:52 PM, Brijesh Singh wrote:
> >> Hi Min,
> >>
> >> On 9/17/21 7:55 AM, Xu, Min M wrote:
> >> ...
> >>
> >>> As I mentioned in my last mail, in the beginning I missed the
> >>> limitation of
> >> smsw.
> >>> So I update the code (ResetVectorVtf0.asm) as below using mov CRx.
> >>>                                                            <1> BITS 16
> >>>     176 00000800 0F20C0              <1>     mov   eax, cr0    <-- previously it
> >> was smsw
> >>>     177 00000803 A801                  <1>     test    al, 1
> >>>     178 00000805 7405                  <1>     jz       .Real
> >>>     179                                               <1> BITS 32
> >>>     180 00000807 E951FFFFFF      <1>     jmp   Main32
> >>>     181                                               <1> BITS 16
> >>>     182                                               <1> .Real:
> >>>     183 0000080C E939FF              <1>     jmp   EarlyBspInitReal16
> >>>
> >>> I test the code in a AMD SEV server and try to launch a SEV guest.
> >>> This time
> >> it stuck at the *mov eax, cr0*.
> >>> I am curious if *mov eax, cr0* works in real mode in a SEV guest?
> >>> I also test the code in a legacy vm guest and td guest, all passed.
> >>> Did I miss something?
> >>>
> >> Hmm, I am not aware of any limitation w.r.t encrypted VMs. I just
> >> added the below code in my branch and I do not see any issues, my
> >> SEV, SEV-ES and SEV-SNP all are able to boot fine. And KVM trace
> >> confirms that code it read
> >>
> >> diff --git a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
> >> b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
> >> index f0e509d0672e..98e34332b04c 100644
> >> --- a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
> >> +++ b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
> >> @@ -175,9 +175,21 @@ resetVector:
> >>   ;
> >>   ; This is where the processor will begin execution
> >>   ;
> >> +%ifdef ARCH_IA32
> >>       nop
> >>       nop
> >>       jmp     EarlyBspInitReal16
> >> +%else
> >> +    mov     eax, cr0
> >> +    test    al, 1
> >> +    jz      .Real
> >> +BITS 32
> >> +    hlt
> >> +    ;jmp     Main32
> >> +BITS 16
> >> +.Real:
> >> +    jmp     EarlyBspInitReal16
> >> +%endif
> >>
> >>   ALIGN   16
> >>
> >>
> >> And KVM trace:
> >>
> >> kvm_exit: vcpu 0 reason npf rip 0xfff0 info1 0x0000000500000014 info2
> >> 0x00000000fffff000 intr_info 0x00000000 error_code 0x00000000
> >> kvm_page_fault: address fffff000 error_code 500000014
> >> kvm_entry: vcpu 0, rip 0xfff0
> >> kvm_exit: vcpu 0 reason read_cr0 rip 0xfff0 info1 0x8000000000000000
> >> info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
> >> kvm_cr: cr_read 0 = 0x60000010
> >> kvm_entry: vcpu 0, rip 0xfff3
> >>
> >> As we can see from the kvm trace, the first instruction here is the
> >> Cr0 read and it was successfully intercepted and rip moved to next
> instruction.
> >>
> >> Can you please provide me KVM trace for your failure case ? Also,
> >> provide me the output of "lscpu" and "dmesg" from the host.
> > The OVMF image you tested is built with GCC tool chain, right?
> 
> Yes, we have been using the GCC tool chain only.
>
Is VS Tool chain (VS2017, VS2019, etc) supported by AMD SEV in OVMF? 
I am a little nervous when the Ovmf img failed to be launched in AMD SEV
server after my TDX patch is applied.
> 
> 
> > I usually do the development in windows and build the OVMF image with
> VS2019.
> > If the new feature works, then I cherry-pick the patch-sets to code
> > base in ubuntu
> > 18.04 and build/test the new feature.
> >
> > The weird thing is that, with VS2019, even the OVMF image is built
> > from edk2-master, such image doesn't work on AMD SEV server either.
> > But if the image is built by Ubuntu 18.04, it does work on AMD SEV server.
> 
> This seems very strange that we are failing to execute the hand written
> assembly code.
Actually even the OvmfPkg from edk2-master (without any changes) cannot be
launched on AMD SEV server if it is built with VS2019 tool chain.

This is the qemu-kvm used:
$/usr/libexec/qemu-kvm --version
QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-48.module_el8.4.0+885+5e18b468.3)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

This is the launch scripts.

QEMU=/usr/libexec/qemu-kvm
DRIVE=rhel-8.qcow2

${QEMU} \
  -enable-kvm -cpu EPYC -machine q35 \
  -smp 4,maxcpus=64 \
  -m 4096M,slots=5,maxmem=30G \
  -drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly \
  -drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \
  -netdev user,id=vmnic \
  -device e1000,netdev=vmnic,romfile= \
  -drive file=${DRIVE},if=none,id=disk0,format=qcow2 \
  -device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true \
  -device scsi-hd,drive=disk0 \
  -object sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1 \
  -machine memory-encryption=sev0 \
  -nographic

>I am wondering if somehow the VS compiler is generating a
> wrong byte code and thus causing a trap on KVM that requires emulation.
> Since the guest memory is encrypted, so KVM emulation code will not be
> able to decode the instruction bytes and thus leading in repetitive nested
> fault. Only way I could verify my theory is if I can get a KVM trace or an OVMF
> binary. If you have have KVM trace or OVMF_CODE.fd handy then please
> share.
>
The OVMF_CODE.fd and OVMF_VARS.fd are attached.

The code base is :
ac6388add4 2021-09-15 (HEAD -> master, origin/master, origin/HEAD) ArmPkg/ProcessorSubClassDxe: Fix the format of ProcessorId

This is the build command:
build -p OvmfPkgX64.dsc -a X64 -t VS2019
> 
> 
> > I applied my TDX patch-sets to the code base on my Ubuntu 18.04, and
> build the image.
> > This image does work in both AMD SEV server and Intel TDX server.
> >

Thanks!
Min

[-- Attachment #2: OVMF_VARS.fd --]
[-- Type: application/octet-stream, Size: 540672 bytes --]

[-- Attachment #3: OVMF_CODE.fd --]
[-- Type: application/octet-stream, Size: 3653632 bytes --]