Re: [edk2-devel] [PATCH v7 00/43] SEV-ES guest support

["Lendacky, Thomas" <thomas.lendacky@amd.com>]

On 5/15/20 12:47 AM, Ni, Ray wrote:
> I just realized my solution doesn't cover some scenarios:
> 1. SMM
> 2. S3 boot path
> 3. CapsuleX64
> 
> If we want to hook #29 in all scenarios, your way
> directly modifying CpuExceptionHandlerLib is the
> easiest way because RegisterInterruptHandler() is supported
> only in DXE/SMM case.
> There is no way to use RegisterXXX() API for #2 and #3 because
> PEI instance doesn't support.
> 
> Do we need to hook in all scenarios? What instructions could
> cause #29?

A #VC can only be genereted by an SEV-ES guest, so in the standard case 
the hook will never be called. For an SEV-ES guest, a number of different 
instructions can cause a #VC. These include IO instructions (such as used 
to output debug message to the serial port), CPUID instructions, anything 
performing MMIO, RDMSR/WRMSR instructions, etc. So we need to hook it in 
all SEV-ES scenarios.

To eliminate all of the handler code in the standard case, I'm planning on 
providing a NULL VmgExitLib library that has a (near) empty #VC handler so 
that the full #VC handler code will only be in the Ovmf package.

> 
> 
> I don't see much difference between the new way
> introducing OverrideCpuExceptionHandler () and directly modifying
> the CpuExceptionHandlerLib. Both ways modifies the library.
> Introducing OverrideCpuExceptionHandler() might be worse because
> it creates an interface which encourages anyone to hook any exceptions.
> 
> Your current way only hooks #29.

Ok, I can go back to the explicit check for exception #29.

I'll work to get these changes and changes from other feedback into the 
next version and out for review early next week.

Thanks for taking the time to work through this with me!

Tom

> 
> Thanks,
> Ray
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Lendacky, Thomas
>> Sent: Friday, May 15, 2020 1:59 AM
>> To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io; afish@apple.com
>> Cc: Justen, Jordan L <jordan.l.justen@intel.com>; Laszlo Ersek <lersek@redhat.com>; Ard Biesheuvel
>> <ard.biesheuvel@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Dong,
>> Eric <eric.dong@intel.com>; Brijesh Singh <brijesh.singh@amd.com>; You, Benjamin <benjamin.you@intel.com>; Bi,
>> Dandan <dandan.bi@intel.com>; Dong, Guo <guo.dong@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; Wang, Jian J
>> <jian.j.wang@intel.com>; Ma, Maurice <maurice.ma@intel.com>; Fan Jeff <vanjeff_919@hotmail.com>
>> Subject: Re: [edk2-devel] [PATCH v7 00/43] SEV-ES guest support
>>
>> On 5/14/20 8:10 AM, Ni, Ray wrote:
>>> Tom,
>>
>> Hi Ray,
>>
>>> I just discussed with original CPU owner Jeff and went through how IDT is setup in the boot flow.
>>> Here is what I think you can do to avoid modifying the CpuExceptionHandlerLib.
>>> 1. SecPlatformMain() modifies IDT[29] to point to your VC handler. This step helps to build the VC handler in whole 32bit
>> mode SEC+PEI.
>>
>> That can probably be done, but duplicates a lot of code - all of the
>> exception entry assembler code.
>>
>> Additionally, UefiCpuPkg/CpuMpPei/CpuMpPei.c will also invoke
>> InitializeCpuExceptionHandlers() registering a new IDT[29] entry.
>>
>>> 2. Create a new DXE driver with dependency set to TRUE and call RegisterCpuInteruptHandler(29, xx) in its entrypoint to
>> register VC handler for whole 64bit mode DXE.
>>> 3. Platform FDF uses apriori file mechanism to make sure the driver created in step #2 is dispatched as the 1st driver in
>> DXE phase. This step is optional if you accept there is some time that VC handler is not setup in early DXE phase.
>>
>> Tracing the execution of an apriori driver shows that this happens after
>> DXE has initialized its exception handler and #VCs occur before a handler
>> can be reigstered by the new driver, causing a failure.
>>
>>> 4. In the new DXE driver, gets the EFI_VECTOR_HANDOFF_INFO (MdePkg\Include\Ppi\VectorHandoffInfo.h) from
>> configuration table.
>>>        It reports failure if the vector_handoff table says DO_NOT_HOOK for #29.
>>>        It re-produces vector_handoff table with #29 set to DO_NOT_HOOK so that no one could use CpuArch protocol to
>> override #29 handler.
>>>
>>>
>>> In general, I want to use the API/capability provided by CpuExceptionHandlerLib instead of directly modifying it for
>> handler registration.
>>> Directly modifying it gives an improper code reference/example for future developers.
>>
>> I also don't see how this method will allow me to easily propagate a new
>> exception value through the exception handling stack.
>>
>> My current plan was to create a CpuExceptionOverrideLib library that is
>> invoked as part of exception handling. This allows immediate ability to
>> hook any exception without having to wait for an opportunity to register a
>> handler - which in the case of #VC, is too late. Future developers that
>> need immediate exception handling will be able to override the default
>> library without any modification to CpuExceptionHandlerLib.
>>
>>
>> The change would look something like this:
>>
>> diff --git a/UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuException.c
>> b/UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuException.c
>> index 20148db74cf8..7ac86f56d7d2 100644
>> --- a/UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuException.c
>> +++ b/UefiCpuPkg/Library/CpuExceptionHandlerLib/SecPeiCpuException.c
>> @@ -7,6 +7,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
>>   **/
>>
>>   #include <PiPei.h>
>> +#include <Library/CpuExceptionOverrideLib.h>
>>   #include "CpuExceptionCommon.h"
>>
>>   CONST UINTN    mDoFarReturnFlag  = 0;
>> @@ -24,6 +25,22 @@ CommonExceptionHandler (
>>     IN EFI_SYSTEM_CONTEXT   SystemContext
>>     )
>>   {
>> +  EFI_STATUS  Status;
>> +
>> +  //
>> +  // If the exception is overridden, exit early.
>> +  //
>> +  Status = OverrideCpuExceptionHandler (ExceptionType, SystemContext);
>> +  if (Status == EFI_SUCCESS) {
>> +    return;
>> +  }
>> +
>> +  //
>> +  // If the exception was not overridden, then the extract the exception value
>> +  // to continue with.
>> +  //
>> +  ExceptionType = OVERRIDE_EXCEPTION (Status);
>> +
>>
>> (To request vector 0 (#DE), the return is encoded to be non-zero and the
>>   exception value extracted)
>>
>>
>> The NULL implementation of the override library would just return the
>> current exception type so that exception processing continues as today.
>>
>> This seems to be the best way to handle the #VC exception without hard
>> coding it into CpuExceptionHandlerLib and being able to catch a #VC as
>> soon as possible.
>>
>> Thoughts?
>>
>> Thanks,
>> Tom
>>
>>>
>>> Thanks,
>>> Ray
>>>
>>>> -----Original Message-----
>>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Lendacky, Thomas
>>>> Sent: Tuesday, May 12, 2020 11:00 PM
>>>> To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io; afish@apple.com
>>>> Cc: Justen, Jordan L <jordan.l.justen@intel.com>; Laszlo Ersek <lersek@redhat.com>; Ard Biesheuvel
>>>> <ard.biesheuvel@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>;
>> Dong,
>>>> Eric <eric.dong@intel.com>; Brijesh Singh <brijesh.singh@amd.com>; You, Benjamin <benjamin.you@intel.com>; Bi,
>>>> Dandan <dandan.bi@intel.com>; Dong, Guo <guo.dong@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; Wang, Jian J
>>>> <jian.j.wang@intel.com>; Ma, Maurice <maurice.ma@intel.com>
>>>> Subject: Re: [edk2-devel] [PATCH v7 00/43] SEV-ES guest support
>>>>
>>>> On 5/11/20 12:24 AM, Ni, Ray wrote:
>>>>> Tom,
>>>>>
>>>>> I agree with the first issue. I am not quite clear on the second one.
>>>>
>>>> In regards to the exception propagation, the hypervisor is allowed to
>>>> request an exception as part of the return information. For example, the
>>>> guest issues a RDMSR instruction for an invalid MSR. The hypervisor would
>>>> normally inject a #GP into the guest. With SEV-ES, the VC handler has to
>>>> do this. Hence the need to possibly propogate to other exception handlers
>>>> after handling the #VC.
>>>>
>>>>>
>>>>> SourceLevelDebugPkg provides source level debugging support early in SEC
>>>>> through SourceLevelDebugPkg\Library\DebugAgent\SecPeiDebugAgent\.
>>>>>
>>>>> It hooks all Intel SDM defined exceptions. It hooks INT32 additionally to
>>>>> support breaking from HOST.
>>>>>
>>>>> It doesn't use CpuExceptionLib because it hooks in very early SEC phase.
>>>>>
>>>>> Can you use the same way?
>>>>
>>>> I can look at trying to do something like this. I guess the source level
>>>> debug needs to be aware of all the exceptions, which is why it hooks all
>>>> them. The SEV-ES support is only concerned with the #VC exception. It just
>>>> seems like a lot of duplicated and extra code vs. checking for / handling
>>>> the #VC exception in the CpuExceptionHandler library.
>>>>
>>>> My plan for v8 is/was to have a NULL VmgExitLib library, of which the #VC
>>>> handler would be part of the interface, with the CpuExceptionHandler
>>>> library invoking the #VC handler on #VC exception and having the OvmfPkg
>>>> provide a VmgExitLib library with all the functionality.
>>>>
>>>> Thanks,
>>>> Tom
>>>>
>>>>>
>>>>> Thanks,
>>>>> Ray
>>>>>
>>>>> *From:* devel@edk2.groups.io <devel@edk2.groups.io> *On Behalf Of *Andrew
>>>>> Fish via groups.io
>>>>> *Sent:* Sunday, May 10, 2020 3:10 AM
>>>>> *To:* devel@edk2.groups.io; thomas.lendacky@amd.com
>>>>> *Cc:* Ni, Ray <ray.ni@intel.com>; Justen, Jordan L
>>>>> <jordan.l.justen@intel.com>; Laszlo Ersek <lersek@redhat.com>; Ard
>>>>> Biesheuvel <ard.biesheuvel@linaro.org>; Kinney, Michael D
>>>>> <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Dong,
>>>>> Eric <eric.dong@intel.com>; Brijesh Singh <brijesh.singh@amd.com>; You,
>>>>> Benjamin <benjamin.you@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Dong,
>>>>> Guo <guo.dong@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; Wang, Jian J
>>>>> <jian.j.wang@intel.com>; Ma, Maurice <maurice.ma@intel.com>
>>>>> *Subject:* Re: [edk2-devel] [PATCH v7 00/43] SEV-ES guest support
>>>>>
>>>>>
>>>>>
>>>>>       On May 9, 2020, at 7:34 AM, Lendacky, Thomas <thomas.lendacky@amd.com
>>>>>       <mailto:thomas.lendacky@amd.com>> wrote:
>>>>>
>>>>>       On 5/9/20 1:44 AM, Ni, Ray wrote:
>>>>>
>>>>>           Tom,
>>>>>
>>>>>
>>>>>       Hi Ray,
>>>>>
>>>>>
>>>>>           I have a bit concern on your change that directly modifies
>>>>>           CpuExceptionHandlerLib to handle
>>>>>           exception #29. Today's CpuExceptionHandlerLib simplify dumps the
>>>>>           exception context for
>>>>>           every exception. Any component which wants to do specific handling
>>>>>           of certain exceptions
>>>>>           should call RegisterCpuInterruptHandler(). Such as code in CpuDxe
>>>>>           driver:
>>>>>              if (HEAP_GUARD_NONSTOP_MODE || NULL_DETECTION_NONSTOP_MODE) {
>>>>>                RegisterCpuInterruptHandler (EXCEPT_IA32_DEBUG,
>>>>>           DebugExceptionHandler);
>>>>>                RegisterCpuInterruptHandler (EXCEPT_IA32_PAGE_FAULT,
>>>>>           PageFaultExceptionHandler);
>>>>>              }
>>>>>           Is it possible for your feature to follow the same pattern?
>>>>>
>>>>>
>>>>>       There are two problems:
>>>>>
>>>>>       The first is that RegisterCpuInterruptHandler() is not implemented for
>>>>>       both the SEC and PEI phases, so it is not currently possible to
>>>>>       register a handler that early.
>>>>>
>>>>>       The second is that I need to be able to propagate an exception request
>>>>>       from the hypervisor. With the current implementation there doesn't
>>>>>       appear to be an easy way to perform this propagation.
>>>>>
>>>>>       If there's a way to accomplish both of the above I wouldn't be opposed
>>>>>       to using RegisterCpuInterruptHandler() as long as there are no #VCs
>>>>>       that can occur between initializing exception handling and and
>>>>>       registering the #VC handler.
>>>>>
>>>>> Thomas,
>>>>>
>>>>> As you point out it is tricky dealing with XIP code. You can't have
>>>>> globals that you can write and generally you use a PEI service to look
>>>>> tings up, the most common thing being using a HOB. But SEC has no services
>>>>> and I'm not sure you really want to be calling into the PEI Core on a
>>>>> random  exception.
>>>>>
>>>>> Here are the best options that popped into my head after reading your email
>>>>>
>>>>> 1) IDT in RAM
>>>>>
>>>>> If your code populates the IDT the IDTR gives you access to the address of
>>>>> the IDTR via an instruction. The PI Spec reserves IDT - sizeof (UNITN) for
>>>>> a cached copy of the PEI Services Table, but otther than that you are good
>>>>> to go. It should be possible to have a global so you can have the table
>>>>> required to implement RegisterCpuInterruptHandler(). There might be some
>>>>> usage  of IDT - ( 2* sizeof(UINTN)), I know I'm guilty, so storing data
>>>>> after the IDT would be a good option. In general if your code allocates
>>>>> the memory for the IDT then you can treat the IDT as part of your private
>>>>> context data structure and that gives you access
>>>>>
>>>>> 2) IDT in ROM.
>>>>>
>>>>> For this it seems like you need a library to link in to
>>>>> the CpuExceptionHandlerLib that allows you to override the handler. If
>>>>> CpuInterruptHandlerOverride() returns NULL you do the current behavior if
>>>>> not NULL then you call the returned handler.
>>>>>
>>>>> EFI_CPU_INTERRUPT_HANDLER
>>>>>
>>>>> EFIAPI
>>>>>
>>>>> OverrideCpuInterruptHandler (
>>>>>
>>>>>      IN EFI_EXCEPTION_TYPE            InterruptType
>>>>>
>>>>>      );
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Andrew Fish
>>>>>
>>>>> PS Off topic, but it would also be useful to have a library that overrides
>>>>> the state dump display. For example using Xcode you can always display a
>>>>> stack frame from the exception handler.
>>>>>
>>>>>
>>>>>
>>>>>       Thanks,
>>>>>       Tom
>>>>>
>>>>>
>>>>>           Thanks,
>>>>>           Ray
>>>>>
>>>>>               -----Original Message-----
>>>>>               From: Tom Lendacky <thomas.lendacky@amd.com
>>>>>               <mailto:thomas.lendacky@amd.com>>
>>>>>               Sent: Saturday, May 9, 2020 3:16 AM
>>>>>               To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
>>>>>               Cc: Justen, Jordan L <jordan.l.justen@intel.com
>>>>>               <mailto:jordan.l.justen@intel.com>>; Laszlo Ersek
>>>>>               <lersek@redhat.com <mailto:lersek@redhat.com>>; Ard Biesheuvel
>>>>>               <ard.biesheuvel@linaro.org
>>>>>               <mailto:ard.biesheuvel@linaro.org>>; Kinney, Michael D
>>>>>               <michael.d.kinney@intel.com
>>>>>               <mailto:michael.d.kinney@intel.com>>; Gao, Liming
>>>>>               <liming.gao@intel.com <mailto:liming.gao@intel.com>>; Dong,
>>>>>               Eric <eric.dong@intel.com <mailto:eric.dong@intel.com>>; Ni,
>>>>>               Ray <ray.ni@intel.com <mailto:ray.ni@intel.com>>; Brijesh
>>>>>               Singh <brijesh.singh@amd.com <mailto:brijesh.singh@amd.com>>;
>>>>>               You, Benjamin
>>>>>               <benjamin.you@intel.com <mailto:benjamin.you@intel.com>>; Bi,
>>>>>               Dandan <dandan.bi@intel.com <mailto:dandan.bi@intel.com>>;
>>>>>               Dong, Guo <guo.dong@intel.com <mailto:guo.dong@intel.com>>;
>>>>>               Wu, Hao A
>>>>>               <hao.a.wu@intel.com <mailto:hao.a.wu@intel.com>>; Wang, Jian J
>>>>>               <jian.j.wang@intel.com <mailto:jian.j.wang@intel.com>>; Ma,
>>>>>               Maurice <maurice.ma@intel.com <mailto:maurice.ma@intel.com>>
>>>>>               Subject: Re: [PATCH v7 00/43] SEV-ES guest support
>>>>>
>>>>>               I was able to use the pull request method that Laszlo
>>>>>               documented and fixed
>>>>>               up all of the issues identified by the VS compiler.
>>>>>
>>>>>               An additional change I'm planning to make for the next version
>>>>>               (v8) of the
>>>>>               patches is to create a NULL library instance of the VmgExitLib
>>>>>               that will
>>>>>               also include the #VC handler function. This will reduce the
>>>>>               amount of code
>>>>>               associated with this feature for platforms that don't
>>>>>               use/support SEV-ES.
>>>>>
>>>>>               Laszlo, this will mean that I will introduce a version of the
>>>>>               VmgExitLib
>>>>>               under OvmfPkg that will provide the majority of the
>>>>>               functionality that is
>>>>>               present today in UefiCpuPkg. In essence, the functionality in
>>>>>               v7 patches 8
>>>>>               and 11 - 25 will now live under OvmfPkg instead of UefiCpuPkg.
>>>>>               I think
>>>>>               this is the better way to do this. Let me know if you have any
>>>>>               concerns.
>>>>>
>>>>>               Thanks,
>>>>>               Tom
>>>>>
>>>>>               On 4/22/20 12:41 PM, Tom Lendacky wrote:
>>>>>
>>>>>                   This patch series provides support for running EDK2/OVMF
>>>>>                   under SEV-ES.
>>>>>
>>>>>                   Secure Encrypted Virtualization - Encrypted State (SEV-ES)
>>>>>                   expands on the
>>>>>                   SEV support to protect the guest register state from the
>>>>>                   hypervisor. See
>>>>>                   "AMD64 Architecture Programmer's Manual Volume 2: System
>>>>>                   Programming",
>>>>>                   section "15.35 Encrypted State (SEV-ES)" [1].
>>>>>
>>>>>                   In order to allow a hypervisor to perform functions on
>>>>>                   behalf of a guest,
>>>>>                   there is architectural support for notifying a guest's
>>>>>                   operating system
>>>>>                   when certain types of VMEXITs are about to occur. This
>>>>>                   allows the guest to
>>>>>                   selectively share information with the hypervisor to
>>>>>                   satisfy the requested
>>>>>                   function. The notification is performed using a new
>>>>>                   exception, the VMM
>>>>>                   Communication exception (#VC). The information is shared
>>>>>                   through the
>>>>>                   Guest-Hypervisor Communication Block (GHCB) using the
>>>>>                   VMGEXIT instruction.
>>>>>                   The GHCB format and the protocol for using it is
>>>>>                   documented in "SEV-ES
>>>>>                   Guest-Hypervisor Communication Block Standardization" [2].
>>>>>
>>>>>                   The main areas of the EDK2 code that are updated to
>>>>>                   support SEV-ES are
>>>>>                   around the exception handling support and the AP boot support.
>>>>>
>>>>>                   Exception support is required starting in Sec, continuing
>>>>>                   through Pei
>>>>>                   and into Dxe in order to handle #VC exceptions that are
>>>>>                   generated.  Each
>>>>>                   AP requires it's own GHCB page as well as a page to hold
>>>>>                   values specific
>>>>>                   to that AP.
>>>>>
>>>>>                   AP booting poses some interesting challenges. The
>>>>>                   INIT-SIPI-SIPI sequence
>>>>>                   is typically used to boot the APs. However, the hypervisor
>>>>>                   is not allowed
>>>>>                   to update the guest registers. The GHCB document [2] talks
>>>>>                   about how SMP
>>>>>                   booting under SEV-ES is performed.
>>>>>
>>>>>                   Since the GHCB page must be a shared (unencrypted) page,
>>>>>                   the processor
>>>>>                   must be running in long mode in order for the guest and
>>>>>                   hypervisor to
>>>>>                   communicate with each other. As a result, SEV-ES is only
>>>>>                   supported under
>>>>>                   the X64 architecture.
>>>>>
>>>>>
>>>>
>> [1]https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%25
>> 25
>>>>
>> 2F24593.pdf&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cf5d7875dfcf54e45c42208d7f3e4676b%7C3dd8961fe
>>>>
>> 4884e608e11a82d994e183d%7C0%7C0%7C637246036118033165&amp;sdata=H74fQl1n2sXzCMSoGm1tGOKc5epMtVkGJFCid
>>>> wLMl5c%3D&amp;reserved=0
>>>>>
>>>>
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%252
>> 52
>>>>
>> F24593.pdf&data=02%7C01%7Cthomas.lendacky%40amd.com%7Ca6a68a0fea9147d39c2508d7f56ba3c1%7C3dd8961fe4884
>>>>
>> e608e11a82d994e183d%7C0%7C0%7C637247716490462692&sdata=i3CuKMgAY08Cl%2FZWool7SIc3DTf%2BVA9HE%2BwpC8
>>>> lyZo0%3D&reserved=0>
>>>>>                   [2]https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-
>>>>
>> content%2Fresources%2F56421.pdf&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cf5d7875dfcf54e45c42208d7f
>>>>
>> 3e4676b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637246036118033165&amp;sdata=EwW9575nJMaWxizo2
>>>> XrLHjrbUMJIB0WFTDLjwy%2BM%2F4k%3D&amp;reserved=0
>>>>>                   <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.amd.com%2Fwp-
>>>>
>> content%2Fresources%2F56421.pdf&data=02%7C01%7Cthomas.lendacky%40amd.com%7Ca6a68a0fea9147d39c2508d7f56b
>>>>
>> a3c1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247716490472688&sdata=7GPXxfEPOzDIg8uFx2rx108eY4B
>>>> NIeKe0Of4K5Kuix4%3D&reserved=0>
>>>>>
>>>>>                   ---
>>>>>
>>>>>                   These patches are based on commit:
>>>>>                   be7295b36405 (".python/SpellCheck: Increase SpellCheck
>>>>>                   plugin max failures")
>>>>>
>>>>>                   Proper execution of SEV-ES relies on Bugzilla 2340 being
>>>>>                   fixed.
>>>>>
>>>>>                   A version of the tree (with an extra patch to workaround
>>>>>                   Bugzilla 2340) can
>>>>>                   be found at:
>>>>>
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAMDESE%2Fovmf%2Ftree%2Fsev-
>> es-
>>>>
>> v14&amp;data=02%7C01%7Cthomas.lendacky%40amd.com%7Cf5d7875dfcf54e45c42208d7f3e4676b%7C3dd8961fe4884e60
>>>>
>> 8e11a82d994e183d%7C0%7C0%7C637246036118033165&amp;sdata=U8fIzb%2F4A8WBaiVbScxUuGDw22kyxxnRP5olSyTedv
>>>> E%3D&amp;reserved=0
>>>>>
>>>>
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAMDESE%2Fovmf%2Ftree%2Fsev-
>>>> es-
>>>>
>> v14&data=02%7C01%7Cthomas.lendacky%40amd.com%7Ca6a68a0fea9147d39c2508d7f56ba3c1%7C3dd8961fe4884e608e1
>>>>
>> 1a82d994e183d%7C0%7C0%7C637247716490482690&sdata=27Er3PcupFhMsb%2F%2F5%2B9we7gW9NaDcjbVRgNp%2F%2F
>>>> 6vqMg%3D&reserved=0>
>>>>>
>>>>>                   Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org
>>>>>                   <mailto:ard.biesheuvel@linaro.org>>
>>>>>                   Cc: Benjamin You <benjamin.you@intel.com
>>>>>                   <mailto:benjamin.you@intel.com>>
>>>>>                   Cc: Dandan Bi <dandan.bi@intel.com
>>>>>                   <mailto:dandan.bi@intel.com>>
>>>>>                   Cc: Eric Dong <eric.dong@intel.com
>>>>>                   <mailto:eric.dong@intel.com>>
>>>>>                   Cc: Guo Dong <guo.dong@intel.com <mailto:guo.dong@intel.com>>
>>>>>                   Cc: Hao A Wu <hao.a.wu@intel.com <mailto:hao.a.wu@intel.com>>
>>>>>                   Cc: Jian J Wang <jian.j.wang@intel.com
>>>>>                   <mailto:jian.j.wang@intel.com>>
>>>>>                   Cc: Jordan Justen <jordan.l.justen@intel.com
>>>>>                   <mailto:jordan.l.justen@intel.com>>
>>>>>                   Cc: Laszlo Ersek <lersek@redhat.com
>>>>>                   <mailto:lersek@redhat.com>>
>>>>>                   Cc: Liming Gao <liming.gao@intel.com
>>>>>                   <mailto:liming.gao@intel.com>>
>>>>>                   Cc: Maurice Ma <maurice.ma@intel.com
>>>>>                   <mailto:maurice.ma@intel.com>>
>>>>>                   Cc: Michael D Kinney <michael.d.kinney@intel.com
>>>>>                   <mailto:michael.d.kinney@intel.com>>
>>>>>                   Cc: Ray Ni <ray.ni@intel.com <mailto:ray.ni@intel.com>>
>>>>>
>>>>>                   Changes since v6:
>>>>>                   - Add function comments to all functions, including local
>>>>>                   functions
>>>>>                   - Add function parameter direction to all functions (in/out)
>>>>>                   - Add support for MMIO MOVZX/MOVSX instructions
>>>>>                   - Ensure the per-CPU variable page remains encrypted
>>>>>                   - Coding-style fixes as identified by Ecc
>>>>>
>>>>>                   Changes since v5:
>>>>>                   - Remove extraneous VmgExitLib usage
>>>>>                   - Miscellaneous changes to address feedback (coding style,
>>>>>                   etc.)
>>>>>
>>>>>                   Changes since v4:
>>>>>                   - Move the SEV-ES protocol negotiation out of the SEC
>>>>>                   exception handler
>>>>>                       and into the SecMain.c file. As a result:
>>>>>                       - Move the SecGhcb related PCDs out of UefiCpuPkg and
>>>>>                   into OvmfPkg
>>>>>                       - Combine SecAMDSevVcHandler.c and
>>>>>                   PeiDxeAMDSevVcHandler.c into a
>>>>>                         single AMDSevVcHandler.c
>>>>>                   - Consolidate VmgExitLib usage into common LibraryClasses
>>>>>                   sections
>>>>>                   - Add documentation comments to the VmgExitLib functions
>>>>>
>>>>>                   Changes since v3:
>>>>>                   - Remove the need for the MP library finalization routine.
>>>>>                   The AP
>>>>>                       jump table address will be held by the hypervisor
>>>>>                   rather than
>>>>>                       communicated via the GHCB MSR. This removes some
>>>>>                   fragility around
>>>>>                       the UEFI to OS transition.
>>>>>                   - Rename the SEV-ES RIP reset area to SEV-ES workarea and
>>>>>                   use it to
>>>>>                       communicate the SEV-ES status, so that SEC CPU
>>>>>                   exception handling is
>>>>>                       only established for an SEV-ES guest.
>>>>>                   - Fix SMM build breakageAdd around QemuFlashPtrWrite().
>>>>>                   - Fix SMM build breakage by adding VC exception support
>>>>>                   the SMM CPU
>>>>>                       exception handling.
>>>>>                   - Add memory fencing around the invocation of AsmVmgExit().
>>>>>                   - Clarify comments around the SEV-ES AP reset RIP values
>>>>>                   and usage.
>>>>>                   - Move some PCD definitions from MdeModulePkg to UefiCpuPkg.
>>>>>                   - Remove the 16-bit code selector definition from MdeModulePkg
>>>>>
>>>>>                   Changes since v2:
>>>>>                   - Added a way to locate the SEV-ES fixed AP RIP address
>>>>>                   for starting
>>>>>                       AP's to avoid updating the actual flash image (build
>>>>>                   time location
>>>>>                       that is identified with a GUID value).
>>>>>                   - Create a VmgExit library to replace static inline functions.
>>>>>                   - Move some PCDs to the appropriate packages
>>>>>                   - Add support for writing to QEMU flash under SEV-ES
>>>>>                   - Add additional MMIO opcode support
>>>>>                   - Cleaned up the GHCB MSR CPUID protocol support
>>>>>
>>>>>                   Changes since v1:
>>>>>                   - Patches reworked to be more specific to the
>>>>>                   component/area being updated
>>>>>                       and order of definition/usage
>>>>>                   - Created a library for VMGEXIT-related functions to
>>>>>                   replace use of inline
>>>>>                       functions
>>>>>                   - Allocation method for GDT changed from AllocatePool to
>>>>>                   AllocatePages
>>>>>                   - Early caching only enabled for SEV-ES guests
>>>>>                   - Ensure AP loop mode set to halt loop mode for SEV-ES guests
>>>>>                   - Reserved SEC GHCB-related memory areas when S3 is enabled
>>>>>
>>>>>                   Tom Lendacky (43):
>>>>>                       MdeModulePkg: Create PCDs to be used in support of SEV-ES
>>>>>                       UefiCpuPkg: Create PCD to be used in support of SEV-ES
>>>>>                       MdePkg: Add the MSR definition for the GHCB register
>>>>>                       MdePkg: Add a structure definition for the GHCB
>>>>>                       MdeModulePkg/DxeIplPeim: Support GHCB pages when
>>>>>                   creating page tables
>>>>>                       MdePkg/BaseLib: Add support for the XGETBV instruction
>>>>>                       MdePkg/BaseLib: Add support for the VMGEXIT instruction
>>>>>                       UefiCpuPkg: Implement library support for VMGEXIT
>>>>>                       OvmfPkg: Prepare OvmfPkg to use the VmgExitLib library
>>>>>                       UefiPayloadPkg: Prepare UefiPayloadPkg to use the
>>>>>                   VmgExitLib library
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add base support for
>>>>>                   the #VC exception
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for
>>>>>                   IOIO_PROT NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Support string IO for
>>>>>                   IOIO_PROT NAE
>>>>>                         events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for CPUID
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for
>>>>>                   MSR_PROT NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for NPF
>>>>>                   NAE events (MMIO)
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for WBINVD
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for RDTSC
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for RDPMC
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for INVD
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for
>>>>>                   VMMCALL NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for RDTSCP
>>>>>                   NAE events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for
>>>>>                   MONITOR/MONITORX NAE
>>>>>                         events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for
>>>>>                   MWAIT/MWAITX NAE
>>>>>                         events
>>>>>                       UefiCpuPkg/CpuExceptionHandler: Add support for DR7
>>>>>                   Read/Write NAE
>>>>>                         events
>>>>>                       OvmfPkg/MemEncryptSevLib: Add an SEV-ES guest
>>>>>                   indicator function
>>>>>                       OvmfPkg: Add support to perform SEV-ES initialization
>>>>>                       OvmfPkg: Create a GHCB page for use during Sec phase
>>>>>                       OvmfPkg/PlatformPei: Reserve GHCB-related areas if S3
>>>>>                   is supported
>>>>>                       OvmfPkg: Create GHCB pages for use during Pei and Dxe
>>>>>                   phase
>>>>>                       OvmfPkg/PlatformPei: Move early GDT into ram when
>>>>>                   SEV-ES is enabled
>>>>>                       UefiCpuPkg: Create an SEV-ES workarea PCD
>>>>>                       OvmfPkg: Reserve a page in memory for the SEV-ES usage
>>>>>                       OvmfPkg/ResetVector: Add support for a 32-bit SEV check
>>>>>                       OvmfPkg/Sec: Add #VC exception handling for Sec phase
>>>>>                       OvmfPkg/Sec: Enable cache early to speed up booting
>>>>>                       OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash
>>>>>                   detection with
>>>>>                         SEV-ES is enabled
>>>>>                       UefiCpuPkg: Add a 16-bit protected mode code segment
>>>>>                   descriptor
>>>>>                       UefiCpuPkg/MpInitLib: Add CPU MP data flag to indicate
>>>>>                   if SEV-ES is
>>>>>                         enabled
>>>>>                       UefiCpuPkg: Allow AP booting under SEV-ES
>>>>>                       OvmfPkg: Use the SEV-ES work area for the SEV-ES AP
>>>>>                   reset vector
>>>>>                       OvmfPkg: Move the GHCB allocations into reserved memory
>>>>>                       UefiCpuPkg/MpInitLib: Prepare SEV-ES guest APs for OS use
>>>>>
>>>>>                      MdeModulePkg/MdeModulePkg.dec                 |    9 +
>>>>>                      OvmfPkg/OvmfPkg.dec                           |    9 +
>>>>>                      UefiCpuPkg/UefiCpuPkg.dec                     |   17 +
>>>>>                      OvmfPkg/OvmfPkgIa32.dsc                       |    6 +
>>>>>                      OvmfPkg/OvmfPkgIa32X64.dsc                    |    6 +
>>>>>                      OvmfPkg/OvmfPkgX64.dsc                        |    6 +
>>>>>                      OvmfPkg/OvmfXen.dsc                           |    1 +
>>>>>                      UefiCpuPkg/UefiCpuPkg.dsc                     |    2 +
>>>>>                      UefiPayloadPkg/UefiPayloadPkgIa32.dsc         |    2 +
>>>>>                      UefiPayloadPkg/UefiPayloadPkgIa32X64.dsc      |    2 +
>>>>>                      OvmfPkg/OvmfPkgX64.fdf                        |    9 +
>>>>>                      MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf       |    2 +
>>>>>                      MdePkg/Library/BaseLib/BaseLib.inf            |    4 +
>>>>>                      OvmfPkg/PlatformPei/PlatformPei.inf           |    7 +
>>>>>                      .../FvbServicesRuntimeDxe.inf                 |    2 +
>>>>>                      OvmfPkg/ResetVector/ResetVector.inf           |    8 +
>>>>>                      OvmfPkg/Sec/SecMain.inf                       |    4 +
>>>>>                      .../DxeCpuExceptionHandlerLib.inf             |    5 +
>>>>>                      .../PeiCpuExceptionHandlerLib.inf             |    5 +
>>>>>                      .../SecPeiCpuExceptionHandlerLib.inf          |    5 +
>>>>>                      .../SmmCpuExceptionHandlerLib.inf             |    5 +
>>>>>                      UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |    4 +
>>>>>                      UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |    4 +
>>>>>                      UefiCpuPkg/Library/VmgExitLib/VmgExitLib.inf  |   33 +
>>>>>                      .../Core/DxeIplPeim/X64/VirtualMemory.h       |   12 +-
>>>>>                      MdePkg/Include/Library/BaseLib.h              |   31 +
>>>>>                      MdePkg/Include/Register/Amd/Fam17Msr.h        |   42 +
>>>>>                      MdePkg/Include/Register/Amd/Ghcb.h            |  136 ++
>>>>>                      OvmfPkg/Include/Library/MemEncryptSevLib.h    |   12 +
>>>>>                      .../QemuFlash.h                               |   13 +
>>>>>                      UefiCpuPkg/CpuDxe/CpuGdt.h                    |    4 +-
>>>>>                      UefiCpuPkg/Include/Library/VmgExitLib.h       |  117 ++
>>>>>                      .../CpuExceptionHandlerLib/AMDSevVcCommon.h   |   49 +
>>>>>                      .../CpuExceptionCommon.h                      |    2 +
>>>>>                      UefiCpuPkg/Library/MpInitLib/MpLib.h          |   68 +-
>>>>>                      .../Core/DxeIplPeim/Ia32/DxeLoadFunc.c        |    4 +-
>>>>>                      .../Core/DxeIplPeim/X64/DxeLoadFunc.c         |   11 +-
>>>>>                      .../Core/DxeIplPeim/X64/VirtualMemory.c       |   57 +-
>>>>>                      MdePkg/Library/BaseLib/Ia32/GccInline.c       |   45 +
>>>>>                      MdePkg/Library/BaseLib/X64/GccInline.c        |   47 +
>>>>>                      .../MemEncryptSevLibInternal.c                |   75 +-
>>>>>                      OvmfPkg/PlatformPei/AmdSev.c                  |   89 +
>>>>>                      OvmfPkg/PlatformPei/MemDetect.c               |   23 +
>>>>>                      .../QemuFlash.c                               |   23 +-
>>>>>                      .../QemuFlashDxe.c                            |   22 +
>>>>>                      .../QemuFlashSmm.c                            |   16 +
>>>>>                      OvmfPkg/Sec/SecMain.c                         |  188 +-
>>>>>                      UefiCpuPkg/CpuDxe/CpuGdt.c                    |    8 +-
>>>>>                      .../CpuExceptionHandlerLib/AMDSevVcHandler.c  |   40 +
>>>>>                      .../CpuExceptionCommon.c                      |    2 +-
>>>>>                      .../Ia32/ArchAMDSevVcHandler.c                |   38 +
>>>>>                      .../PeiDxeSmmCpuException.c                   |   16 +
>>>>>                      .../SecPeiCpuException.c                      |   16 +
>>>>>                      .../X64/ArchAMDSevVcHandler.c                 | 1699
>>>>>                   +++++++++++++++++
>>>>>                      UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  113 +-
>>>>>                      UefiCpuPkg/Library/MpInitLib/MpLib.c          |  265 ++-
>>>>>                      UefiCpuPkg/Library/MpInitLib/PeiMpLib.c       |   19 +
>>>>>                      UefiCpuPkg/Library/VmgExitLib/VmgExitLib.c    |  293 +++
>>>>>                      UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c  |    2 +-
>>>>>                      MdeModulePkg/MdeModulePkg.uni                 |    8 +
>>>>>                      MdePkg/Library/BaseLib/Ia32/VmgExit.nasm      |   37 +
>>>>>                      MdePkg/Library/BaseLib/Ia32/XGetBv.nasm       |   31 +
>>>>>                      MdePkg/Library/BaseLib/X64/VmgExit.nasm       |   32 +
>>>>>                      MdePkg/Library/BaseLib/X64/XGetBv.nasm        |   34 +
>>>>>                      OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm  |  100 +
>>>>>                      OvmfPkg/ResetVector/Ia32/PageTables64.asm     |  350 +++-
>>>>>                      OvmfPkg/ResetVector/ResetVector.nasmb         |   20 +
>>>>>                      .../X64/ExceptionHandlerAsm.nasm              |   17 +
>>>>>                      UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc   |    2 +-
>>>>>                      .../Library/MpInitLib/Ia32/MpFuncs.nasm       |   15 +
>>>>>                      UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc    |    4 +-
>>>>>                      UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm |  370 +++-
>>>>>                      UefiCpuPkg/Library/VmgExitLib/VmgExitLib.uni  |   15 +
>>>>>                      .../ResetVector/Vtf0/Ia16/Real16ToFlat32.asm  |    9 +
>>>>>                      UefiCpuPkg/UefiCpuPkg.uni                     |   11 +
>>>>>                      75 files changed, 4707 insertions(+), 102 deletions(-)
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/VmgExitLib/VmgExitLib.inf
>>>>>                      create mode 100644 MdePkg/Include/Register/Amd/Ghcb.h
>>>>>                      create mode 100644 UefiCpuPkg/Include/Library/VmgExitLib.h
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/CpuExceptionHandlerLib/AMDSevVcCommon.h
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/CpuExceptionHandlerLib/AMDSevVcHandler.c
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/CpuExceptionHandlerLib/Ia32/ArchAMDSevVcHandler.c
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/ArchAMDSevVcHandler.c
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/VmgExitLib/VmgExitLib.c
>>>>>                      create mode 100644 MdePkg/Library/BaseLib/Ia32/VmgExit.nasm
>>>>>                      create mode 100644 MdePkg/Library/BaseLib/Ia32/XGetBv.nasm
>>>>>                      create mode 100644 MdePkg/Library/BaseLib/X64/VmgExit.nasm
>>>>>                      create mode 100644 MdePkg/Library/BaseLib/X64/XGetBv.nasm
>>>>>                      create mode 100644
>>>>>                   OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
>>>>>                      create mode 100644
>>>>>                   UefiCpuPkg/Library/VmgExitLib/VmgExitLib.uni
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>> 
>