Re: [edk2-devel] [PATCH v4 37/40] UefiCpuPkg: Allow AP booting under SEV-ES

["Lendacky, Thomas" <thomas.lendacky@amd.com>]

On 2/6/20 3:35 AM, Laszlo Ersek wrote:
> On 02/05/20 00:01, Lendacky, Thomas wrote:
>> BZ: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2198&data=02%7C01%7Cthomas.lendacky%40amd.com%7Cf4521a4f24d54b8f122108d7aae7de74%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637165785175188824&sdata=l7ff9Q7OFJ5JYGHt0meU04nBuM9aP1o5RT0fl8s9PoA%3D&reserved=0
>>
>> Typically, an AP is booted using the INIT-SIPI-SIPI sequence. This
>> sequence is intercepted by the hypervisor, which sets the AP's registers
>> to the values requested by the sequence. At that point, the hypervisor can
>> start the AP, which will then begin execution at the appropriate location.
>>
>> Under SEV-ES, AP booting presents some challenges since the hypervisor is
>> not allowed to alter the AP's register state. In this situation, we have
>> to distinguish between the AP's first boot and AP's subsequent boots.
>>
>> First boot:
>>  Once the AP's register state has been defined (which is before the guest
>>  is first booted) it cannot be altered. Should the hypervisor attempt to
>>  alter the register state, the change would be detected by the hardware
>>  and the VMRUN instruction would fail. Given this, the first boot for the
>>  AP is required to begin execution with this initial register state, which
>>  is typically the reset vector. This prevents the BSP from directing the
>>  AP startup location through the INIT-SIPI-SIPI sequence.
>>
>>  To work around this, the firmware will provide a build time reserved area
>>  that can be used as the initial IP value. The hypervisor can extract this
>>  location value by checking for the SEV-ES reset block GUID that must be
>>  located 48-bytes from the end of the firmware. The format of the SEV-ES
>>  reset block area is:
>>
>>    0x00 - 0x01 - SEV-ES Reset IP
>>    0x02 - 0x03 - SEV-ES Reset CS Segment Base[32:16]
> 
> Slight typo here I think: in the next patch, you correctly write [31:16].

Yup, good catch, should be 31.

Thanks,
Tom

> 
> Thanks
> Laszlo
> 
>>    0x04 - 0x05 - Size of the SEV-ES reset block
>>    0x06 - 0x15 - SEV-ES Reset Block GUID
>>                    (00f771de-1a7e-4fcb-890e-68c77e2fb44e)
>>
>>    The total size is 22 bytes. Any expansion to this block must be done
>>    by adding new values before existing values.
>>
>>  The hypervisor will use the IP and CS values obtained from the SEV-ES
>>  reset block to set as the AP's initial values. The CS Segment Base
>>  represents the upper 16 bits of the CS segment base and must be left
>>  shifted by 16 bits to form the complete CS segment base value.
>>
>>  Before booting the AP for the first time, the BSP must initialize the
>>  SEV-ES reset area. This consists of programming a FAR JMP instruction
>>  to the contents of a memory location that is also located in the SEV-ES
>>  reset area. The BSP must program the IP and CS values for the FAR JMP
>>  based on values drived from the INIT-SIPI-SIPI sequence.
>>
>> Subsequent boots:
>>  Again, the hypervisor cannot alter the AP register state, so a method is
>>  required to take the AP out of halt state and redirect it to the desired
>>  IP location. If it is determined that the AP is running in an SEV-ES
>>  guest, then instead of calling CpuSleep(), a VMGEXIT is issued with the
>>  AP Reset Hold exit code (0x80000004). The hypervisor will put the AP in
>>  a halt state, waiting for an INIT-SIPI-SIPI sequence. Once the sequence
>>  is recognized, the hypervisor will resume the AP. At this point the AP
>>  must transition from the current 64-bit long mode down to 16-bit real
>>  mode and begin executing at the derived location from the INIT-SIPI-SIPI
>>  sequence.
>>
>>  Another change is around the area of obtaining the (x2)APIC ID during AP
>>  startup. During AP startup, the AP can't take a #VC exception before the
>>  AP has established a stack. However, the AP stack is set by using the
>>  (x2)APIC ID, which is obtained through CPUID instructions. A CPUID
>>  instruction will cause a #VC, so a different method must be used. The
>>  GHCB protocol supports a method to obtain CPUID information from the
>>  hypervisor through the GHCB MSR. This method does not require a stack,
>>  so it is used to obtain the necessary CPUID information to determine the
>>  (x2)APIC ID.
>>
>> The new 16-bit protected mode GDT entry is used in order to transition
>> from 64-bit long mode down to 16-bit real mode.
>>
>> A new assembler routine is created that takes the AP from 64-bit long mode
>> to 16-bit real mode.  This is located under 1MB in memory and transitions
>> from 64-bit long mode to 32-bit compatibility mode to 16-bit protected
>> mode and finally 16-bit real mode.
>>
>> Cc: Eric Dong <eric.dong@intel.com>
>> Cc: Ray Ni <ray.ni@intel.com>
>> Cc: Laszlo Ersek <lersek@redhat.com>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>>  UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |   3 +
>>  UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |   3 +
>>  UefiCpuPkg/Library/MpInitLib/MpLib.h          |  60 ++++
>>  UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  70 ++++-
>>  UefiCpuPkg/Library/MpInitLib/MpLib.c          | 256 +++++++++++++++++-
>>  UefiCpuPkg/Library/MpInitLib/PeiMpLib.c       |  19 ++
>>  UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c  |   2 +-
>>  UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc   |   2 +-
>>  .../Library/MpInitLib/Ia32/MpFuncs.nasm       |  15 +
>>  UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc    |   4 +-
>>  UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm | 239 ++++++++++++++++
>>  11 files changed, 659 insertions(+), 14 deletions(-)
>>
>> diff --git a/UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf b/UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
>> index 2c26f20c1972..c52951651851 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
>> +++ b/UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
>> @@ -51,6 +51,7 @@ [LibraryClasses]
>>    UefiBootServicesTableLib
>>    DebugAgentLib
>>    SynchronizationLib
>> +  VmgExitLib
>>  
>>  [Protocols]
>>    gEfiTimerArchProtocolGuid                     ## SOMETIMES_CONSUMES
>> @@ -70,5 +71,7 @@ [Pcd]
>>    gUefiCpuPkgTokenSpaceGuid.PcdCpuApTargetCstate                   ## SOMETIMES_CONSUMES
>>    gUefiCpuPkgTokenSpaceGuid.PcdCpuShadowMicrocodeByFit             ## CONSUMES
>>    gUefiCpuPkgTokenSpaceGuid.PcdSevEsIsEnabled                      ## CONSUMES
>> +  gUefiCpuPkgTokenSpaceGuid.PcdSevEsWorkAreaBase                   ## SOMETIMES_CONSUMES
>>    gEfiMdeModulePkgTokenSpaceGuid.PcdCpuStackGuard                  ## CONSUMES
>> +  gEfiMdeModulePkgTokenSpaceGuid.PcdGhcbBase                       ## CONSUMES
>>  
>> diff --git a/UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf b/UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf
>> index 66b2acfe98e7..ff392aeec763 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf
>> +++ b/UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf
>> @@ -50,6 +50,7 @@ [LibraryClasses]
>>    UefiCpuLib
>>    SynchronizationLib
>>    PeiServicesLib
>> +  VmgExitLib
>>  
>>  [Pcd]
>>    gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber        ## CONSUMES
>> @@ -62,6 +63,8 @@ [Pcd]
>>    gUefiCpuPkgTokenSpaceGuid.PcdCpuApTargetCstate                   ## SOMETIMES_CONSUMES
>>    gUefiCpuPkgTokenSpaceGuid.PcdCpuShadowMicrocodeByFit             ## CONSUMES
>>    gUefiCpuPkgTokenSpaceGuid.PcdSevEsIsEnabled                      ## CONSUMES
>> +  gUefiCpuPkgTokenSpaceGuid.PcdSevEsWorkAreaBase                   ## SOMETIMES_CONSUMES
>> +  gEfiMdeModulePkgTokenSpaceGuid.PcdGhcbBase                       ## CONSUMES
>>  
>>  [Guids]
>>    gEdkiiS3SmmInitDoneGuid
>> diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.h b/UefiCpuPkg/Library/MpInitLib/MpLib.h
>> index 864b16872010..63d81ac3e42e 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/MpLib.h
>> +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.h
>> @@ -170,6 +170,11 @@ typedef struct {
>>    UINT8             *RelocateApLoopFuncAddress;
>>    UINTN             RelocateApLoopFuncSize;
>>    UINTN             ModeTransitionOffset;
>> +  UINTN             SwitchToRealSize;
>> +  UINTN             SwitchToRealOffset;
>> +  UINTN             SwitchToRealNoNxOffset;
>> +  UINTN             SwitchToRealPM16ModeOffset;
>> +  UINTN             SwitchToRealPM16ModeSize;
>>  } MP_ASSEMBLY_ADDRESS_MAP;
>>  
>>  typedef struct _CPU_MP_DATA  CPU_MP_DATA;
>> @@ -208,6 +213,8 @@ typedef struct {
>>    // Enable5LevelPaging indicates whether 5-level paging is enabled in long mode.
>>    //
>>    BOOLEAN               Enable5LevelPaging;
>> +  BOOLEAN               SevEsIsEnabled;
>> +  UINTN                 GhcbBase;
>>  } MP_CPU_EXCHANGE_INFO;
>>  
>>  #pragma pack()
>> @@ -256,6 +263,7 @@ struct _CPU_MP_DATA {
>>    UINT8                          ApLoopMode;
>>    UINT8                          ApTargetCState;
>>    UINT16                         PmCodeSegment;
>> +  UINT16                         Pm16CodeSegment;
>>    CPU_AP_DATA                    *CpuData;
>>    volatile MP_CPU_EXCHANGE_INFO  *MpCpuExchangeInfo;
>>  
>> @@ -275,8 +283,47 @@ struct _CPU_MP_DATA {
>>    BOOLEAN                        WakeUpByInitSipiSipi;
>>  
>>    BOOLEAN                        SevEsIsEnabled;
>> +  UINTN                          SevEsAPBuffer;
>> +  UINTN                          SevEsAPResetStackStart;
>> +  CPU_MP_DATA                    *NewCpuMpData;
>> +
>> +  UINT64                         GhcbBase;
>>  };
>>  
>> +#define AP_RESET_STACK_SIZE 64
>> +
>> +#pragma pack(1)
>> +
>> +typedef struct {
>> +  UINT8   InsnBuffer[8];
>> +  UINT16  Rip;
>> +  UINT16  Segment;
>> +} SEV_ES_AP_JMP_FAR;
>> +
>> +#pragma pack()
>> +
>> +/**
>> +  Assembly code to move an AP from long mode to real mode.
>> +
>> +  Move an AP from long mode to real mode in preparation to invoking
>> +  the reset vector.  This is used for SEV-ES guests where a hypervisor
>> +  is not allowed to set the CS and RIP to point to the reset vector.
>> +
>> +  @param[in]  BufferStart  The reset vector target.
>> +  @param[in]  Code16       16-bit protected mode code segment value.
>> +  @param[in]  Code32       32-bit protected mode code segment value.
>> +  @param[in]  StackStart   The start of a stack to be used for transitioning
>> +                           from long mode to real mode.
>> +**/
>> +typedef
>> +VOID
>> +(EFIAPI AP_RESET) (
>> +  IN UINTN    BufferStart,
>> +  IN UINT16   Code16,
>> +  IN UINT16   Code32,
>> +  IN UINTN    StackStart
>> +  );
>> +
>>  extern EFI_GUID mCpuInitMpLibHobGuid;
>>  
>>  /**
>> @@ -382,6 +429,19 @@ GetModeTransitionBuffer (
>>    IN UINTN                BufferSize
>>    );
>>  
>> +/**
>> +  Return the address of the SEV-ES AP jump table.
>> +
>> +  This buffer is required in order for an SEV-ES guest to transition from
>> +  UEFI into an OS.
>> +
>> +  @retval other   Return SEV-ES AP jump table buffer
>> +**/
>> +UINTN
>> +GetSevEsAPMemory (
>> +  VOID
>> +  );
>> +
>>  /**
>>    This function will be called by BSP to wakeup AP.
>>  
>> diff --git a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c b/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c
>> index b17e287bbf49..8df5b6d919e6 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c
>> +++ b/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c
>> @@ -12,6 +12,8 @@
>>  #include <Library/UefiBootServicesTableLib.h>
>>  #include <Library/DebugAgentLib.h>
>>  #include <Library/DxeServicesTableLib.h>
>> +#include <Register/Amd/Fam17Msr.h>
>> +#include <Register/Amd/Ghcb.h>
>>  
>>  #include <Protocol/Timer.h>
>>  
>> @@ -145,6 +147,39 @@ GetModeTransitionBuffer (
>>    return (UINTN)StartAddress;
>>  }
>>  
>> +/**
>> +  Return the address of the SEV-ES AP jump table.
>> +
>> +  This buffer is required in order for an SEV-ES guest to transition from
>> +  UEFI into an OS.
>> +
>> +  @retval other   Return SEV-ES AP jump table buffer
>> +**/
>> +UINTN
>> +GetSevEsAPMemory (
>> +  VOID
>> +  )
>> +{
>> +  EFI_STATUS            Status;
>> +  EFI_PHYSICAL_ADDRESS  StartAddress;
>> +
>> +  //
>> +  // Allocate 1 page for AP jump table page
>> +  //
>> +  StartAddress = BASE_4GB - 1;
>> +  Status = gBS->AllocatePages (
>> +                  AllocateMaxAddress,
>> +                  EfiReservedMemoryType,
>> +                  1,
>> +                  &StartAddress
>> +                  );
>> +  ASSERT_EFI_ERROR (Status);
>> +
>> +  DEBUG ((DEBUG_INFO, "Dxe: SevEsAPMemory = %lx\n", (UINTN) StartAddress));
>> +
>> +  return (UINTN) StartAddress;
>> +}
>> +
>>  /**
>>    Checks APs status and updates APs status if needed.
>>  
>> @@ -219,6 +254,38 @@ CheckApsStatus (
>>    }
>>  }
>>  
>> +/**
>> +  Get Protected mode code segment with 16-bit default addressing
>> +  from current GDT table.
>> +
>> +  @return  Protected mode 16-bit code segment value.
>> +**/
>> +UINT16
>> +GetProtectedMode16CS (
>> +  VOID
>> +  )
>> +{
>> +  IA32_DESCRIPTOR          GdtrDesc;
>> +  IA32_SEGMENT_DESCRIPTOR  *GdtEntry;
>> +  UINTN                    GdtEntryCount;
>> +  UINT16                   Index;
>> +
>> +  Index = (UINT16) -1;
>> +  AsmReadGdtr (&GdtrDesc);
>> +  GdtEntryCount = (GdtrDesc.Limit + 1) / sizeof (IA32_SEGMENT_DESCRIPTOR);
>> +  GdtEntry = (IA32_SEGMENT_DESCRIPTOR *) GdtrDesc.Base;
>> +  for (Index = 0; Index < GdtEntryCount; Index++) {
>> +    if (GdtEntry->Bits.L == 0) {
>> +      if (GdtEntry->Bits.Type > 8 && GdtEntry->Bits.DB == 0) {
>> +        break;
>> +      }
>> +    }
>> +    GdtEntry++;
>> +  }
>> +  ASSERT (Index != GdtEntryCount);
>> +  return Index * 8;
>> +}
>> +
>>  /**
>>    Get Protected mode code segment from current GDT table.
>>  
>> @@ -239,7 +306,7 @@ GetProtectedModeCS (
>>    GdtEntry = (IA32_SEGMENT_DESCRIPTOR *) GdtrDesc.Base;
>>    for (Index = 0; Index < GdtEntryCount; Index++) {
>>      if (GdtEntry->Bits.L == 0) {
>> -      if (GdtEntry->Bits.Type > 8 && GdtEntry->Bits.L == 0) {
>> +      if (GdtEntry->Bits.Type > 8 && GdtEntry->Bits.DB == 1) {
>>          break;
>>        }
>>      }
>> @@ -301,6 +368,7 @@ MpInitChangeApLoopCallback (
>>  
>>    CpuMpData = GetCpuMpData ();
>>    CpuMpData->PmCodeSegment = GetProtectedModeCS ();
>> +  CpuMpData->Pm16CodeSegment = GetProtectedMode16CS ();
>>    CpuMpData->ApLoopMode = PcdGet8 (PcdCpuApLoopMode);
>>    mNumberToFinish = CpuMpData->CpuCount - 1;
>>    WakeUpAP (CpuMpData, TRUE, 0, RelocateApLoop, NULL, TRUE);
>> diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.c b/UefiCpuPkg/Library/MpInitLib/MpLib.c
>> index 5e3183c2493b..ca8a3a3a7be9 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/MpLib.c
>> +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.c
>> @@ -7,6 +7,9 @@
>>  **/
>>  
>>  #include "MpLib.h"
>> +#include <Library/VmgExitLib.h>
>> +#include <Register/Amd/Fam17Msr.h>
>> +#include <Register/Amd/Ghcb.h>
>>  
>>  EFI_GUID mCpuInitMpLibHobGuid = CPU_INIT_MP_LIB_HOB_GUID;
>>  
>> @@ -288,6 +291,14 @@ GetApLoopMode (
>>        //
>>        ApLoopMode = ApInHltLoop;
>>      }
>> +
>> +    if (PcdGetBool (PcdSevEsIsEnabled)) {
>> +      //
>> +      // For SEV-ES, force AP in Hlt-loop mode in order to use the GHCB
>> +      // protocol for starting APs
>> +      //
>> +      ApLoopMode = ApInHltLoop;
>> +    }
>>    }
>>  
>>    if (ApLoopMode != ApInMwaitLoop) {
>> @@ -579,6 +590,108 @@ InitializeApData (
>>    SetApState (&CpuMpData->CpuData[ProcessorNumber], CpuStateIdle);
>>  }
>>  
>> +/**
>> +  Get Protected mode code segment with 16-bit default addressing
>> +  from current GDT table.
>> +
>> +  @return  Protected mode 16-bit code segment value.
>> +**/
>> +STATIC
>> +UINT16
>> +GetProtectedMode16CS (
>> +  VOID
>> +  )
>> +{
>> +  IA32_DESCRIPTOR          GdtrDesc;
>> +  IA32_SEGMENT_DESCRIPTOR  *GdtEntry;
>> +  UINTN                    GdtEntryCount;
>> +  UINT16                   Index;
>> +
>> +  Index = (UINT16) -1;
>> +  AsmReadGdtr (&GdtrDesc);
>> +  GdtEntryCount = (GdtrDesc.Limit + 1) / sizeof (IA32_SEGMENT_DESCRIPTOR);
>> +  GdtEntry = (IA32_SEGMENT_DESCRIPTOR *) GdtrDesc.Base;
>> +  for (Index = 0; Index < GdtEntryCount; Index++) {
>> +    if (GdtEntry->Bits.L == 0 &&
>> +        GdtEntry->Bits.DB == 0 &&
>> +        GdtEntry->Bits.Type > 8) {
>> +      break;
>> +    }
>> +    GdtEntry++;
>> +  }
>> +  ASSERT (Index != GdtEntryCount);
>> +  return Index * 8;
>> +}
>> +
>> +/**
>> +  Get Protected mode code segment with 32-bit default addressing
>> +  from current GDT table.
>> +
>> +  @return  Protected mode 32-bit code segment value.
>> +**/
>> +STATIC
>> +UINT16
>> +GetProtectedMode32CS (
>> +  VOID
>> +  )
>> +{
>> +  IA32_DESCRIPTOR          GdtrDesc;
>> +  IA32_SEGMENT_DESCRIPTOR  *GdtEntry;
>> +  UINTN                    GdtEntryCount;
>> +  UINT16                   Index;
>> +
>> +  Index = (UINT16) -1;
>> +  AsmReadGdtr (&GdtrDesc);
>> +  GdtEntryCount = (GdtrDesc.Limit + 1) / sizeof (IA32_SEGMENT_DESCRIPTOR);
>> +  GdtEntry = (IA32_SEGMENT_DESCRIPTOR *) GdtrDesc.Base;
>> +  for (Index = 0; Index < GdtEntryCount; Index++) {
>> +    if (GdtEntry->Bits.L == 0 &&
>> +        GdtEntry->Bits.DB == 1 &&
>> +        GdtEntry->Bits.Type > 8) {
>> +      break;
>> +    }
>> +    GdtEntry++;
>> +  }
>> +  ASSERT (Index != GdtEntryCount);
>> +  return Index * 8;
>> +}
>> +
>> +/**
>> +  Reset an AP when in SEV-ES mode.
>> +
>> +  @retval EFI_DEVICE_ERROR        Reset of AP failed.
>> +**/
>> +STATIC
>> +VOID
>> +MpInitLibSevEsAPReset (
>> +  GHCB                         *Ghcb,
>> +  CPU_MP_DATA                  *CpuMpData
>> +  )
>> +{
>> +  UINT16           Code16, Code32;
>> +  AP_RESET         *APResetFn;
>> +  UINTN            BufferStart;
>> +  UINTN            StackStart;
>> +
>> +  Code16 = GetProtectedMode16CS ();
>> +  Code32 = GetProtectedMode32CS ();
>> +
>> +  if (CpuMpData->WakeupBufferHigh != 0) {
>> +    APResetFn = (AP_RESET *) (CpuMpData->WakeupBufferHigh + CpuMpData->AddressMap.SwitchToRealNoNxOffset);
>> +  } else {
>> +    APResetFn = (AP_RESET *) (CpuMpData->MpCpuExchangeInfo->BufferStart + CpuMpData->AddressMap.SwitchToRealOffset);
>> +  }
>> +
>> +  BufferStart = CpuMpData->MpCpuExchangeInfo->BufferStart;
>> +  StackStart = CpuMpData->SevEsAPResetStackStart -
>> +                 (AP_RESET_STACK_SIZE * GetApicId ());
>> +
>> +  //
>> +  // This call never returns.
>> +  //
>> +  APResetFn (BufferStart, Code16, Code32, StackStart);
>> +}
>> +
>>  /**
>>    This function will be called from AP reset code if BSP uses WakeUpAP.
>>  
>> @@ -734,7 +847,28 @@ ApWakeupFunction (
>>        //
>>        while (TRUE) {
>>          DisableInterrupts ();
>> -        CpuSleep ();
>> +        if (CpuMpData->SevEsIsEnabled) {
>> +          MSR_SEV_ES_GHCB_REGISTER  Msr;
>> +          GHCB                      *Ghcb;
>> +
>> +          Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
>> +          Ghcb = Msr.Ghcb;
>> +
>> +          VmgInit (Ghcb);
>> +          VmgExit (Ghcb, SvmExitApResetHold, 0, 0);
>> +          /*TODO: Check return value to verify SIPI issued */
>> +
>> +          //
>> +          // Awakened in a new phase? Use the new CpuMpData
>> +          //
>> +          if (CpuMpData->NewCpuMpData) {
>> +            CpuMpData = CpuMpData->NewCpuMpData;
>> +          }
>> +
>> +          MpInitLibSevEsAPReset (Ghcb, CpuMpData);
>> +        } else {
>> +          CpuSleep ();
>> +        }
>>          CpuPause ();
>>        }
>>      }
>> @@ -847,6 +981,9 @@ FillExchangeInfoData (
>>    ExchangeInfo->Enable5LevelPaging = (BOOLEAN) (Cr4.Bits.LA57 == 1);
>>    DEBUG ((DEBUG_INFO, "%a: 5-Level Paging = %d\n", gEfiCallerBaseName, ExchangeInfo->Enable5LevelPaging));
>>  
>> +  ExchangeInfo->SevEsIsEnabled  = CpuMpData->SevEsIsEnabled;
>> +  ExchangeInfo->GhcbBase        = CpuMpData->GhcbBase;
>> +
>>    //
>>    // Get the BSP's data of GDT and IDT
>>    //
>> @@ -873,8 +1010,9 @@ FillExchangeInfoData (
>>    // EfiBootServicesCode to avoid page fault if NX memory protection is enabled.
>>    //
>>    if (CpuMpData->WakeupBufferHigh != 0) {
>> -    Size = CpuMpData->AddressMap.RendezvousFunnelSize -
>> -           CpuMpData->AddressMap.ModeTransitionOffset;
>> +    Size = CpuMpData->AddressMap.RendezvousFunnelSize +
>> +             CpuMpData->AddressMap.SwitchToRealSize -
>> +             CpuMpData->AddressMap.ModeTransitionOffset;
>>      CopyMem (
>>        (VOID *)CpuMpData->WakeupBufferHigh,
>>        CpuMpData->AddressMap.RendezvousFunnelAddress +
>> @@ -927,7 +1065,8 @@ BackupAndPrepareWakeupBuffer(
>>    CopyMem (
>>      (VOID *) CpuMpData->WakeupBuffer,
>>      (VOID *) CpuMpData->AddressMap.RendezvousFunnelAddress,
>> -    CpuMpData->AddressMap.RendezvousFunnelSize
>> +    CpuMpData->AddressMap.RendezvousFunnelSize +
>> +      CpuMpData->AddressMap.SwitchToRealSize
>>      );
>>  }
>>  
>> @@ -948,6 +1087,40 @@ RestoreWakeupBuffer(
>>      );
>>  }
>>  
>> +/**
>> +  Calculate the size of the reset stack.
>> +**/
>> +STATIC
>> +UINTN
>> +GetApResetStackSize(
>> +  VOID
>> +  )
>> +{
>> +  return AP_RESET_STACK_SIZE * PcdGet32(PcdCpuMaxLogicalProcessorNumber);
>> +}
>> +
>> +/**
>> +  Calculate the size of the reset vector.
>> +
>> +  @param[in]  AddressMap  The pointer to Address Map structure.
>> +**/
>> +STATIC
>> +UINTN
>> +GetApResetVectorSize(
>> +  IN MP_ASSEMBLY_ADDRESS_MAP  *AddressMap
>> +  )
>> +{
>> +  UINTN  Size;
>> +
>> +  Size = ALIGN_VALUE (AddressMap->RendezvousFunnelSize +
>> +                        AddressMap->SwitchToRealSize +
>> +                        sizeof (MP_CPU_EXCHANGE_INFO),
>> +                      CPU_STACK_ALIGNMENT);
>> +  Size += GetApResetStackSize ();
>> +
>> +  return Size;
>> +}
>> +
>>  /**
>>    Allocate reset vector buffer.
>>  
>> @@ -961,16 +1134,22 @@ AllocateResetVector (
>>    UINTN           ApResetVectorSize;
>>  
>>    if (CpuMpData->WakeupBuffer == (UINTN) -1) {
>> -    ApResetVectorSize = CpuMpData->AddressMap.RendezvousFunnelSize +
>> -                          sizeof (MP_CPU_EXCHANGE_INFO);
>> +    ApResetVectorSize = GetApResetVectorSize (&CpuMpData->AddressMap);
>>  
>>      CpuMpData->WakeupBuffer      = GetWakeupBuffer (ApResetVectorSize);
>>      CpuMpData->MpCpuExchangeInfo = (MP_CPU_EXCHANGE_INFO *) (UINTN)
>> -                    (CpuMpData->WakeupBuffer + CpuMpData->AddressMap.RendezvousFunnelSize);
>> +                    (CpuMpData->WakeupBuffer +
>> +                       CpuMpData->AddressMap.RendezvousFunnelSize +
>> +                       CpuMpData->AddressMap.SwitchToRealSize);
>>      CpuMpData->WakeupBufferHigh  = GetModeTransitionBuffer (
>> -                                    CpuMpData->AddressMap.RendezvousFunnelSize -
>> +                                    CpuMpData->AddressMap.RendezvousFunnelSize +
>> +                                    CpuMpData->AddressMap.SwitchToRealSize -
>>                                      CpuMpData->AddressMap.ModeTransitionOffset
>>                                      );
>> +    //
>> +    // The reset stack starts at the end of the buffer.
>> +    //
>> +    CpuMpData->SevEsAPResetStackStart = CpuMpData->WakeupBuffer + ApResetVectorSize;
>>    }
>>    BackupAndPrepareWakeupBuffer (CpuMpData);
>>  }
>> @@ -985,7 +1164,31 @@ FreeResetVector (
>>    IN CPU_MP_DATA              *CpuMpData
>>    )
>>  {
>> -  RestoreWakeupBuffer (CpuMpData);
>> +  //
>> +  // If SEV-ES is enabled, the reset area is needed for AP parking and
>> +  // and AP startup in the OS, so the reset area is reserved. Do not
>> +  // perform the restore as this will overwrite memory which has data
>> +  // needed by SEV-ES.
>> +  //
>> +  if (!CpuMpData->SevEsIsEnabled) {
>> +    RestoreWakeupBuffer (CpuMpData);
>> +  }
>> +}
>> +
>> +/**
>> +  Allocate the SEV-ES AP jump table buffer.
>> +
>> +  @param[in, out]  CpuMpData  The pointer to CPU MP Data structure.
>> +**/
>> +VOID
>> +AllocateSevEsAPMemory (
>> +  IN OUT CPU_MP_DATA          *CpuMpData
>> +  )
>> +{
>> +  if (CpuMpData->SevEsAPBuffer == (UINTN) -1) {
>> +    CpuMpData->SevEsAPBuffer =
>> +      CpuMpData->SevEsIsEnabled ? GetSevEsAPMemory () : 0;
>> +  }
>>  }
>>  
>>  /**
>> @@ -1022,6 +1225,7 @@ WakeUpAP (
>>        CpuMpData->InitFlag   != ApInitDone) {
>>      ResetVectorRequired = TRUE;
>>      AllocateResetVector (CpuMpData);
>> +    AllocateSevEsAPMemory (CpuMpData);
>>      FillExchangeInfoData (CpuMpData);
>>      SaveLocalApicTimerSetting (CpuMpData);
>>    }
>> @@ -1058,6 +1262,35 @@ WakeUpAP (
>>        }
>>      }
>>      if (ResetVectorRequired) {
>> +      //
>> +      // For SEV-ES, the initial AP boot address will be defined by
>> +      // PcdSevEsWorkAreaBase. The Segment/Rip must be the jump address
>> +      // from the original INIT-SIPI-SIPI.
>> +      //
>> +      if (CpuMpData->SevEsIsEnabled) {
>> +        SEV_ES_AP_JMP_FAR *JmpFar;
>> +        UINT32            Offset, InsnByte;
>> +        UINT8             LoNib, HiNib;
>> +
>> +        JmpFar = (SEV_ES_AP_JMP_FAR *) FixedPcdGet32 (PcdSevEsWorkAreaBase);
>> +        ASSERT (JmpFar != NULL);
>> +
>> +        Offset = FixedPcdGet32 (PcdSevEsWorkAreaBase);
>> +        Offset += sizeof(JmpFar->InsnBuffer);
>> +        LoNib = (UINT8) Offset;
>> +        HiNib = (UINT8) (Offset >> 8);
>> +
>> +        // JMP FAR [CS:XXYY] => 2E FF 2E YY XX
>> +        InsnByte = 0;
>> +        JmpFar->InsnBuffer[InsnByte++] = 0x2E;  // CS override prefix
>> +        JmpFar->InsnBuffer[InsnByte++] = 0xFF;  // JMP (FAR)
>> +        JmpFar->InsnBuffer[InsnByte++] = 0x2E;  // ModRM (JMP memory location)
>> +        JmpFar->InsnBuffer[InsnByte++] = LoNib; // YY offset ...
>> +        JmpFar->InsnBuffer[InsnByte++] = HiNib; // XX offset ...
>> +
>> +        JmpFar->Rip = 0;
>> +        JmpFar->Segment = (UINT16) (ExchangeInfo->BufferStart >> 4);
>> +      }
>>        //
>>        // Wakeup all APs
>>        //
>> @@ -1625,7 +1858,7 @@ MpInitLibInitialize (
>>    ASSERT (MaxLogicalProcessorNumber != 0);
>>  
>>    AsmGetAddressMap (&AddressMap);
>> -  ApResetVectorSize = AddressMap.RendezvousFunnelSize + sizeof (MP_CPU_EXCHANGE_INFO);
>> +  ApResetVectorSize = GetApResetVectorSize (&AddressMap);
>>    ApStackSize = PcdGet32(PcdCpuApStackSize);
>>    ApLoopMode  = GetApLoopMode (&MonitorFilterSize);
>>  
>> @@ -1688,6 +1921,8 @@ MpInitLibInitialize (
>>    }
>>    InitializeSpinLock(&CpuMpData->MpLock);
>>    CpuMpData->SevEsIsEnabled = PcdGetBool (PcdSevEsIsEnabled);
>> +  CpuMpData->SevEsAPBuffer  = (UINTN) -1;
>> +  CpuMpData->GhcbBase       = PcdGet64 (PcdGhcbBase);
>>  
>>    //
>>    // Make sure no memory usage outside of the allocated buffer.
>> @@ -1751,6 +1986,7 @@ MpInitLibInitialize (
>>      // APs have been wakeup before, just get the CPU Information
>>      // from HOB
>>      //
>> +    OldCpuMpData->NewCpuMpData = CpuMpData;
>>      CpuMpData->CpuCount  = OldCpuMpData->CpuCount;
>>      CpuMpData->BspNumber = OldCpuMpData->BspNumber;
>>      CpuMpData->CpuInfoInHob = OldCpuMpData->CpuInfoInHob;
>> diff --git a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c b/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c
>> index 06e3f5d0d3da..e8103a9ce094 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c
>> +++ b/UefiCpuPkg/Library/MpInitLib/PeiMpLib.c
>> @@ -280,6 +280,25 @@ GetModeTransitionBuffer (
>>    return 0;
>>  }
>>  
>> +/**
>> +  Return the address of the SEV-ES AP jump table.
>> +
>> +  This buffer is required in order for an SEV-ES guest to transition from
>> +  UEFI into an OS.
>> +
>> +  @retval other   Return SEV-ES AP jump table buffer
>> +**/
>> +UINTN
>> +GetSevEsAPMemory (
>> +  VOID
>> +  )
>> +{
>> +  //
>> +  // PEI phase doesn't need to do such transition. So simply return 0.
>> +  //
>> +  return 0;
>> +}
>> +
>>  /**
>>    Checks APs status and updates APs status if needed.
>>  
>> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c b/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
>> index 6298571e29b2..28f8e8e133e5 100644
>> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
>> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
>> @@ -121,7 +121,7 @@ GetProtectedModeCS (
>>    GdtEntry = (IA32_SEGMENT_DESCRIPTOR *) GdtrDesc.Base;
>>    for (Index = 0; Index < GdtEntryCount; Index++) {
>>      if (GdtEntry->Bits.L == 0) {
>> -      if (GdtEntry->Bits.Type > 8 && GdtEntry->Bits.L == 0) {
>> +      if (GdtEntry->Bits.Type > 8 && GdtEntry->Bits.DB == 1) {
>>          break;
>>        }
>>      }
>> diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc b/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc
>> index efb1bc2bf7cb..4f5a7c859a56 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc
>> +++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpEqu.inc
>> @@ -19,7 +19,7 @@ CPU_SWITCH_STATE_IDLE         equ        0
>>  CPU_SWITCH_STATE_STORED       equ        1
>>  CPU_SWITCH_STATE_LOADED       equ        2
>>  
>> -LockLocation                  equ        (RendezvousFunnelProcEnd - RendezvousFunnelProcStart)
>> +LockLocation                  equ        (SwitchToRealProcEnd - RendezvousFunnelProcStart)
>>  StackStartAddressLocation     equ        LockLocation + 04h
>>  StackSizeLocation             equ        LockLocation + 08h
>>  ApProcedureLocation           equ        LockLocation + 0Ch
>> diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
>> index b74046b76af3..309d53bf3b37 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
>> +++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
>> @@ -215,6 +215,16 @@ CProcedureInvoke:
>>      jmp        $                 ; Never reach here
>>  RendezvousFunnelProcEnd:
>>  
>> +;-------------------------------------------------------------------------------------
>> +;SwitchToRealProc procedure follows.
>> +;NOT USED IN 32 BIT MODE.
>> +;-------------------------------------------------------------------------------------
>> +global ASM_PFX(SwitchToRealProc)
>> +ASM_PFX(SwitchToRealProc):
>> +SwitchToRealProcStart:
>> +    jmp        $                 ; Never reach here
>> +SwitchToRealProcEnd:
>> +
>>  ;-------------------------------------------------------------------------------------
>>  ;  AsmRelocateApLoop (MwaitSupport, ApTargetCState, PmCodeSegment, TopOfApStack, CountTofinish);
>>  ;-------------------------------------------------------------------------------------
>> @@ -263,6 +273,11 @@ ASM_PFX(AsmGetAddressMap):
>>      mov        dword [ebx + 0Ch], AsmRelocateApLoopStart
>>      mov        dword [ebx + 10h], AsmRelocateApLoopEnd - AsmRelocateApLoopStart
>>      mov        dword [ebx + 14h], Flat32Start - RendezvousFunnelProcStart
>> +    mov        dword [ebx + 18h], SwitchToRealProcEnd - SwitchToRealProcStart       ; SwitchToRealSize
>> +    mov        dword [ebx + 1Ch], SwitchToRealProcStart - RendezvousFunnelProcStart ; SwitchToRealOffset
>> +    mov        dword [ebx + 20h], SwitchToRealProcStart - Flat32Start               ; SwitchToRealNoNxOffset
>> +    mov        dword [ebx + 24h], 0                                                 ; SwitchToRealPM16ModeOffset
>> +    mov        dword [ebx + 28h], 0                                                 ; SwitchToRealPM16ModeSize
>>  
>>      popad
>>      ret
>> diff --git a/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc b/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc
>> index 58ef369342a7..c92daaaffd6b 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc
>> +++ b/UefiCpuPkg/Library/MpInitLib/X64/MpEqu.inc
>> @@ -19,7 +19,7 @@ CPU_SWITCH_STATE_IDLE         equ        0
>>  CPU_SWITCH_STATE_STORED       equ        1
>>  CPU_SWITCH_STATE_LOADED       equ        2
>>  
>> -LockLocation                  equ        (RendezvousFunnelProcEnd - RendezvousFunnelProcStart)
>> +LockLocation                  equ        (SwitchToRealProcEnd - RendezvousFunnelProcStart)
>>  StackStartAddressLocation     equ        LockLocation + 08h
>>  StackSizeLocation             equ        LockLocation + 10h
>>  ApProcedureLocation           equ        LockLocation + 18h
>> @@ -41,3 +41,5 @@ ModeTransitionSegmentLocation       equ  LockLocation + 98h
>>  ModeHighMemoryLocation              equ  LockLocation + 9Ah
>>  ModeHighSegmentLocation             equ  LockLocation + 9Eh
>>  Enable5LevelPagingLocation          equ  LockLocation + 0A0h
>> +SevEsIsEnabledLocation              equ  LockLocation + 0A1h
>> +GhcbBaseLocation                    equ  LockLocation + 0A2h
>> diff --git a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm b/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm
>> index 87f2523e856f..6956b408d004 100644
>> --- a/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm
>> +++ b/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm
>> @@ -184,9 +184,97 @@ Releaselock:
>>      add        edi, StackStartAddressLocation
>>      add        rax, qword [edi]
>>      mov        rsp, rax
>> +
>> +    lea        edi, [esi + SevEsIsEnabledLocation]
>> +    cmp        byte [edi], 1        ; SevEsIsEnabled
>> +    jne        CProcedureInvoke
>> +
>> +    ;
>> +    ; program GHCB
>> +    ;   Each page after the GHCB is a per-CPU page, so the calculation programs
>> +    ;   a GHCB to be every 8KB.
>> +    ;
>> +    mov        eax, SIZE_4KB
>> +    shl        eax, 1                            ; EAX = SIZE_4K * 2
>> +    mov        ecx, ebx
>> +    mul        ecx                               ; EAX = SIZE_4K * 2 * CpuNumber
>> +    mov        edi, esi
>> +    add        edi, GhcbBaseLocation
>> +    add        rax, qword [edi]
>> +    mov        rdx, rax
>> +    shr        rdx, 32
>> +    mov        rcx, 0xc0010130
>> +    wrmsr
>>      jmp        CProcedureInvoke
>>  
>>  GetApicId:
>> +    lea        edi, [esi + SevEsIsEnabledLocation]
>> +    cmp        byte [edi], 1        ; SevEsIsEnabled
>> +    jne        DoCpuid
>> +
>> +    ;
>> +    ; Since we don't have a stack yet, we can't take a #VC
>> +    ; exception. Use the GHCB protocol to perform the CPUID
>> +    ; calls.
>> +    ;
>> +    mov        rcx, 0xc0010130
>> +    rdmsr
>> +    shl        rdx, 32
>> +    or         rax, rdx
>> +    mov        rdi, rax             ; RDI now holds the original GHCB GPA
>> +
>> +    mov        rdx, 0               ; CPUID function 0
>> +    mov        rax, 0               ; RAX register requested
>> +    or         rax, 4
>> +    wrmsr
>> +    rep vmmcall
>> +    rdmsr
>> +    cmp        edx, 0bh
>> +    jb         NoX2ApicSevEs        ; CPUID level below CPUID_EXTENDED_TOPOLOGY
>> +
>> +    mov        rdx, 0bh             ; CPUID function 0x0b
>> +    mov        rax, 040000000h      ; RBX register requested
>> +    or         rax, 4
>> +    wrmsr
>> +    rep vmmcall
>> +    rdmsr
>> +    test       edx, 0ffffh
>> +    jz         NoX2ApicSevEs        ; CPUID.0BH:EBX[15:0] is zero
>> +
>> +    mov        rdx, 0bh             ; CPUID function 0x0b
>> +    mov        rax, 0c0000000h      ; RDX register requested
>> +    or         rax, 4
>> +    wrmsr
>> +    rep vmmcall
>> +    rdmsr
>> +
>> +    ; Processor is x2APIC capable; 32-bit x2APIC ID is now in EDX
>> +    jmp        RestoreGhcb
>> +
>> +NoX2ApicSevEs:
>> +    ; Processor is not x2APIC capable, so get 8-bit APIC ID
>> +    mov        rdx, 1               ; CPUID function 1
>> +    mov        rax, 040000000h      ; RBX register requested
>> +    or         rax, 4
>> +    wrmsr
>> +    rep vmmcall
>> +    rdmsr
>> +    shr        edx, 24
>> +
>> +RestoreGhcb:
>> +    mov        rbx, rdx             ; Save x2APIC/APIC ID
>> +
>> +    mov        rdx, rdi             ; RDI holds the saved GHCB GPA
>> +    shr        rdx, 32
>> +    mov        eax, edi
>> +    wrmsr
>> +
>> +    mov        rdx, rbx
>> +
>> +    ; x2APIC ID or APIC ID is in EDX
>> +    jmp        GetProcessorNumber
>> +
>> +DoCpuid:
>>      mov        eax, 0
>>      cpuid
>>      cmp        eax, 0bh
>> @@ -253,12 +341,158 @@ CProcedureInvoke:
>>  
>>  RendezvousFunnelProcEnd:
>>  
>> +;-------------------------------------------------------------------------------------
>> +;SwitchToRealProc procedure follows.
>> +;ALSO THIS PROCEDURE IS EXECUTED BY APs TRANSITIONING TO 16 BIT MODE. HENCE THIS PROC
>> +;IS IN MACHINE CODE.
>> +;  SwitchToRealProc (UINTN BufferStart, UINT16 Code16, UINT16 Code32, UINTN StackStart)
>> +;  rcx - Buffer Start
>> +;  rdx - Code16 Selector Offset
>> +;  r8  - Code32 Selector Offset
>> +;  r9  - Stack Start
>> +;-------------------------------------------------------------------------------------
>> +global ASM_PFX(SwitchToRealProc)
>> +ASM_PFX(SwitchToRealProc):
>> +SwitchToRealProcStart:
>> +BITS 64
>> +    cli
>> +
>> +    ;
>> +    ; Get RDX reset value before changing stacks since the
>> +    ; new stack won't be able to accomodate a #VC exception.
>> +    ;
>> +    push       rax
>> +    push       rbx
>> +    push       rcx
>> +    push       rdx
>> +
>> +    mov        rax, 1
>> +    cpuid
>> +    mov        rsi, rax                    ; Save off the reset value for RDX
>> +
>> +    pop        rdx
>> +    pop        rcx
>> +    pop        rbx
>> +    pop        rax
>> +
>> +    ;
>> +    ; Establish stack below 1MB
>> +    ;
>> +    mov        rsp, r9
>> +
>> +    ;
>> +    ; Push ultimate Reset Vector onto the stack
>> +    ;
>> +    mov        rax, rcx
>> +    shr        rax, 4
>> +    push       word 0x0002                 ; RFLAGS
>> +    push       ax                          ; CS
>> +    push       word 0x0000                 ; RIP
>> +    push       word 0x0000                 ; For alignment, will be discarded
>> +
>> +    ;
>> +    ; Get address of "16-bit operand size" label
>> +    ;
>> +    lea        rbx, [PM16Mode]
>> +
>> +    ;
>> +    ; Push addresses used to change to compatibility mode
>> +    ;
>> +    lea        rax, [CompatMode]
>> +    push       r8
>> +    push       rax
>> +
>> +    ;
>> +    ; Clear R8 - R15, for reset, before going into 32-bit mode
>> +    ;
>> +    xor        r8, r8
>> +    xor        r9, r9
>> +    xor        r10, r10
>> +    xor        r11, r11
>> +    xor        r12, r12
>> +    xor        r13, r13
>> +    xor        r14, r14
>> +    xor        r15, r15
>> +
>> +    ;
>> +    ; Far return into 32-bit mode
>> +    ;
>> +o64 retf
>> +
>> +BITS 32
>> +CompatMode:
>> +    ;
>> +    ; Set up stack to prepare for exiting protected mode
>> +    ;
>> +    push       edx                         ; Code16 CS
>> +    push       ebx                         ; PM16Mode label address
>> +
>> +    ;
>> +    ; Disable paging
>> +    ;
>> +    mov        eax, cr0                    ; Read CR0
>> +    btr        eax, 31                     ; Set PG=0
>> +    mov        cr0, eax                    ; Write CR0
>> +
>> +    ;
>> +    ; Disable long mode
>> +    ;
>> +    mov        ecx, 0c0000080h             ; EFER MSR number
>> +    rdmsr                                  ; Read EFER
>> +    btr        eax, 8                      ; Set LME=0
>> +    wrmsr                                  ; Write EFER
>> +
>> +    ;
>> +    ; Disable PAE
>> +    ;
>> +    mov        eax, cr4                    ; Read CR4
>> +    btr        eax, 5                      ; Set PAE=0
>> +    mov        cr4, eax                    ; Write CR4
>> +
>> +    mov        edx, esi                    ; Restore RDX reset value
>> +
>> +    ;
>> +    ; Switch to 16-bit operand size
>> +    ;
>> +    retf
>> +
>> +BITS 16
>> +    ;
>> +    ; At entry to this label
>> +    ;   - RDX will have its reset value
>> +    ;   - On the top of the stack
>> +    ;     - Alignment data (two bytes) to be discarded
>> +    ;     - IP for Real Mode (two bytes)
>> +    ;     - CS for Real Mode (two bytes)
>> +    ;
>> +PM16Mode:
>> +    mov        eax, cr0                    ; Read CR0
>> +    btr        eax, 0                      ; Set PE=0
>> +    mov        cr0, eax                    ; Write CR0
>> +
>> +    pop        ax                          ; Discard alignment data
>> +
>> +    ;
>> +    ; Clear registers (except RDX and RSP) before going into 16-bit mode
>> +    ;
>> +    xor        eax, eax
>> +    xor        ebx, ebx
>> +    xor        ecx, ecx
>> +    xor        esi, esi
>> +    xor        edi, edi
>> +    xor        ebp, ebp
>> +
>> +    iret
>> +
>> +SwitchToRealProcEnd:
>> +
>>  ;-------------------------------------------------------------------------------------
>>  ;  AsmRelocateApLoop (MwaitSupport, ApTargetCState, PmCodeSegment, TopOfApStack, CountTofinish);
>>  ;-------------------------------------------------------------------------------------
>>  global ASM_PFX(AsmRelocateApLoop)
>>  ASM_PFX(AsmRelocateApLoop):
>>  AsmRelocateApLoopStart:
>> +BITS 64
>>      cli                          ; Disable interrupt before switching to 32-bit mode
>>      mov        rax, [rsp + 40]   ; CountTofinish
>>      lock dec   dword [rax]       ; (*CountTofinish)--
>> @@ -324,6 +558,11 @@ ASM_PFX(AsmGetAddressMap):
>>      mov        qword [rcx + 18h], rax
>>      mov        qword [rcx + 20h], AsmRelocateApLoopEnd - AsmRelocateApLoopStart
>>      mov        qword [rcx + 28h], Flat32Start - RendezvousFunnelProcStart
>> +    mov        qword [rcx + 30h], SwitchToRealProcEnd - SwitchToRealProcStart          ; SwitchToRealSize
>> +    mov        qword [rcx + 38h], SwitchToRealProcStart - RendezvousFunnelProcStart    ; SwitchToRealOffset
>> +    mov        qword [rcx + 40h], SwitchToRealProcStart - Flat32Start                  ; SwitchToRealNoNxOffset
>> +    mov        qword [rcx + 48h], PM16Mode - RendezvousFunnelProcStart                 ; SwitchToRealPM16ModeOffset
>> +    mov        qword [rcx + 50h], SwitchToRealProcEnd - PM16Mode                       ; SwitchToRealPM16ModeSize
>>      ret
>>  
>>  ;-------------------------------------------------------------------------------------
>>
>