[PATCH v2 0/5] Target to enable paging from temporary RAM Done

[PATCH v2 0/5] Target to enable paging from temporary RAM Done

[Wu, Jiaxin]

For arch X64, system will enable the page table in SPI to cover 0-512G
range via CR4.PAE & MSR.LME & CR0.PG & CR3 setting. Existing code doesn't
cover the higher address access above 512G before memory-discovered
callback. This series patches provide the solution to enable paging from
temporary RAM Done.

Jiaxin Wu (5):
  UefiCpuPkg/SecCore: Migrate page table to permanent memory
  UefiCpuPkg/CpuMpPei: Enable PAE page table if CR0.PG is not set
  MdeModulePkg/DxeIpl: Align Page table Level setting with previous
    level.
  OvmfPkg: Add CpuPageTableLib required by SecCore & CpuMpPei
  UefiPayloadPkg: Add CpuPageTableLib required by SecCore & CpuMpPei

 MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c |  39 ++--
 OvmfPkg/AmdSev/AmdSevX64.dsc                     |   2 +-
 OvmfPkg/CloudHv/CloudHvX64.dsc                   |   2 +-
 OvmfPkg/IntelTdx/IntelTdxX64.dsc                 |   3 +-
 OvmfPkg/Microvm/MicrovmX64.dsc                   |   2 +-
 OvmfPkg/OvmfPkgIa32.dsc                          |   1 +
 OvmfPkg/OvmfPkgIa32X64.dsc                       |   2 +-
 OvmfPkg/OvmfPkgX64.dsc                           |   3 +-
 OvmfPkg/OvmfXen.dsc                              |   2 +-
 UefiCpuPkg/CpuMpPei/CpuMpPei.h                   |   1 +
 UefiCpuPkg/CpuMpPei/CpuMpPei.inf                 |   1 +
 UefiCpuPkg/CpuMpPei/CpuPaging.c                  | 215 +++++++++--------------
 UefiCpuPkg/SecCore/SecCore.inf                   |   1 +
 UefiCpuPkg/SecCore/SecCoreNative.inf             |   1 +
 UefiCpuPkg/SecCore/SecMain.c                     | 152 ++++++++++++++++
 UefiCpuPkg/SecCore/SecMain.h                     |   4 +
 UefiPayloadPkg/UefiPayloadPkg.dsc                |   2 +-
 17 files changed, 282 insertions(+), 151 deletions(-)

-- 
2.16.2.windows.1

[PATCH v2 1/5] UefiCpuPkg/SecCore: Migrate page table to permanent memory

[Wu, Jiaxin]

Background:
For arch X64, system will enable the page table in SPI to cover 0-512G range
via CR4.PAE & MSR.LME & CR0.PG & CR3 setting (see ResetVector code). Existing
code doesn't cover the higher address access above 512G before memory-discovered
callback. That will be potential problem if system access the higher address
after the transition from temporary RAM to permanent MEM RAM.

Solution:
This patch is to migrate page table to permanent memory to map entire physical
address space if CR0.PG is set during temporary RAM Done.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
---
 UefiCpuPkg/SecCore/SecCore.inf       |   1 +
 UefiCpuPkg/SecCore/SecCoreNative.inf |   1 +
 UefiCpuPkg/SecCore/SecMain.c         | 152 +++++++++++++++++++++++++++++++++++
 UefiCpuPkg/SecCore/SecMain.h         |   4 +
 4 files changed, 158 insertions(+)

diff --git a/UefiCpuPkg/SecCore/SecCore.inf b/UefiCpuPkg/SecCore/SecCore.inf
index 3758aded3b..cab69b8b97 100644
--- a/UefiCpuPkg/SecCore/SecCore.inf
+++ b/UefiCpuPkg/SecCore/SecCore.inf
@@ -53,10 +53,11 @@
   CpuExceptionHandlerLib
   ReportStatusCodeLib
   PeiServicesLib
   PeiServicesTablePointerLib
   HobLib
+  CpuPageTableLib
 
 [Ppis]
   ## SOMETIMES_CONSUMES
   ## PRODUCES
   gEfiSecPlatformInformationPpiGuid
diff --git a/UefiCpuPkg/SecCore/SecCoreNative.inf b/UefiCpuPkg/SecCore/SecCoreNative.inf
index 1ee6ff7d88..fa241cca94 100644
--- a/UefiCpuPkg/SecCore/SecCoreNative.inf
+++ b/UefiCpuPkg/SecCore/SecCoreNative.inf
@@ -50,10 +50,11 @@
   CpuExceptionHandlerLib
   ReportStatusCodeLib
   PeiServicesLib
   PeiServicesTablePointerLib
   HobLib
+  CpuPageTableLib
 
 [Ppis]
   ## SOMETIMES_CONSUMES
   ## PRODUCES
   gEfiSecPlatformInformationPpiGuid
diff --git a/UefiCpuPkg/SecCore/SecMain.c b/UefiCpuPkg/SecCore/SecMain.c
index 95375850ec..8ec0b654fb 100644
--- a/UefiCpuPkg/SecCore/SecMain.c
+++ b/UefiCpuPkg/SecCore/SecMain.c
@@ -70,10 +70,139 @@ MigrateGdt (
   AsmWriteGdtr (&Gdtr);
 
   return EFI_SUCCESS;
 }
 
+/**
+  Migrate page table to permanent memory mapping entire physical address space.
+
+  @retval   EFI_SUCCESS           The PageTable was migrated successfully.
+  @retval   EFI_UNSUPPORTED       Unsupport to migrate page table to permanent memory if IA-32e Mode not actived.
+  @retval   EFI_OUT_OF_RESOURCES  The PageTable could not be migrated due to lack of available memory.
+
+**/
+EFI_STATUS
+MigratePageTable (
+  VOID
+  )
+{
+  EFI_STATUS                      Status;
+  IA32_CR4                        Cr4;
+  BOOLEAN                         Page5LevelSupport;
+  UINT32                          RegEax;
+  CPUID_EXTENDED_CPU_SIG_EDX      RegEdx;
+  BOOLEAN                         Page1GSupport;
+  PAGING_MODE                     PagingMode;
+  CPUID_VIR_PHY_ADDRESS_SIZE_EAX  VirPhyAddressSize;
+  UINT32                          MaxExtendedFunctionId;
+  UINTN                           PageTable;
+  EFI_PHYSICAL_ADDRESS            Buffer;
+  UINTN                           BufferSize;
+  IA32_MAP_ATTRIBUTE              MapAttribute;
+  IA32_MAP_ATTRIBUTE              MapMask;
+
+  VirPhyAddressSize.Uint32    = 0;
+  PageTable                   = 0;
+  BufferSize                  = 0;
+  MapAttribute.Uint64         = 0;
+  MapMask.Uint64              = MAX_UINT64;
+  MapAttribute.Bits.Present   = 1;
+  MapAttribute.Bits.ReadWrite = 1;
+
+  //
+  // Check Page5Level Support or not.
+  //
+  Cr4.UintN         = AsmReadCr4 ();
+  Page5LevelSupport = (Cr4.Bits.LA57 ? TRUE : FALSE);
+
+  //
+  // Check Page1G Support or not.
+  //
+  Page1GSupport = FALSE;
+  AsmCpuid (CPUID_EXTENDED_FUNCTION, &RegEax, NULL, NULL, NULL);
+  if (RegEax >= CPUID_EXTENDED_CPU_SIG) {
+    AsmCpuid (CPUID_EXTENDED_CPU_SIG, NULL, NULL, NULL, &(RegEdx.Uint32));
+    if ((RegEdx.Bits.Page1GB) != 0) {
+      Page1GSupport = TRUE;
+    }
+  }
+
+  //
+  // Decide Paging Mode according Page5LevelSupport & Page1GSupport.
+  //
+  if (Page5LevelSupport) {
+    PagingMode = Page1GSupport ? Paging5Level1GB : Paging5Level;
+  } else {
+    PagingMode = Page1GSupport ? Paging4Level1GB : Paging4Level;
+  }
+
+  //
+  // Get Maximum Physical Address Bits
+  // Get the number of address lines; Maximum Physical Address is 2^PhysicalAddressBits - 1.
+  // If CPUID does not supported, then use a max value of 36 as per SDM 3A, 4.1.4.
+  //
+  AsmCpuid (CPUID_EXTENDED_FUNCTION, &MaxExtendedFunctionId, NULL, NULL, NULL);
+  if (MaxExtendedFunctionId >= CPUID_VIR_PHY_ADDRESS_SIZE) {
+    AsmCpuid (CPUID_VIR_PHY_ADDRESS_SIZE, &VirPhyAddressSize.Uint32, NULL, NULL, NULL);
+  } else {
+    VirPhyAddressSize.Bits.PhysicalAddressBits = 36;
+  }
+
+  if ((PagingMode == Paging4Level1GB) || (PagingMode == Paging4Level)) {
+    //
+    // The max lineaddress bits is 48 for 4 level page table.
+    //
+    VirPhyAddressSize.Bits.PhysicalAddressBits = MIN (VirPhyAddressSize.Bits.PhysicalAddressBits, 48);
+  }
+
+  //
+  // Get required buffer size for the pagetable that will be created.
+  //
+  Status = PageTableMap (&PageTable, PagingMode, 0, &BufferSize, 0, LShiftU64 (1, VirPhyAddressSize.Bits.PhysicalAddressBits), &MapAttribute, &MapMask, NULL);
+  ASSERT (Status == EFI_BUFFER_TOO_SMALL);
+  if (Status != EFI_BUFFER_TOO_SMALL) {
+    return Status;
+  }
+
+  //
+  // Allocate required Buffer.
+  //
+  Status = PeiServicesAllocatePages (
+             EfiBootServicesData,
+             EFI_SIZE_TO_PAGES (BufferSize),
+             &Buffer
+             );
+  if (EFI_ERROR (Status)) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+
+  //
+  // Create PageTable in permanent memory.
+  //
+  Status = PageTableMap (&PageTable, PagingMode, (VOID *)(UINTN)Buffer, &BufferSize, 0, LShiftU64 (1, VirPhyAddressSize.Bits.PhysicalAddressBits), &MapAttribute, &MapMask, NULL);
+  ASSERT_EFI_ERROR (Status);
+  if (EFI_ERROR (Status) || (PageTable == 0)) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+
+  //
+  // Write the Pagetable to CR3.
+  //
+  AsmWriteCr3 (PageTable);
+
+  DEBUG ((
+    DEBUG_INFO,
+    "MigratePageTable: Created PageTable = 0x%lx, BufferSize = %x, PagingMode = 0x%lx, Support Max Physical Address Bits = %d\n",
+    PageTable,
+    BufferSize,
+    (UINTN)PagingMode,
+    VirPhyAddressSize.Bits.PhysicalAddressBits
+    ));
+
+  return Status;
+}
+
 //
 // These are IDT entries pointing to 10:FFFFFFE4h.
 //
 UINT64  mIdtEntryTemplate = 0xffff8e000010ffe4ULL;
 
@@ -451,10 +580,11 @@ SecTemporaryRamDone (
   EFI_STATUS              Status2;
   UINTN                   Index;
   BOOLEAN                 State;
   EFI_PEI_PPI_DESCRIPTOR  *PeiPpiDescriptor;
   REPUBLISH_SEC_PPI_PPI   *RepublishSecPpiPpi;
+  IA32_CR0                Cr0;
 
   //
   // Republish Sec Platform Information(2) PPI
   //
   RepublishSecPlatformInformationPpi ();
@@ -492,10 +622,32 @@ SecTemporaryRamDone (
   if (PcdGetBool (PcdMigrateTemporaryRamFirmwareVolumes)) {
     Status = MigrateGdt ();
     ASSERT_EFI_ERROR (Status);
   }
 
+  //
+  // Migrate page table to permanent memory mapping entire physical address space if CR0.PG is set.
+  //
+  Cr0.UintN = AsmReadCr0 ();
+  if (Cr0.Bits.PG != 0) {
+    //
+    // CR4.PAE must be enabled.
+    //
+    ASSERT ((AsmReadCr4 () & BIT5) != 0);
+
+    //
+    // Assume CPU runs in 64bit mode if paging is enabled.
+    //
+    ASSERT (sizeof (UINTN) == sizeof (UINT64));
+
+    Status = MigratePageTable ();
+    if (EFI_ERROR (Status)) {
+      DEBUG ((DEBUG_WARN, "SecTemporaryRamDone: Failed to migrate page table to permanent memory: %r.\n", Status));
+      ASSERT_EFI_ERROR (Status);
+    }
+  }
+
   //
   // Disable Temporary RAM after Stack and Heap have been migrated at this point.
   //
   SecPlatformDisableTemporaryMemory ();
 
diff --git a/UefiCpuPkg/SecCore/SecMain.h b/UefiCpuPkg/SecCore/SecMain.h
index 880e6cd1b8..b50d96e45b 100644
--- a/UefiCpuPkg/SecCore/SecMain.h
+++ b/UefiCpuPkg/SecCore/SecMain.h
@@ -17,10 +17,11 @@
 #include <Ppi/PeiCoreFvLocation.h>
 #include <Ppi/RepublishSecPpi.h>
 
 #include <Guid/FirmwarePerformance.h>
 
+#include <Library/BaseLib.h>
 #include <Library/DebugLib.h>
 #include <Library/PcdLib.h>
 #include <Library/BaseMemoryLib.h>
 #include <Library/PlatformSecLib.h>
 #include <Library/CpuLib.h>
@@ -30,10 +31,13 @@
 #include <Library/CpuExceptionHandlerLib.h>
 #include <Library/ReportStatusCodeLib.h>
 #include <Library/PeiServicesTablePointerLib.h>
 #include <Library/HobLib.h>
 #include <Library/PeiServicesLib.h>
+#include <Library/CpuPageTableLib.h>
+#include <Register/Intel/Cpuid.h>
+#include <Register/Intel/Msr.h>
 
 #define SEC_IDT_ENTRY_COUNT  34
 
 typedef struct _SEC_IDT_TABLE {
   //
-- 
2.16.2.windows.1

[PATCH v2 2/5] UefiCpuPkg/CpuMpPei: Enable PAE page table if CR0.PG is not set

[Wu, Jiaxin]

Some security features depends on the page table enabling. So, This patch
is to enable the page table if page table has not been enabled during the
transition from Temporary RAM to Permanent RAM.

Note: If page table is not enabled before this point, which means the system
IA-32e Mode is not activated. Because on Intel 64 processors, IA-32e Mode
operation requires physical address extensions with 4 or 5 levels of enhanced
paging structures (see Section 4.5, "4 - Level Paging and 5 -Level Paging"
and Section 9.8, "Initializing IA-32e Mode"). So, just enable PAE page table
if CR0.PG is not set.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
---
 UefiCpuPkg/CpuMpPei/CpuMpPei.h   |   1 +
 UefiCpuPkg/CpuMpPei/CpuMpPei.inf |   1 +
 UefiCpuPkg/CpuMpPei/CpuPaging.c  | 215 ++++++++++++++++-----------------------
 3 files changed, 88 insertions(+), 129 deletions(-)

diff --git a/UefiCpuPkg/CpuMpPei/CpuMpPei.h b/UefiCpuPkg/CpuMpPei/CpuMpPei.h
index 0649c48d14..1b9a94e18f 100644
--- a/UefiCpuPkg/CpuMpPei/CpuMpPei.h
+++ b/UefiCpuPkg/CpuMpPei/CpuMpPei.h
@@ -26,10 +26,11 @@
 #include <Library/ReportStatusCodeLib.h>
 #include <Library/CpuExceptionHandlerLib.h>
 #include <Library/MpInitLib.h>
 #include <Library/BaseMemoryLib.h>
 #include <Library/MemoryAllocationLib.h>
+#include <Library/CpuPageTableLib.h>
 
 extern EFI_PEI_PPI_DESCRIPTOR  mPeiCpuMpPpiDesc;
 
 /**
   This service retrieves the number of logical processor in the platform
diff --git a/UefiCpuPkg/CpuMpPei/CpuMpPei.inf b/UefiCpuPkg/CpuMpPei/CpuMpPei.inf
index 7444bdb968..865be5627e 100644
--- a/UefiCpuPkg/CpuMpPei/CpuMpPei.inf
+++ b/UefiCpuPkg/CpuMpPei/CpuMpPei.inf
@@ -44,10 +44,11 @@
   CpuExceptionHandlerLib
   MpInitLib
   BaseMemoryLib
   CpuLib
   MemoryAllocationLib
+  CpuPageTableLib
 
 [Guids]
   gEdkiiMigratedFvInfoGuid                                             ## SOMETIMES_CONSUMES     ## HOB
 
 [Ppis]
diff --git a/UefiCpuPkg/CpuMpPei/CpuPaging.c b/UefiCpuPkg/CpuMpPei/CpuPaging.c
index a471f089c8..b282e1cf14 100644
--- a/UefiCpuPkg/CpuMpPei/CpuPaging.c
+++ b/UefiCpuPkg/CpuMpPei/CpuPaging.c
@@ -115,42 +115,10 @@ AllocatePageTableMemory (
   }
 
   return Address;
 }
 
-/**
-  Get the address width supported by current processor.
-
-  @retval 32      If processor is in 32-bit mode.
-  @retval 36-48   If processor is in 64-bit mode.
-
-**/
-UINTN
-GetPhysicalAddressWidth (
-  VOID
-  )
-{
-  UINT32  RegEax;
-
-  if (sizeof (UINTN) == 4) {
-    return 32;
-  }
-
-  AsmCpuid (CPUID_EXTENDED_FUNCTION, &RegEax, NULL, NULL, NULL);
-  if (RegEax >= CPUID_VIR_PHY_ADDRESS_SIZE) {
-    AsmCpuid (CPUID_VIR_PHY_ADDRESS_SIZE, &RegEax, NULL, NULL, NULL);
-    RegEax &= 0xFF;
-    if (RegEax > 48) {
-      return 48;
-    }
-
-    return (UINTN)RegEax;
-  }
-
-  return 36;
-}
-
 /**
   Get the type of top level page table.
 
   @retval Page512G  PML4 paging.
   @retval Page1G    PAE paging.
@@ -381,120 +349,91 @@ ConvertMemoryPageAttributes (
 
   return RETURN_SUCCESS;
 }
 
 /**
-  Get maximum size of page memory supported by current processor.
-
-  @param[in]   TopLevelType     The type of top level page entry.
+  Enable PAE Page Table.
 
-  @retval Page1G     If processor supports 1G page and PML4.
-  @retval Page2M     For all other situations.
+  @retval   EFI_SUCCESS           The PAE Page Table was enabled successfully.
+  @retval   EFI_OUT_OF_RESOURCES  The PAE Page Table could not be enabled due to lack of available memory.
 
 **/
-PAGE_ATTRIBUTE
-GetMaxMemoryPage (
-  IN  PAGE_ATTRIBUTE  TopLevelType
-  )
-{
-  UINT32  RegEax;
-  UINT32  RegEdx;
-
-  if (TopLevelType == Page512G) {
-    AsmCpuid (CPUID_EXTENDED_FUNCTION, &RegEax, NULL, NULL, NULL);
-    if (RegEax >= CPUID_EXTENDED_CPU_SIG) {
-      AsmCpuid (CPUID_EXTENDED_CPU_SIG, NULL, NULL, NULL, &RegEdx);
-      if ((RegEdx & BIT26) != 0) {
-        return Page1G;
-      }
-    }
-  }
-
-  return Page2M;
-}
-
-/**
-  Create PML4 or PAE page table.
-
-  @return The address of page table.
-
-**/
-UINTN
-CreatePageTable (
+EFI_STATUS
+EnablePaePageTable (
   VOID
   )
 {
-  RETURN_STATUS         Status;
-  UINTN                 PhysicalAddressBits;
-  UINTN                 NumberOfEntries;
-  PAGE_ATTRIBUTE        TopLevelPageAttr;
-  UINTN                 PageTable;
-  PAGE_ATTRIBUTE        MaxMemoryPage;
-  UINTN                 Index;
-  UINT64                AddressEncMask;
-  UINT64                *PageEntry;
-  EFI_PHYSICAL_ADDRESS  PhysicalAddress;
-
-  TopLevelPageAttr    = (PAGE_ATTRIBUTE)GetPageTableTopLevelType ();
-  PhysicalAddressBits = GetPhysicalAddressWidth ();
-  NumberOfEntries     = (UINTN)1 << (PhysicalAddressBits -
-                                     mPageAttributeTable[TopLevelPageAttr].AddressBitOffset);
+  EFI_STATUS   Status;
+  PAGING_MODE  PagingMode;
+
+  UINTN               PageTable;
+  VOID                *Buffer;
+  UINTN               BufferSize;
+  IA32_MAP_ATTRIBUTE  MapAttribute;
+  IA32_MAP_ATTRIBUTE  MapMask;
+
+  PagingMode                  = PagingPae;
+  PageTable                   = 0;
+  Buffer                      = NULL;
+  BufferSize                  = 0;
+  MapAttribute.Uint64         = 0;
+  MapMask.Uint64              = MAX_UINT64;
+  MapAttribute.Bits.Present   = 1;
+  MapAttribute.Bits.ReadWrite = 1;
 
-  PageTable = (UINTN)AllocatePageTableMemory (1);
-  if (PageTable == 0) {
-    return 0;
+  //
+  // Get required buffer size for the pagetable that will be created.
+  // The Max size of LinearAddress for PAE is 2^32.
+  //
+  Status = PageTableMap (&PageTable, PagingMode, 0, &BufferSize, 0, SIZE_4GB, &MapAttribute, &MapMask, NULL);
+  ASSERT (Status == EFI_BUFFER_TOO_SMALL);
+  if (Status != EFI_BUFFER_TOO_SMALL) {
+    return Status;
   }
 
-  AddressEncMask  = PcdGet64 (PcdPteMemoryEncryptionAddressOrMask);
-  AddressEncMask &= mPageAttributeTable[TopLevelPageAttr].AddressMask;
-  MaxMemoryPage   = GetMaxMemoryPage (TopLevelPageAttr);
-  PageEntry       = (UINT64 *)PageTable;
-
-  PhysicalAddress = 0;
-  for (Index = 0; Index < NumberOfEntries; ++Index) {
-    *PageEntry = PhysicalAddress | AddressEncMask | PAGE_ATTRIBUTE_BITS;
+  //
+  // Allocate required Buffer.
+  //
+  Buffer = AllocatePageTableMemory (EFI_SIZE_TO_PAGES (BufferSize));
+  ASSERT (Buffer != NULL);
+  if (Buffer == NULL) {
+    return EFI_OUT_OF_RESOURCES;
+  }
 
-    //
-    // Split the top page table down to the maximum page size supported
-    //
-    if (MaxMemoryPage < TopLevelPageAttr) {
-      Status = SplitPage (PageEntry, TopLevelPageAttr, MaxMemoryPage, TRUE);
-      ASSERT_EFI_ERROR (Status);
-    }
+  //
+  // Create PageTable in permanent memory.
+  // The Max size of LinearAddress for PAE is 2^32.
+  //
+  Status = PageTableMap (&PageTable, PagingMode, Buffer, &BufferSize, 0, LShiftU64 (1, 32), &MapAttribute, &MapMask, NULL);
+  ASSERT_EFI_ERROR (Status);
+  if (EFI_ERROR (Status) || (PageTable == 0)) {
+    return EFI_OUT_OF_RESOURCES;
+  }
 
-    if (TopLevelPageAttr == Page1G) {
-      //
-      // PDPTE[2:1] (PAE Paging) must be 0. SplitPage() might change them to 1.
-      //
-      *PageEntry &= ~(UINT64)(IA32_PG_RW | IA32_PG_U);
-    }
+  //
+  // Write the Pagetable to CR3.
+  //
+  AsmWriteCr3 (PageTable);
 
-    PageEntry       += 1;
-    PhysicalAddress += mPageAttributeTable[TopLevelPageAttr].Length;
-  }
+  //
+  // Enable CR4.PAE
+  //
+  AsmWriteCr4 (AsmReadCr4 () | BIT5);
 
-  return PageTable;
-}
+  //
+  // Enable CR0.PG
+  //
+  AsmWriteCr0 (AsmReadCr0 () | BIT31);
 
-/**
-  Setup page tables and make them work.
+  DEBUG ((
+    DEBUG_INFO,
+    "EnablePaePageTable: Created PageTable = 0x%x, BufferSize = %x\n",
+    PageTable,
+    BufferSize
+    ));
 
-**/
-VOID
-EnablePaging (
-  VOID
-  )
-{
-  UINTN  PageTable;
-
-  PageTable = CreatePageTable ();
-  ASSERT (PageTable != 0);
-  if (PageTable != 0) {
-    AsmWriteCr3 (PageTable);
-    AsmWriteCr4 (AsmReadCr4 () | BIT5);   // CR4.PAE
-    AsmWriteCr0 (AsmReadCr0 () | BIT31);  // CR0.PG
-  }
+  return Status;
 }
 
 /**
   Get the base address of current AP's stack.
 
@@ -622,10 +561,11 @@ MemoryDiscoveredPpiNotifyCallback (
 {
   EFI_STATUS              Status;
   BOOLEAN                 InitStackGuard;
   EDKII_MIGRATED_FV_INFO  *MigratedFvInfo;
   EFI_PEI_HOB_POINTERS    Hob;
+  IA32_CR0                Cr0;
 
   //
   // Paging must be setup first. Otherwise the exception TSS setup during MP
   // initialization later will not contain paging information and then fail
   // the task switch (for the sake of stack switch).
@@ -635,12 +575,29 @@ MemoryDiscoveredPpiNotifyCallback (
   if (IsIa32PaeSupported ()) {
     Hob.Raw        = GetFirstGuidHob (&gEdkiiMigratedFvInfoGuid);
     InitStackGuard = PcdGetBool (PcdCpuStackGuard);
   }
 
-  if (InitStackGuard || (Hob.Raw != NULL)) {
-    EnablePaging ();
+  //
+  // Some security features depends on the page table enabling. So, here
+  // is to enable the page table if page table has not been enabled yet.
+  // If page table is not enabled before this point, which means the system
+  // IA-32e Mode is not activated. Because on Intel 64 processors, IA-32e Mode
+  // operation requires physical address extensions with 4 or 5 levels of
+  // enhanced paging structures (see Section 4.5, "4 - Level Paging and 5 -
+  // Level Paging" and Section 9.8, "Initializing IA-32e Mode"). So, just
+  // enable PAE page table if CR0.PG is not set.
+  //
+  Cr0.UintN = AsmReadCr0 ();
+  if ((Cr0.Bits.PG == 0) && (InitStackGuard || (Hob.Raw != NULL))) {
+    ASSERT (sizeof (UINTN) == sizeof (UINT32));
+
+    Status = EnablePaePageTable ();
+    if (EFI_ERROR (Status)) {
+      DEBUG ((DEBUG_WARN, "MemoryDiscoveredPpiNotifyCallback: Failed to enable PAE page table: %r.\n", Status));
+      ASSERT_EFI_ERROR (Status);
+    }
   }
 
   Status = InitializeCpuMpWorker ((CONST EFI_PEI_SERVICES **)PeiServices);
   ASSERT_EFI_ERROR (Status);
 
-- 
2.16.2.windows.1

[PATCH v2 3/5] MdeModulePkg/DxeIpl: Align Page table Level setting with previous level.

[Wu, Jiaxin]

System paging 5 level enabled or not can be checked via CR4.LA57, system
preferred Page table Level (PcdUse5LevelPageTable) must align with previous
level for 64bit long mode.

This patch is to do the wise check:
If cpu has already runned in 64bit long mode PEI, Page table Level in DXE
must align with previous level.
If cpu runs in 32bit protected mode PEI, Page table Level in DXE is decided
by PCD and feature capability.

Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
---
 MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c | 39 ++++++++++++++++--------
 1 file changed, 27 insertions(+), 12 deletions(-)

diff --git a/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c b/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c
index 18b121d768..a8e46eacc3 100644
--- a/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c
+++ b/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c
@@ -737,22 +737,37 @@ CreateIdentityMappingPageTables (
     } else {
       PhysicalAddressBits = 36;
     }
   }
 
-  Page5LevelSupport = FALSE;
-  if (PcdGetBool (PcdUse5LevelPageTable)) {
-    AsmCpuidEx (
-      CPUID_STRUCTURED_EXTENDED_FEATURE_FLAGS,
-      CPUID_STRUCTURED_EXTENDED_FEATURE_FLAGS_SUB_LEAF_INFO,
-      NULL,
-      NULL,
-      &EcxFlags.Uint32,
-      NULL
-      );
-    if (EcxFlags.Bits.FiveLevelPage != 0) {
-      Page5LevelSupport = TRUE;
+  //
+  // Check cpu runs in 64bit long mode or 32bit protected mode.
+  //
+  if (sizeof (UINTN) == sizeof (UINT64)) {
+    //
+    // If cpu has already runned in 64bit long mode PEI, Page table Level in DXE must align with previous level.
+    //
+    Cr4.UintN         = AsmReadCr4 ();
+    Page5LevelSupport = Cr4.Bits.LA57 ? TRUE : FALSE;
+    ASSERT (PcdGetBool (PcdUse5LevelPageTable) == Page5LevelSupport);
+  } else {
+    //
+    // If cpu runs in 32bit protected mode PEI, Page table Level in DXE is decided by PCD and feature capability.
+    //
+    Page5LevelSupport = FALSE;
+    if (PcdGetBool (PcdUse5LevelPageTable)) {
+      AsmCpuidEx (
+        CPUID_STRUCTURED_EXTENDED_FEATURE_FLAGS,
+        CPUID_STRUCTURED_EXTENDED_FEATURE_FLAGS_SUB_LEAF_INFO,
+        NULL,
+        NULL,
+        &EcxFlags.Uint32,
+        NULL
+        );
+      if (EcxFlags.Bits.FiveLevelPage != 0) {
+        Page5LevelSupport = TRUE;
+      }
     }
   }
 
   DEBUG ((DEBUG_INFO, "AddressBits=%u 5LevelPaging=%u 1GPage=%u\n", PhysicalAddressBits, Page5LevelSupport, Page1GSupport));
 
-- 
2.16.2.windows.1

[PATCH v2 4/5] OvmfPkg: Add CpuPageTableLib required by SecCore & CpuMpPei

[Wu, Jiaxin]

Add CpuPageTableLib required by SecCore & CpuMpPei in OvmfPkg.

Cc: Ard Biesheuvel <ardb+tianocore@kernel.org>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
---
 OvmfPkg/AmdSev/AmdSevX64.dsc     | 2 +-
 OvmfPkg/CloudHv/CloudHvX64.dsc   | 2 +-
 OvmfPkg/IntelTdx/IntelTdxX64.dsc | 3 +--
 OvmfPkg/Microvm/MicrovmX64.dsc   | 2 +-
 OvmfPkg/OvmfPkgIa32.dsc          | 1 +
 OvmfPkg/OvmfPkgIa32X64.dsc       | 2 +-
 OvmfPkg/OvmfPkgX64.dsc           | 3 +--
 OvmfPkg/OvmfXen.dsc              | 2 +-
 8 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/OvmfPkg/AmdSev/AmdSevX64.dsc b/OvmfPkg/AmdSev/AmdSevX64.dsc
index b32049194d..dfcef0efca 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.dsc
+++ b/OvmfPkg/AmdSev/AmdSevX64.dsc
@@ -145,10 +145,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -347,11 +348,10 @@
   LockBoxLib|OvmfPkg/Library/LockBoxLib/LockBoxDxeLib.inf
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/GenericQemuLoadImageLib/GenericQemuLoadImageLib.inf
 
diff --git a/OvmfPkg/CloudHv/CloudHvX64.dsc b/OvmfPkg/CloudHv/CloudHvX64.dsc
index 2a1139daaa..d4f86f8b71 100644
--- a/OvmfPkg/CloudHv/CloudHvX64.dsc
+++ b/OvmfPkg/CloudHv/CloudHvX64.dsc
@@ -162,10 +162,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -388,11 +389,10 @@
 !endif
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
 
diff --git a/OvmfPkg/IntelTdx/IntelTdxX64.dsc b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
index d4403f11a7..0c21a713fc 100644
--- a/OvmfPkg/IntelTdx/IntelTdxX64.dsc
+++ b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
@@ -149,10 +149,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -304,11 +305,10 @@
   PlatformBmPrintScLib|OvmfPkg/Library/PlatformBmPrintScLib/PlatformBmPrintScLib.inf
   QemuBootOrderLib|OvmfPkg/Library/QemuBootOrderLib/QemuBootOrderLib.inf
   CpuExceptionHandlerLib|UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.inf
   LockBoxLib|OvmfPkg/Library/LockBoxLib/LockBoxDxeLib.inf
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
 
@@ -576,11 +576,10 @@
     <LibraryClasses>
       #
       # Directly use DxeMpInitLib. It depends on DxeMpInitLibMpDepLib which
       # checks the Protocol of gEfiMpInitLibMpDepProtocolGuid.
       #
-      CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
       MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
       NULL|OvmfPkg/Library/MpInitLibDepLib/DxeMpInitLibMpDepLib.inf
   }
 
   UefiCpuPkg/CpuDxe/CpuDxe.inf {
diff --git a/OvmfPkg/Microvm/MicrovmX64.dsc b/OvmfPkg/Microvm/MicrovmX64.dsc
index 5f671bc384..5739e98f78 100644
--- a/OvmfPkg/Microvm/MicrovmX64.dsc
+++ b/OvmfPkg/Microvm/MicrovmX64.dsc
@@ -160,10 +160,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -387,11 +388,10 @@
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
   PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
   PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
 
diff --git a/OvmfPkg/OvmfPkgIa32.dsc b/OvmfPkg/OvmfPkgIa32.dsc
index e333b8b418..0c044e34f8 100644
--- a/OvmfPkg/OvmfPkgIa32.dsc
+++ b/OvmfPkg/OvmfPkgIa32.dsc
@@ -165,10 +165,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
diff --git a/OvmfPkg/OvmfPkgIa32X64.dsc b/OvmfPkg/OvmfPkgIa32X64.dsc
index 25974230a2..20f8686ee5 100644
--- a/OvmfPkg/OvmfPkgIa32X64.dsc
+++ b/OvmfPkg/OvmfPkgIa32X64.dsc
@@ -169,10 +169,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -400,11 +401,10 @@
 !endif
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
 
diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc
index c1762ffca4..ce89817dda 100644
--- a/OvmfPkg/OvmfPkgX64.dsc
+++ b/OvmfPkg/OvmfPkgX64.dsc
@@ -182,10 +182,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -421,11 +422,10 @@
 !endif
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
 
@@ -814,11 +814,10 @@
     <LibraryClasses>
       #
       # Directly use DxeMpInitLib. It depends on DxeMpInitLibMpDepLib which
       # checks the Protocol of gEfiMpInitLibMpDepProtocolGuid.
       #
-      CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
       MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
       NULL|OvmfPkg/Library/MpInitLibDepLib/DxeMpInitLibMpDepLib.inf
   }
 
   UefiCpuPkg/CpuDxe/CpuDxe.inf {
diff --git a/OvmfPkg/OvmfXen.dsc b/OvmfPkg/OvmfXen.dsc
index 1f44ec86c9..de445ebc0b 100644
--- a/OvmfPkg/OvmfXen.dsc
+++ b/OvmfPkg/OvmfXen.dsc
@@ -153,10 +153,11 @@
   IoLib|MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsicSev.inf
   OemHookStatusCodeLib|MdeModulePkg/Library/OemHookStatusCodeLibNull/OemHookStatusCodeLibNull.inf
   SerialPortLib|PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
   UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
   UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
   UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
@@ -334,11 +335,10 @@
   LockBoxLib|OvmfPkg/Library/LockBoxLib/LockBoxDxeLib.inf
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   NestedInterruptTplLib|OvmfPkg/Library/NestedInterruptTplLib/NestedInterruptTplLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
 
 [LibraryClasses.common.UEFI_APPLICATION]
-- 
2.16.2.windows.1

[PATCH v2 5/5] UefiPayloadPkg: Add CpuPageTableLib required by SecCore & CpuMpPei

[Wu, Jiaxin]

Add CpuPageTableLib required by SecCore & CpuMpPei in UefiPayloadPkg.

Cc: Guo Dong <guo.dong@intel.com>
Cc: Sean Rhodes <sean@starlabs.systems>
Cc: James Lu <james.lu@intel.com>
Cc: Gua Guo <gua.guo@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Zeng Star <star.zeng@intel.com>
Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
---
 UefiPayloadPkg/UefiPayloadPkg.dsc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/UefiPayloadPkg/UefiPayloadPkg.dsc b/UefiPayloadPkg/UefiPayloadPkg.dsc
index 998d222909..615c5b4c6d 100644
--- a/UefiPayloadPkg/UefiPayloadPkg.dsc
+++ b/UefiPayloadPkg/UefiPayloadPkg.dsc
@@ -240,10 +240,11 @@
   # CPU
   #
   MtrrLib|UefiCpuPkg/Library/MtrrLib/MtrrLib.inf
   LocalApicLib|UefiCpuPkg/Library/BaseXApicX2ApicLib/BaseXApicX2ApicLib.inf
   MicrocodeLib|UefiCpuPkg/Library/MicrocodeLib/MicrocodeLib.inf
+  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
 
   #
   # Platform
   #
 !if $(CPU_TIMER_LIB_ENABLE) == TRUE && $(UNIVERSAL_PAYLOAD) == TRUE
@@ -343,11 +344,10 @@
   ReportStatusCodeLib|MdeModulePkg/Library/DxeReportStatusCodeLib/DxeReportStatusCodeLib.inf
 !if $(SOURCE_DEBUG_ENABLE)
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
   CpuExceptionHandlerLib|UefiCpuPkg/Library/CpuExceptionHandlerLib/DxeCpuExceptionHandlerLib.inf
-  CpuPageTableLib|UefiCpuPkg/Library/CpuPageTableLib/CpuPageTableLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
 !if $(PERFORMANCE_MEASUREMENT_ENABLE)
   PerformanceLib|MdeModulePkg/Library/DxePerformanceLib/DxePerformanceLib.inf
 !endif
 
-- 
2.16.2.windows.1

Re: [PATCH v2 1/5] UefiCpuPkg/SecCore: Migrate page table to permanent memory

[Ni, Ray]

> -----Original Message-----
> From: Wu, Jiaxin <jiaxin.wu@intel.com>
> Sent: Friday, May 12, 2023 12:16 PM
> To: devel@edk2.groups.io
> Cc: Dong, Eric <eric.dong@intel.com>; Ni, Ray <ray.ni@intel.com>; Zeng, Star
> <star.zeng@intel.com>; Gerd Hoffmann <kraxel@redhat.com>; Kumar, Rahul R
> <rahul.r.kumar@intel.com>
> Subject: [PATCH v2 1/5] UefiCpuPkg/SecCore: Migrate page table to permanent
> memory
> 
> Background:
> For arch X64, system will enable the page table in SPI to cover 0-512G range
> via CR4.PAE & MSR.LME & CR0.PG & CR3 setting (see ResetVector code).
> Existing
> code doesn't cover the higher address access above 512G before memory-
> discovered
> callback. That will be potential problem if system access the higher address
> after the transition from temporary RAM to permanent MEM RAM.
> 
> Solution:
> This patch is to migrate page table to permanent memory to map entire physical
> address space if CR0.PG is set during temporary RAM Done.
> 
> Cc: Eric Dong <eric.dong@intel.com>
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Zeng Star <star.zeng@intel.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Rahul Kumar <rahul1.kumar@intel.com>
> Signed-off-by: Jiaxin Wu <jiaxin.wu@intel.com>
> ---
>  UefiCpuPkg/SecCore/SecCore.inf       |   1 +
>  UefiCpuPkg/SecCore/SecCoreNative.inf |   1 +
>  UefiCpuPkg/SecCore/SecMain.c         | 152
> +++++++++++++++++++++++++++++++++++
>  UefiCpuPkg/SecCore/SecMain.h         |   4 +
>  4 files changed, 158 insertions(+)
> 
> diff --git a/UefiCpuPkg/SecCore/SecCore.inf b/UefiCpuPkg/SecCore/SecCore.inf
> index 3758aded3b..cab69b8b97 100644
> --- a/UefiCpuPkg/SecCore/SecCore.inf
> +++ b/UefiCpuPkg/SecCore/SecCore.inf
> @@ -53,10 +53,11 @@
>    CpuExceptionHandlerLib
>    ReportStatusCodeLib
>    PeiServicesLib
>    PeiServicesTablePointerLib
>    HobLib
> +  CpuPageTableLib
> 
>  [Ppis]
>    ## SOMETIMES_CONSUMES
>    ## PRODUCES
>    gEfiSecPlatformInformationPpiGuid
> diff --git a/UefiCpuPkg/SecCore/SecCoreNative.inf
> b/UefiCpuPkg/SecCore/SecCoreNative.inf
> index 1ee6ff7d88..fa241cca94 100644
> --- a/UefiCpuPkg/SecCore/SecCoreNative.inf
> +++ b/UefiCpuPkg/SecCore/SecCoreNative.inf
> @@ -50,10 +50,11 @@
>    CpuExceptionHandlerLib
>    ReportStatusCodeLib
>    PeiServicesLib
>    PeiServicesTablePointerLib
>    HobLib
> +  CpuPageTableLib
> 
>  [Ppis]
>    ## SOMETIMES_CONSUMES
>    ## PRODUCES
>    gEfiSecPlatformInformationPpiGuid
> diff --git a/UefiCpuPkg/SecCore/SecMain.c b/UefiCpuPkg/SecCore/SecMain.c
> index 95375850ec..8ec0b654fb 100644
> --- a/UefiCpuPkg/SecCore/SecMain.c
> +++ b/UefiCpuPkg/SecCore/SecMain.c
> @@ -70,10 +70,139 @@ MigrateGdt (
>    AsmWriteGdtr (&Gdtr);
> 
>    return EFI_SUCCESS;
>  }
> 
> +/**
> +  Migrate page table to permanent memory mapping entire physical address
> space.
> +
> +  @retval   EFI_SUCCESS           The PageTable was migrated successfully.
> +  @retval   EFI_UNSUPPORTED       Unsupport to migrate page table to
> permanent memory if IA-32e Mode not actived.
> +  @retval   EFI_OUT_OF_RESOURCES  The PageTable could not be migrated due
> to lack of available memory.
> +
> +**/
> +EFI_STATUS
> +MigratePageTable (
> +  VOID
> +  )
> +{
> +  EFI_STATUS                      Status;
> +  IA32_CR4                        Cr4;
> +  BOOLEAN                         Page5LevelSupport;
> +  UINT32                          RegEax;
> +  CPUID_EXTENDED_CPU_SIG_EDX      RegEdx;
> +  BOOLEAN                         Page1GSupport;
> +  PAGING_MODE                     PagingMode;
> +  CPUID_VIR_PHY_ADDRESS_SIZE_EAX  VirPhyAddressSize;
> +  UINT32                          MaxExtendedFunctionId;
> +  UINTN                           PageTable;
> +  EFI_PHYSICAL_ADDRESS            Buffer;
> +  UINTN                           BufferSize;
> +  IA32_MAP_ATTRIBUTE              MapAttribute;
> +  IA32_MAP_ATTRIBUTE              MapMask;
> +
> +  VirPhyAddressSize.Uint32    = 0;
> +  PageTable                   = 0;
> +  BufferSize                  = 0;
> +  MapAttribute.Uint64         = 0;
> +  MapMask.Uint64              = MAX_UINT64;
> +  MapAttribute.Bits.Present   = 1;
> +  MapAttribute.Bits.ReadWrite = 1;
> +
> +  //
> +  // Check Page5Level Support or not.
> +  //
> +  Cr4.UintN         = AsmReadCr4 ();
> +  Page5LevelSupport = (Cr4.Bits.LA57 ? TRUE : FALSE);

1. "Cr4.Bits.LA57 != 0", to be consistent with other code in your patch.

> +
> +  //
> +  // Check Page1G Support or not.
> +  //
> +  Page1GSupport = FALSE;
> +  AsmCpuid (CPUID_EXTENDED_FUNCTION, &RegEax, NULL, NULL, NULL);
> +  if (RegEax >= CPUID_EXTENDED_CPU_SIG) {
> +    AsmCpuid (CPUID_EXTENDED_CPU_SIG, NULL, NULL, NULL,
> &(RegEdx.Uint32));

2. &RegEdx.Uint32. No need for the bracket.

> +    if ((RegEdx.Bits.Page1GB) != 0) {
3. No need for the bracket.

> +  //
> +  Cr0.UintN = AsmReadCr0 ();
> +  if (Cr0.Bits.PG != 0) {
> +    //
> +    // CR4.PAE must be enabled.
> +    //
> +    ASSERT ((AsmReadCr4 () & BIT5) != 0);

4. No need to check PAE bit because 64bit long mode should set PAE bit.

> +
> +    //
> +    // Assume CPU runs in 64bit mode if paging is enabled.
> +    //
> +    ASSERT (sizeof (UINTN) == sizeof (UINT64));
> +
> +    Status = MigratePageTable ();
> +    if (EFI_ERROR (Status)) {
> +      DEBUG ((DEBUG_WARN, "SecTemporaryRamDone: Failed to migrate page
> table to permanent memory: %r.\n", Status));
> +      ASSERT_EFI_ERROR (Status);

5. Can you add CpuDeadLoop ()? Failure of page table migration is unacceptable.

Re: [edk2-devel] [PATCH v2 2/5] UefiCpuPkg/CpuMpPei: Enable PAE page table if CR0.PG is not set

[Ni, Ray]

1. can you please change the commit title as "UefiCpuPkg/CpuMpPei: Conditionally enable PAE paging in 32bit mode"?

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Wu, Jiaxin
> Sent: Friday, May 12, 2023 12:16 PM
> To: devel@edk2.groups.io
> Cc: Dong, Eric <eric.dong@intel.com>; Ni, Ray <ray.ni@intel.com>; Zeng, Star
> <star.zeng@intel.com>; Gerd Hoffmann <kraxel@redhat.com>; Kumar, Rahul R
> <rahul.r.kumar@intel.com>
> Subject: [edk2-devel] [PATCH v2 2/5] UefiCpuPkg/CpuMpPei: Enable PAE page
> table if CR0.PG is not set
> 
> Some security features depends on the page table enabling. So, This patch
> is to enable the page table if page table has not been enabled during the
> transition from Temporary RAM to Permanent RAM.

2. No need to emphasize paging is not enabled during temp-ram to permanent-ram migration.
We can just say "Some security features depend on the page table enabling. So, This patch
is to enable paging if it is not enabled (32bit mode)"

3. following commit message might not be needed.

> 
> Note: If page table is not enabled before this point, which means the system
> IA-32e Mode is not activated. Because on Intel 64 processors, IA-32e Mode
> operation requires physical address extensions with 4 or 5 levels of enhanced
> paging structures (see Section 4.5, "4 - Level Paging and 5 -Level Paging"
> and Section 9.8, "Initializing IA-32e Mode"). So, just enable PAE page table
> if CR0.PG is not set.
> 

> -UINTN
> -CreatePageTable (
> +EFI_STATUS
> +EnablePaePageTable (
>    VOID
>    )
>  {
> -  RETURN_STATUS         Status;
> -  UINTN                 PhysicalAddressBits;
> -  UINTN                 NumberOfEntries;
> -  PAGE_ATTRIBUTE        TopLevelPageAttr;
> -  UINTN                 PageTable;
> -  PAGE_ATTRIBUTE        MaxMemoryPage;
> -  UINTN                 Index;
> -  UINT64                AddressEncMask;
> -  UINT64                *PageEntry;
> -  EFI_PHYSICAL_ADDRESS  PhysicalAddress;
> -
> -  TopLevelPageAttr    = (PAGE_ATTRIBUTE)GetPageTableTopLevelType ();
> -  PhysicalAddressBits = GetPhysicalAddressWidth ();
> -  NumberOfEntries     = (UINTN)1 << (PhysicalAddressBits -
> -                                     mPageAttributeTable[TopLevelPageAttr].AddressBitOffset);
> +  EFI_STATUS   Status;
> +  PAGING_MODE  PagingMode;
> +
> +  UINTN               PageTable;
> +  VOID                *Buffer;
> +  UINTN               BufferSize;
> +  IA32_MAP_ATTRIBUTE  MapAttribute;
> +  IA32_MAP_ATTRIBUTE  MapMask;
> +
> +  PagingMode                  = PagingPae;

4. No need of local variable PagingMode.

> +  PageTable                   = 0;
> +  Buffer                      = NULL;
> +  BufferSize                  = 0;
> +  MapAttribute.Uint64         = 0;
> +  MapMask.Uint64              = MAX_UINT64;
> +  MapAttribute.Bits.Present   = 1;
> +  MapAttribute.Bits.ReadWrite = 1;
> 
> -  PageTable = (UINTN)AllocatePageTableMemory (1);
> -  if (PageTable == 0) {
> -    return 0;
> +  //
> +  // Get required buffer size for the pagetable that will be created.
> +  // The Max size of LinearAddress for PAE is 2^32.

5. Let's say "1:1 map 4GB in 32bit mode." Because people might say "PAE can support up to 2^36 address, why 2^32 here".

> +  // Create PageTable in permanent memory.
> +  // The Max size of LinearAddress for PAE is 2^32.

6. above comments seem to be redundant.

>    EFI_PEI_HOB_POINTERS    Hob;
> +  IA32_CR0                Cr0;
> 
>    //
>    // Paging must be setup first. Otherwise the exception TSS setup during MP
>    // initialization later will not contain paging information and then fail
>    // the task switch (for the sake of stack switch).
> @@ -635,12 +575,29 @@ MemoryDiscoveredPpiNotifyCallback (
>    if (IsIa32PaeSupported ()) {
>      Hob.Raw        = GetFirstGuidHob (&gEdkiiMigratedFvInfoGuid);
>      InitStackGuard = PcdGetBool (PcdCpuStackGuard);
>    }
> 
> -  if (InitStackGuard || (Hob.Raw != NULL)) {
> -    EnablePaging ();
> +  //
> +  // Some security features depends on the page table enabling. So, here
> +  // is to enable the page table if page table has not been enabled yet.
> +  // If page table is not enabled before this point, which means the system
> +  // IA-32e Mode is not activated. Because on Intel 64 processors, IA-32e Mode
> +  // operation requires physical address extensions with 4 or 5 levels of
> +  // enhanced paging structures (see Section 4.5, "4 - Level Paging and 5 -
> +  // Level Paging" and Section 9.8, "Initializing IA-32e Mode"). So, just
> +  // enable PAE page table if CR0.PG is not set.

7. Maybe simply say " Some security features depend on the page table enabling. So, here
is to enable paging if it is not enabled (only in 32bit mode)"

> +  //
> +  Cr0.UintN = AsmReadCr0 ();
> +  if ((Cr0.Bits.PG == 0) && (InitStackGuard || (Hob.Raw != NULL))) {
> +    ASSERT (sizeof (UINTN) == sizeof (UINT32));
> +
> +    Status = EnablePaePageTable ();
> +    if (EFI_ERROR (Status)) {
> +      DEBUG ((DEBUG_WARN, "MemoryDiscoveredPpiNotifyCallback: Failed to
> enable PAE page table: %r.\n", Status));
> +      ASSERT_EFI_ERROR (Status);

8. Shall we use "CpuDeadLoop()" if PAE paging enabling is failed? 

Re: [PATCH v2 3/5] MdeModulePkg/DxeIpl: Align Page table Level setting with previous level.

[Ni, Ray]

> +  //
> +  // Check cpu runs in 64bit long mode or 32bit protected mode.
1. this comment seems to be redundant.

> +  //
> +  if (sizeof (UINTN) == sizeof (UINT64)) {
> +    //
> +    // If cpu has already runned in 64bit long mode PEI, Page table Level in DXE
> must align with previous level.
2. runned -> run

> +    //
> +    Cr4.UintN         = AsmReadCr4 ();
> +    Page5LevelSupport = Cr4.Bits.LA57 ? TRUE : FALSE;

3. "Cr4.Bits.LA57 != 0"

> +    ASSERT (PcdGetBool (PcdUse5LevelPageTable) == Page5LevelSupport);

4. Good check!