[PATCH 00/14] Introduce Lazy-accept for Tdx guest

[PATCH 00/14] Introduce Lazy-accept for Tdx guest

[Min Xu]

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

UnacceptedMemory is one of the four defined types of TD memory in Intel
TDX guest. TDVF must invoke TDCALL [TDG.MEM.PAGE.ACCEPT] the unaccepted
memory before use it. See [TDVF] Section 7.1.
TDVF: https://www.intel.com/content/dam/develop/external/us/en/
documents/tdx-virtual-firmware-design-guide-rev-1.01.pdf 

It is a time-consuming task which impacts the boot performance badly.
One of the mitigation is the lazy-accept mechanism. That the whole system
memory is divided into 2 parts, one is accepted in bios phase, the other
is tagged as EfiGcdMemoryTypeUnaccepted and OS will handle these
"unaccepted" memories.
See "UEFI Spec v2.9 Table 7-5 Memory Type Usage before ExitBootServices()"

Patch 1-4: 
  Introduce lazy-accept related definitions.

Patch 5-6:
  Update Dxe and shell for unaccepted memory.

Patch 7 - 11:
  Update OvmfPkg for unaccepted memory.

Patch 12 - 13:
  Introduce EfiMemoryAcceptProtocol and realize it in TdxDxe.

Patch 14:
  Update Pool and Page functions to accept memory when OOM occurs.

Code: https://github.com/mxu9/edk2/tree/lazyaccept.v1

Cc: Zhichao Gao <zhichao.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>

Jiaqi Gao (2):
  MdePkg: The prototype definition of EfiMemoryAcceptProtocol
  MdeModulePkg: Pool and page functions accept memory when OOM occurs

Min M Xu (12):
  MdeModulePkg: Add PrePiHob.h
  MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE
  OvmfPkg: Use EFI_RESOURCE_MEMORY_UNACCEPTED which defined in
    MdeModulePkg
  MdePkg: Add UEFI Unaccepted memory definition
  MdeModulePkg: Update Dxe to handle unaccepted memory type
  ShellPkg: Update shell command memmap to show unaccepted memory
  OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.
  OvmfPkg: Use PcdOvmfWorkAreaBase instead of PcdSevEsWorkAreaBase
  OvmfPkg: Add MaxAcceptedMemoryAddress in TDX work area
  OvmfPkg: Introduce lazy accept in PlatformInitLib and PlatformPei
  OvmfPkg: Update ConstructFwHobList for lazy accept
  OvmfPkg: Realize EfiMemoryAcceptProtocol in TdxDxe

 MdeModulePkg/Core/Dxe/DxeMain.inf             |   1 +
 MdeModulePkg/Core/Dxe/Gcd/Gcd.c               |   5 +
 MdeModulePkg/Core/Dxe/Mem/Imem.h              |  16 ++
 MdeModulePkg/Core/Dxe/Mem/Page.c              | 218 ++++++++++++++++++
 MdeModulePkg/Core/Dxe/Mem/Pool.c              |  14 ++
 MdeModulePkg/Include/Pi/PrePiHob.h            |  20 ++
 MdePkg/Include/Pi/PiDxeCis.h                  |   5 +
 MdePkg/Include/Pi/PiHob.h                     |  11 +-
 MdePkg/Include/Protocol/MemoryAccept.h        |  37 +++
 MdePkg/Include/Uefi/UefiMultiPhase.h          |   5 +
 MdePkg/MdePkg.dec                             |   3 +
 OvmfPkg/Include/Library/PlatformInitLib.h     |   6 +
 OvmfPkg/Include/WorkArea.h                    |   1 +
 OvmfPkg/IntelTdx/IntelTdxX64.dsc              |   8 +
 .../PrePiHobListPointer.c                     |   4 +-
 .../PrePiHobListPointerLibTdx.inf             |   2 +-
 OvmfPkg/Library/PeilessStartupLib/Hob.c       |  26 ++-
 .../PeilessStartupLib/PeilessStartupLib.inf   |   1 +
 OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 145 +++++++++++-
 OvmfPkg/Library/PlatformInitLib/MemDetect.c   |  27 +++
 .../PlatformInitLib/PlatformInitLib.inf       |   1 +
 OvmfPkg/OvmfPkg.dec                           |   4 +
 OvmfPkg/OvmfPkgX64.dsc                        |   9 +
 OvmfPkg/PlatformPei/MemDetect.c               |   5 +
 OvmfPkg/TdxDxe/TdxDxe.c                       | 103 +++++++++
 OvmfPkg/TdxDxe/TdxDxe.inf                     |   2 +
 .../UefiShellDebug1CommandsLib/MemMap.c       |   6 +
 27 files changed, 666 insertions(+), 19 deletions(-)
 create mode 100644 MdeModulePkg/Include/Pi/PrePiHob.h
 create mode 100644 MdePkg/Include/Protocol/MemoryAccept.h

-- 
2.29.2.windows.2

[PATCH 01/14] MdeModulePkg: Add PrePiHob.h

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
But this defitinion has not been officially in the PI spec. Base
on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
MdeModulePkg/Include/Pi/PrePiHob.h.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdeModulePkg/Include/Pi/PrePiHob.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100644 MdeModulePkg/Include/Pi/PrePiHob.h

diff --git a/MdeModulePkg/Include/Pi/PrePiHob.h b/MdeModulePkg/Include/Pi/PrePiHob.h
new file mode 100644
index 000000000000..38a8f1be149d
--- /dev/null
+++ b/MdeModulePkg/Include/Pi/PrePiHob.h
@@ -0,0 +1,20 @@
+/** @file
+  HOB related definitions which has not been officially published in PI.
+
+Copyright (c) 2022, Intel Corporation. All rights reserved.<BR>
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef MDE_MODULEPKG_PRE_PI_HOB_H_
+#define MDE_MODULEPKG_PRE_PI_HOB_H_
+
+//
+// EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
+// But this defitinion has not been officially in the PI spec. Base
+// on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
+// MdeModulePkg/Include/Pi/PrePiHob.h.
+//
+#define EFI_RESOURCE_MEMORY_UNACCEPTED  0x00000007
+
+#endif
-- 
2.29.2.windows.2

[PATCH 02/14] MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
But this defitinion has not been officially in the PI spec. Base
on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
MdeModulePkg/Include/Pi/PrePiHob.h and update EFI_RESOURCE_MAX_MEMORY_TYPE
to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
in PI spec, we will re-visit here.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdePkg/Include/Pi/PiHob.h | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/MdePkg/Include/Pi/PiHob.h b/MdePkg/Include/Pi/PiHob.h
index e9f0ab4309d1..9af2e957fee5 100644
--- a/MdePkg/Include/Pi/PiHob.h
+++ b/MdePkg/Include/Pi/PiHob.h
@@ -232,7 +232,16 @@ typedef UINT32 EFI_RESOURCE_TYPE;
 #define EFI_RESOURCE_MEMORY_MAPPED_IO_PORT  0x00000004
 #define EFI_RESOURCE_MEMORY_RESERVED        0x00000005
 #define EFI_RESOURCE_IO_RESERVED            0x00000006
-#define EFI_RESOURCE_MAX_MEMORY_TYPE        0x00000007
+//
+// EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
+// But this defitinion has not been officially in the PI spec. Base
+// on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
+// MdeModulePkg/Include/Pi/PrePiHob.h and update EFI_RESOURCE_MAX_MEMORY_TYPE
+// to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
+// in PI spec, we will re-visit here.
+//
+// #define EFI_RESOURCE_MEMORY_UNACCEPTED      0x00000007
+#define EFI_RESOURCE_MAX_MEMORY_TYPE  0x00000008
 
 ///
 /// A type of recount attribute type.
-- 
2.29.2.windows.2

[PATCH 03/14] OvmfPkg: Use EFI_RESOURCE_MEMORY_UNACCEPTED which defined in MdeModulePkg

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

EFI_RESOURCE_MEMORY_UNACCEPTED is defined in MdeModulePkg. The files
which use the definition are updated as well.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/Library/PeilessStartupLib/Hob.c    | 3 +--
 OvmfPkg/Library/PlatformInitLib/IntelTdx.c | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/OvmfPkg/Library/PeilessStartupLib/Hob.c b/OvmfPkg/Library/PeilessStartupLib/Hob.c
index 5fc84a809025..a9b92b5fbaba 100644
--- a/OvmfPkg/Library/PeilessStartupLib/Hob.c
+++ b/OvmfPkg/Library/PeilessStartupLib/Hob.c
@@ -20,10 +20,9 @@
 #include <IndustryStandard/UefiTcgPlatform.h>
 #include <Library/PlatformInitLib.h>
 #include <OvmfPlatforms.h>
+#include <Pi/PrePiHob.h>
 #include "PeilessStartupInternal.h"
 
-#define EFI_RESOURCE_MEMORY_UNACCEPTED  7
-
 /**
  * Construct the HobList in SEC phase.
  *
diff --git a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
index c6d7c8bb6e0e..396b14d919d2 100644
--- a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
+++ b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
@@ -20,11 +20,11 @@
 #include <Library/PeiServicesLib.h>
 #include <Library/TdxLib.h>
 #include <Library/SynchronizationLib.h>
+#include <Pi/PrePiHob.h>
 #include <WorkArea.h>
 #include <ConfidentialComputingGuestAttr.h>
 
 #define ALIGNED_2MB_MASK                0x1fffff
-#define EFI_RESOURCE_MEMORY_UNACCEPTED  7
 
 /**
   This function will be called to accept pages. Only BSP accepts pages.
-- 
2.29.2.windows.2

[PATCH 04/14] MdePkg: Add UEFI Unaccepted memory definition

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

Plase refer to:
  UEFI Spec v2.9 Table 7-5 Memory Type Usage before ExitBootServices()

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdePkg/Include/Pi/PiDxeCis.h         | 5 +++++
 MdePkg/Include/Uefi/UefiMultiPhase.h | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/MdePkg/Include/Pi/PiDxeCis.h b/MdePkg/Include/Pi/PiDxeCis.h
index d0f2ed0e58df..67c99801b00d 100644
--- a/MdePkg/Include/Pi/PiDxeCis.h
+++ b/MdePkg/Include/Pi/PiDxeCis.h
@@ -56,6 +56,11 @@ typedef enum {
   /// system. If all memory has the same reliability, then this bit is not used.
   ///
   EfiGcdMemoryTypeMoreReliable,
+  ///
+  /// A memory region that describes system memory that has not been accepted
+  /// by a corresponding call to the underlying isolation architecture.
+  ///
+  EfiGcdMemoryTypeUnaccepted,
   EfiGcdMemoryTypeMaximum
 } EFI_GCD_MEMORY_TYPE;
 
diff --git a/MdePkg/Include/Uefi/UefiMultiPhase.h b/MdePkg/Include/Uefi/UefiMultiPhase.h
index 22bae43e36e8..29aabab5948e 100644
--- a/MdePkg/Include/Uefi/UefiMultiPhase.h
+++ b/MdePkg/Include/Uefi/UefiMultiPhase.h
@@ -103,6 +103,11 @@ typedef enum {
   /// however it happens to also support byte-addressable non-volatility.
   ///
   EfiPersistentMemory,
+  ///
+  /// A memory region that describes system memory that has not been accepted
+  /// by a corresponding call to the underlying isolation architecture.
+  ///
+  EfiUnacceptedMemory,
   EfiMaxMemoryType
 } EFI_MEMORY_TYPE;
 
-- 
2.29.2.windows.2

[PATCH 05/14] MdeModulePkg: Update Dxe to handle unaccepted memory type

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

Unaccepted memory is a kind of new memory type,
CoreInitializeGcdServices() and CoreGetMemoryMap() are updated to handle
the unaccepted memory type.

Ref: microsoft/mu_basecore@97e9c31

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdeModulePkg/Core/Dxe/Gcd/Gcd.c  |  5 +++++
 MdeModulePkg/Core/Dxe/Mem/Page.c | 28 ++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/MdeModulePkg/Core/Dxe/Gcd/Gcd.c b/MdeModulePkg/Core/Dxe/Gcd/Gcd.c
index 3763467bdbc0..b82fe11cb0f3 100644
--- a/MdeModulePkg/Core/Dxe/Gcd/Gcd.c
+++ b/MdeModulePkg/Core/Dxe/Gcd/Gcd.c
@@ -8,6 +8,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
 
 **/
 
+#include <Pi/PrePiHob.h>
 #include "DxeMain.h"
 #include "Gcd.h"
 #include "Mem/HeapGuard.h"
@@ -103,6 +104,7 @@ GLOBAL_REMOVE_IF_UNREFERENCED CONST CHAR8  *mGcdMemoryTypeNames[] = {
   "MMIO     ",  // EfiGcdMemoryTypeMemoryMappedIo
   "PersisMem",  // EfiGcdMemoryTypePersistent
   "MoreRelia",  // EfiGcdMemoryTypeMoreReliable
+  "Unaccepte",  // EfiGcdMemoryTypeUnaccepted
   "Unknown  "   // EfiGcdMemoryTypeMaximum
 };
 
@@ -2600,6 +2602,9 @@ CoreInitializeGcdServices (
         case EFI_RESOURCE_MEMORY_RESERVED:
           GcdMemoryType = EfiGcdMemoryTypeReserved;
           break;
+        case EFI_RESOURCE_MEMORY_UNACCEPTED:
+          GcdMemoryType = EfiGcdMemoryTypeUnaccepted;
+          break;
         case EFI_RESOURCE_IO:
           GcdIoType = EfiGcdIoTypeIo;
           break;
diff --git a/MdeModulePkg/Core/Dxe/Mem/Page.c b/MdeModulePkg/Core/Dxe/Mem/Page.c
index 47d4c5d92e15..008efdf397e0 100644
--- a/MdeModulePkg/Core/Dxe/Mem/Page.c
+++ b/MdeModulePkg/Core/Dxe/Mem/Page.c
@@ -61,6 +61,7 @@ EFI_MEMORY_TYPE_STATISTICS  mMemoryTypeStatistics[EfiMaxMemoryType + 1] = {
   { 0, MAX_ALLOC_ADDRESS, 0, 0, EfiMaxMemoryType, FALSE, FALSE },  // EfiMemoryMappedIOPortSpace
   { 0, MAX_ALLOC_ADDRESS, 0, 0, EfiMaxMemoryType, TRUE,  TRUE  },  // EfiPalCode
   { 0, MAX_ALLOC_ADDRESS, 0, 0, EfiMaxMemoryType, FALSE, FALSE },  // EfiPersistentMemory
+  { 0, MAX_ALLOC_ADDRESS, 0, 0, EfiMaxMemoryType, TRUE,  FALSE },  // EfiUnacceptedMemory
   { 0, MAX_ALLOC_ADDRESS, 0, 0, EfiMaxMemoryType, FALSE, FALSE }   // EfiMaxMemoryType
 };
 
@@ -83,6 +84,7 @@ EFI_MEMORY_TYPE_INFORMATION  gMemoryTypeInformation[EfiMaxMemoryType + 1] = {
   { EfiMemoryMappedIOPortSpace, 0 },
   { EfiPalCode,                 0 },
   { EfiPersistentMemory,        0 },
+  { EfiGcdMemoryTypeUnaccepted, 0 },
   { EfiMaxMemoryType,           0 }
 };
 //
@@ -1947,6 +1949,32 @@ CoreGetMemoryMap (
       MemoryMap = MergeMemoryMapDescriptor (MemoryMapStart, MemoryMap, Size);
     }
 
+    if (MergeGcdMapEntry.GcdMemoryType == EfiGcdMemoryTypeUnaccepted) {
+      //
+      // Page Align GCD range is required. When it is converted to EFI_MEMORY_DESCRIPTOR,
+      // it will be recorded as page PhysicalStart and NumberOfPages.
+      //
+      ASSERT ((MergeGcdMapEntry.BaseAddress & EFI_PAGE_MASK) == 0);
+      ASSERT (((MergeGcdMapEntry.EndAddress - MergeGcdMapEntry.BaseAddress + 1) & EFI_PAGE_MASK) == 0);
+
+      //
+      // Create EFI_MEMORY_DESCRIPTOR for every Unaccepted GCD entries
+      //
+      MemoryMap->PhysicalStart = MergeGcdMapEntry.BaseAddress;
+      MemoryMap->VirtualStart  = 0;
+      MemoryMap->NumberOfPages = RShiftU64 ((MergeGcdMapEntry.EndAddress - MergeGcdMapEntry.BaseAddress + 1), EFI_PAGE_SHIFT);
+      MemoryMap->Attribute     = MergeGcdMapEntry.Attributes |
+                                 (MergeGcdMapEntry.Capabilities & (EFI_MEMORY_RP | EFI_MEMORY_WP | EFI_MEMORY_XP | EFI_MEMORY_RO |
+                                                                   EFI_MEMORY_UC | EFI_MEMORY_UCE | EFI_MEMORY_WC | EFI_MEMORY_WT | EFI_MEMORY_WB));
+      MemoryMap->Type = EfiUnacceptedMemory;
+
+      //
+      // Check to see if the new Memory Map Descriptor can be merged with an
+      // existing descriptor if they are adjacent and have the same attributes
+      //
+      MemoryMap = MergeMemoryMapDescriptor (MemoryMapStart, MemoryMap, Size);
+    }
+
     if (Link == &mGcdMemorySpaceMap) {
       //
       // break loop when arrive at head.
-- 
2.29.2.windows.2

[PATCH 06/14] ShellPkg: Update shell command memmap to show unaccepted memory

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

ShellCommandRunMemMap() is updated to handle the case of unaccepted
memory type.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Zhichao Gao <zhichao.gao@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c b/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
index 72f3c58b0970..41994ae709ce 100644
--- a/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
+++ b/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
@@ -26,6 +26,7 @@ STATIC CONST CHAR16  NameEfiACPIMemoryNVS[]           = L"ACPIMemoryNVS";
 STATIC CONST CHAR16  NameEfiMemoryMappedIO[]          = L"MemoryMappedIO";
 STATIC CONST CHAR16  NameEfiMemoryMappedIOPortSpace[] = L"MemoryMappedIOPortSpace";
 STATIC CONST CHAR16  NameEfiPalCode[]                 = L"PalCode";
+STATIC CONST CHAR16  NameEfiUnacceptedMemory[]        = L"Unaccepted";
 
 //
 // Need short names for some memory types
@@ -303,6 +304,11 @@ ShellCommandRunMemMap (
               TotalPages   += Walker->NumberOfPages;
               PalCodePages += Walker->NumberOfPages;
               break;
+            case EfiUnacceptedMemory:
+              ShellPrintHiiEx (-1, -1, NULL, (EFI_STRING_ID)(!Sfo ? STRING_TOKEN (STR_MEMMAP_LIST_ITEM) : STRING_TOKEN (STR_MEMMAP_LIST_ITEM_SFO)), gShellDebug1HiiHandle, NameEfiUnacceptedMemory, Walker->PhysicalStart, Walker->PhysicalStart+MultU64x64 (SIZE_4KB, Walker->NumberOfPages)-1, Walker->NumberOfPages, Walker->Attribute);
+              TotalPages   += Walker->NumberOfPages;
+              PalCodePages += Walker->NumberOfPages;
+              break;
             default:
               //
               // Shell Spec defines the SFO format.
-- 
2.29.2.windows.2

[PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

Lazy accept page can be controlled in build time like below:
  -D LAZY_ACCEPT_PARTIAL_MEM=512

The unit is MB. If it is 0 then it means Lazy-accept is turned off.

Lazy-accept is turned off by default in OvmfPkgX64.
Lazy-accept is turned on with 512MB by default in IntelTdxX64.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/IntelTdx/IntelTdxX64.dsc | 8 ++++++++
 OvmfPkg/OvmfPkg.dec              | 4 ++++
 OvmfPkg/OvmfPkgX64.dsc           | 9 +++++++++
 3 files changed, 21 insertions(+)

diff --git a/OvmfPkg/IntelTdx/IntelTdxX64.dsc b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
index 43ab8bd089d9..c6195ab9c9d7 100644
--- a/OvmfPkg/IntelTdx/IntelTdxX64.dsc
+++ b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
@@ -62,6 +62,11 @@
   #
   DEFINE UP_CPU_DXE_GUID  = 6490f1c5-ebcc-4665-8892-0075b9bb49b7
 
+  #
+  # Define the size of lazy accepted memory. The unit is MB.
+  #
+  DEFINE LAZY_ACCEPT_PARTIAL_MEM = 512
+
 [BuildOptions]
   GCC:RELEASE_*_*_CC_FLAGS             = -DMDEPKG_NDEBUG
   INTEL:RELEASE_*_*_CC_FLAGS           = /D MDEPKG_NDEBUG
@@ -450,6 +455,9 @@
   # Point to the MdeModulePkg/Application/UiApp/UiApp.inf
   gEfiMdeModulePkgTokenSpaceGuid.PcdBootManagerMenuFile|{ 0x21, 0xaa, 0x2c, 0x46, 0x14, 0x76, 0x03, 0x45, 0x83, 0x6e, 0x8a, 0xb6, 0xf4, 0x66, 0x23, 0x31 }
 
+  # The partial memory size in Lazy accept
+  gUefiOvmfPkgTokenSpaceGuid.PcdLazyAcceptPartialMemorySize|$(LAZY_ACCEPT_PARTIAL_MEM)
+
 ################################################################################
 #
 # Pcd Dynamic Section - list of all EDK II PCD Entries defined by this Platform
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 5fe487f82d1a..7444765fe760 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -399,6 +399,10 @@
   ## The Tdx accept page size. 0x1000(4k),0x200000(2M)
   gUefiOvmfPkgTokenSpaceGuid.PcdTdxAcceptPageSize|0x200000|UINT32|0x65
 
+  ## The partial memory size in Lazy accept. Its unit is MB.
+  ## The default value is 0 which means lazy accept is turned off.
+  gUefiOvmfPkgTokenSpaceGuid.PcdLazyAcceptPartialMemorySize|0|UINT64|0x68
+
 [PcdsDynamic, PcdsDynamicEx]
   gUefiOvmfPkgTokenSpaceGuid.PcdEmuVariableEvent|0|UINT64|2
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashVariablesEnable|FALSE|BOOLEAN|0x10
diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc
index 1448f925b782..398981b6dad8 100644
--- a/OvmfPkg/OvmfPkgX64.dsc
+++ b/OvmfPkg/OvmfPkgX64.dsc
@@ -78,6 +78,12 @@
   DEFINE UP_CPU_PEI_GUID  = 280251c4-1d09-4035-9062-839acb5f18c1
   DEFINE UP_CPU_DXE_GUID  = 6490f1c5-ebcc-4665-8892-0075b9bb49b7
 
+  #
+  # Define the size of lazy accepted memory. The unit is MB.
+  # In OvmfPkgX64, the lazy accept page is disabled by default.
+  #
+  DEFINE LAZY_ACCEPT_PARTIAL_MEM = 0
+
 [BuildOptions]
   GCC:RELEASE_*_*_CC_FLAGS             = -DMDEPKG_NDEBUG
   INTEL:RELEASE_*_*_CC_FLAGS           = /D MDEPKG_NDEBUG
@@ -601,6 +607,9 @@
   # Point to the MdeModulePkg/Application/UiApp/UiApp.inf
   gEfiMdeModulePkgTokenSpaceGuid.PcdBootManagerMenuFile|{ 0x21, 0xaa, 0x2c, 0x46, 0x14, 0x76, 0x03, 0x45, 0x83, 0x6e, 0x8a, 0xb6, 0xf4, 0x66, 0x23, 0x31 }
 
+  # The partial memory size in Lazy accept
+  gUefiOvmfPkgTokenSpaceGuid.PcdLazyAcceptPartialMemorySize|$(LAZY_ACCEPT_PARTIAL_MEM)
+
 ################################################################################
 #
 # Pcd Dynamic Section - list of all EDK II PCD Entries defined by this Platform
-- 
2.29.2.windows.2

[PATCH 08/14] OvmfPkg: Use PcdOvmfWorkAreaBase instead of PcdSevEsWorkAreaBase

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

It is an typo error that HobList pointer should be stored at
PcdOvmfWorkAreaBase, not PcdSevEsWorkAreaBase.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 .../IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointer.c  | 4 ++--
 .../PrePiHobListPointerLibTdx/PrePiHobListPointerLibTdx.inf   | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointer.c b/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointer.c
index 1b41a3983c99..ecd271c0bdff 100644
--- a/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointer.c
+++ b/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointer.c
@@ -27,7 +27,7 @@ PrePeiGetHobList (
 {
   TDX_WORK_AREA  *TdxWorkArea;
 
-  TdxWorkArea = (TDX_WORK_AREA *)(UINTN)FixedPcdGet32 (PcdSevEsWorkAreaBase);
+  TdxWorkArea = (TDX_WORK_AREA *)(UINTN)FixedPcdGet32 (PcdOvmfWorkAreaBase);
   ASSERT (TdxWorkArea != NULL);
   ASSERT (TdxWorkArea->SecTdxWorkArea.HobList != 0);
 
@@ -48,7 +48,7 @@ PrePeiSetHobList (
 {
   TDX_WORK_AREA  *TdxWorkArea;
 
-  TdxWorkArea = (TDX_WORK_AREA *)(UINTN)FixedPcdGet32 (PcdSevEsWorkAreaBase);
+  TdxWorkArea = (TDX_WORK_AREA *)(UINTN)FixedPcdGet32 (PcdOvmfWorkAreaBase);
   ASSERT (TdxWorkArea != NULL);
 
   TdxWorkArea->SecTdxWorkArea.HobList = (UINTN)HobList;
diff --git a/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointerLibTdx.inf b/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointerLibTdx.inf
index 2667f841cea2..fe7ee23fdc8a 100644
--- a/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointerLibTdx.inf
+++ b/OvmfPkg/IntelTdx/PrePiHobListPointerLibTdx/PrePiHobListPointerLibTdx.inf
@@ -23,7 +23,7 @@
   UefiCpuPkg/UefiCpuPkg.dec
 
 [Pcd]
-  gUefiCpuPkgTokenSpaceGuid.PcdSevEsWorkAreaBase
+  gUefiOvmfPkgTokenSpaceGuid.PcdOvmfWorkAreaBase
 
 [LibraryClasses]
   PcdLib
-- 
2.29.2.windows.2

[PATCH 09/14] OvmfPkg: Add MaxAcceptedMemoryAddress in TDX work area

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

In lazy-accept the MaxAcceptedMemoryAddress is needed in
TransferTdxHobList and ContructFwHobList(at PeilessStartupLib). But in
SEC phase we cannot use a global variable to pass this value. So we add
a new field (MaxAcceptedMemoryAddress) in Tdx work area.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/Include/WorkArea.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/OvmfPkg/Include/WorkArea.h b/OvmfPkg/Include/WorkArea.h
index bf56fc4a6f65..2c2a5816b0dc 100644
--- a/OvmfPkg/Include/WorkArea.h
+++ b/OvmfPkg/Include/WorkArea.h
@@ -71,6 +71,7 @@ typedef struct _SEC_TDX_WORK_AREA {
   UINT32    PageTableReady;
   UINT32    Gpaw;
   UINT64    HobList;
+  UINT64    MaxAcceptedMemoryAddress;
 } SEC_TDX_WORK_AREA;
 
 typedef struct _TDX_WORK_AREA {
-- 
2.29.2.windows.2

[PATCH 10/14] OvmfPkg: Introduce lazy accept in PlatformInitLib and PlatformPei

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

There are below major changes in PlatformInitLib/PlatformPei
1. ProcessHobList
  The unaccepted memory is accepted if the accumulated accepted memory is
  smaller than the LazyAcceptMemSize. If a EFI_RESOURCE_MEMORY_UNACCEPTED
  hob is cross the LazyAcceptMemSize, it will be split into 2 parts and
  only the left one is accepted. The max accepted memory address is
  stored in Tdx workarea which will be used in TransferTdxHobList.

2. TransferTdxHobList
  Transfer the unaccepted memory hob to EFI_RESOURCE_SYSTEM_MEMORY hob
  if it is accepted. As it is mentioned in 1), there may be a
  EFI_RESOURCE_MEMORY_UNACCEPTED hob which only part of the memory
  describes in the hob is accepted. We also handles this situation
  in TransferTdxHobList.

3. PlatformAdjustSystemMemorySizeBelow4gbForLazyAccep
  The system memory size below 4GB may be larger than the accepted
  memory. This function is used to handle this situation.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/Include/Library/PlatformInitLib.h     |   6 +
 OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 143 ++++++++++++++++--
 OvmfPkg/Library/PlatformInitLib/MemDetect.c   |  27 ++++
 .../PlatformInitLib/PlatformInitLib.inf       |   1 +
 OvmfPkg/PlatformPei/MemDetect.c               |   5 +
 5 files changed, 171 insertions(+), 11 deletions(-)

diff --git a/OvmfPkg/Include/Library/PlatformInitLib.h b/OvmfPkg/Include/Library/PlatformInitLib.h
index 2987a367cc9c..187efcf34e14 100644
--- a/OvmfPkg/Include/Library/PlatformInitLib.h
+++ b/OvmfPkg/Include/Library/PlatformInitLib.h
@@ -144,6 +144,12 @@ PlatformGetSystemMemorySizeBelow4gb (
   IN EFI_HOB_PLATFORM_INFO  *PlatformInfoHob
   );
 
+UINT32
+EFIAPI
+PlatformAdjustSystemMemorySizeBelow4gbForLazyAccept (
+  IN UINT32  LowerMemorySize
+  );
+
 /**
   Initialize the PhysMemAddressWidth field in PlatformInfoHob based on guest RAM size.
 **/
diff --git a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
index 396b14d919d2..7be466ee34d1 100644
--- a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
+++ b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
@@ -24,7 +24,8 @@
 #include <WorkArea.h>
 #include <ConfidentialComputingGuestAttr.h>
 
-#define ALIGNED_2MB_MASK                0x1fffff
+#define ALIGNED_2MB_MASK  0x1fffff
+#define MEGABYTE_SHIFT    20
 
 /**
   This function will be called to accept pages. Only BSP accepts pages.
@@ -375,15 +376,33 @@ ProcessHobList (
   EFI_STATUS            Status;
   EFI_PEI_HOB_POINTERS  Hob;
   EFI_PHYSICAL_ADDRESS  PhysicalEnd;
+  TDX_WORK_AREA         *WorkArea;
+  UINT64                ResourceLength;
+  UINT64                AccumulateAcceptedMemory;
+  UINT64                LazyAcceptMemSize;
+  UINT64                MaxAcceptedMemoryAddress;
 
   Status = EFI_SUCCESS;
   ASSERT (VmmHobList != NULL);
   Hob.Raw = (UINT8 *)VmmHobList;
 
+  AccumulateAcceptedMemory = 0;
+  MaxAcceptedMemoryAddress = 0;
+  LazyAcceptMemSize        = FixedPcdGet64 (PcdLazyAcceptPartialMemorySize);
+  //
+  // If specified accept size is zero, accept all of the memory.
+  // Else transfer the size in megabyte to the number in byte.
+  //
+  if (LazyAcceptMemSize == 0) {
+    LazyAcceptMemSize = MAX_UINT64;
+  } else {
+    LazyAcceptMemSize <<= MEGABYTE_SHIFT;
+  }
+
   //
   // Parse the HOB list until end of list or matching type is found.
   //
-  while (!END_OF_HOB_LIST (Hob)) {
+  while (!END_OF_HOB_LIST (Hob) && AccumulateAcceptedMemory < LazyAcceptMemSize) {
     if (Hob.Header->HobType == EFI_HOB_TYPE_RESOURCE_DESCRIPTOR) {
       DEBUG ((DEBUG_INFO, "\nResourceType: 0x%x\n", Hob.ResourceDescriptor->ResourceType));
 
@@ -393,7 +412,17 @@ ProcessHobList (
         DEBUG ((DEBUG_INFO, "ResourceLength: 0x%llx\n", Hob.ResourceDescriptor->ResourceLength));
         DEBUG ((DEBUG_INFO, "Owner: %g\n\n", &Hob.ResourceDescriptor->Owner));
 
-        PhysicalEnd = Hob.ResourceDescriptor->PhysicalStart + Hob.ResourceDescriptor->ResourceLength;
+        PhysicalEnd    = Hob.ResourceDescriptor->PhysicalStart + Hob.ResourceDescriptor->ResourceLength;
+        ResourceLength = Hob.ResourceDescriptor->ResourceLength;
+
+        if (AccumulateAcceptedMemory + ResourceLength > LazyAcceptMemSize) {
+          //
+          // If the memory can't be accepted completely, accept the part of it to meet the
+          // PcdLazyAcceptPartialMemorySize.
+          //
+          ResourceLength = LazyAcceptMemSize - AccumulateAcceptedMemory;
+          PhysicalEnd    = Hob.ResourceDescriptor->PhysicalStart + ResourceLength;
+        }
 
         Status = BspAcceptMemoryResourceRange (
                    Hob.ResourceDescriptor->PhysicalStart,
@@ -402,12 +431,26 @@ ProcessHobList (
         if (EFI_ERROR (Status)) {
           break;
         }
+
+        AccumulateAcceptedMemory += ResourceLength;
+        MaxAcceptedMemoryAddress  = PhysicalEnd;
       }
     }
 
     Hob.Raw = GET_NEXT_HOB (Hob);
   }
 
+  //
+  // Record MaxAcceptedMemoryAddress in OvmfWorkArea.
+  // This information is needed in TransferTdxHobList and ContructFwHobList(at PeilessStartupLib).
+  // But in SEC phase we cannot use a global variable to pass this value. So it has
+  // to be stored in OvmfWorkarea.
+  //
+  WorkArea = (TDX_WORK_AREA *)FixedPcdGet32 (PcdOvmfWorkAreaBase);
+  ASSERT (WorkArea != NULL);
+  ASSERT (WorkArea->Header.GuestType == CcGuestTypeIntelTdx);
+  WorkArea->SecTdxWorkArea.MaxAcceptedMemoryAddress = MaxAcceptedMemoryAddress;
+
   return Status;
 }
 
@@ -460,6 +503,74 @@ ProcessTdxHobList (
   return Status;
 }
 
+/**
+ * Build ResourceDescriptorHob for the unaccepted memory region.
+ * This memory region may be splitted into 2 parts because of lazy accept.
+ *
+ * @param Hob     Point to the EFI_HOB_RESOURCE_DESCRIPTOR
+ * @param MaxAcceptedMemoryAddress The max accepted memory address
+ * @return VOID
+ */
+VOID
+BuildResourceDescriptorHobForUnacceptedMemory (
+  IN EFI_HOB_RESOURCE_DESCRIPTOR  *Hob,
+  IN UINT64                       MaxAcceptedMemoryAddress
+  )
+{
+  EFI_PHYSICAL_ADDRESS         PhysicalStart;
+  EFI_PHYSICAL_ADDRESS         PhysicalEnd;
+  UINT64                       ResourceLength;
+  EFI_RESOURCE_TYPE            ResourceType;
+  EFI_RESOURCE_ATTRIBUTE_TYPE  ResourceAttribute;
+  UINT64                       AcceptedResourceLength;
+
+  ASSERT (Hob->ResourceType == EFI_RESOURCE_MEMORY_UNACCEPTED);
+
+  ResourceType      = EFI_RESOURCE_MEMORY_UNACCEPTED;
+  ResourceAttribute = Hob->ResourceAttribute;
+  PhysicalStart     = Hob->PhysicalStart;
+  ResourceLength    = Hob->ResourceLength;
+  PhysicalEnd       = PhysicalStart + ResourceLength;
+
+  if (PhysicalEnd <= MaxAcceptedMemoryAddress) {
+    //
+    // This memory region has been accepted.
+    //
+    ResourceType       = EFI_RESOURCE_SYSTEM_MEMORY;
+    ResourceAttribute |= (EFI_RESOURCE_ATTRIBUTE_PRESENT | EFI_RESOURCE_ATTRIBUTE_INITIALIZED | EFI_RESOURCE_ATTRIBUTE_TESTED);
+  } else if (PhysicalStart >= MaxAcceptedMemoryAddress) {
+    //
+    // This memory region hasn't been accepted.
+    // So keep the ResourceType and ResourceAttribute unchange.
+    //
+  } else {
+    //
+    // This memory region is splitted into 2 parts:
+    // the accepted and unaccepted.
+    //
+    AcceptedResourceLength = MaxAcceptedMemoryAddress - Hob->PhysicalStart;
+
+    // We build the ResourceDescriptorHob for the accepted part.
+    // The unaccepted part will be build out side the if-else block.
+    BuildResourceDescriptorHob (
+      EFI_RESOURCE_SYSTEM_MEMORY,
+      ResourceAttribute | (EFI_RESOURCE_ATTRIBUTE_PRESENT | EFI_RESOURCE_ATTRIBUTE_INITIALIZED | EFI_RESOURCE_ATTRIBUTE_TESTED),
+      Hob->PhysicalStart,
+      AcceptedResourceLength
+      );
+
+    PhysicalStart   = Hob->PhysicalStart + AcceptedResourceLength;
+    ResourceLength -= AcceptedResourceLength;
+  }
+
+  BuildResourceDescriptorHob (
+    ResourceType,
+    ResourceAttribute,
+    PhysicalStart,
+    ResourceLength
+    );
+}
+
 /**
   Transfer the incoming HobList for the TD to the final HobList for Dxe.
   The Hobs transferred in this function are ResourceDescriptor hob and
@@ -477,6 +588,16 @@ TransferTdxHobList (
   EFI_PEI_HOB_POINTERS         Hob;
   EFI_RESOURCE_TYPE            ResourceType;
   EFI_RESOURCE_ATTRIBUTE_TYPE  ResourceAttribute;
+  UINT64                       MaxAcceptedMemoryAddress;
+  TDX_WORK_AREA                *WorkArea;
+
+  WorkArea = (TDX_WORK_AREA *)FixedPcdGet32 (PcdOvmfWorkAreaBase);
+  ASSERT (WorkArea != NULL);
+  ASSERT (WorkArea->Header.GuestType == CcGuestTypeIntelTdx);
+  MaxAcceptedMemoryAddress = WorkArea->SecTdxWorkArea.MaxAcceptedMemoryAddress;
+  if (MaxAcceptedMemoryAddress == 0) {
+    MaxAcceptedMemoryAddress = MAX_UINT64;
+  }
 
   //
   // PcdOvmfSecGhcbBase is used as the TD_HOB in Tdx guest.
@@ -489,16 +610,16 @@ TransferTdxHobList (
         ResourceAttribute = Hob.ResourceDescriptor->ResourceAttribute;
 
         if (ResourceType == EFI_RESOURCE_MEMORY_UNACCEPTED) {
-          ResourceType       = EFI_RESOURCE_SYSTEM_MEMORY;
-          ResourceAttribute |= (EFI_RESOURCE_ATTRIBUTE_PRESENT | EFI_RESOURCE_ATTRIBUTE_INITIALIZED | EFI_RESOURCE_ATTRIBUTE_TESTED);
+          BuildResourceDescriptorHobForUnacceptedMemory (Hob.ResourceDescriptor, MaxAcceptedMemoryAddress);
+        } else {
+          BuildResourceDescriptorHob (
+            ResourceType,
+            ResourceAttribute,
+            Hob.ResourceDescriptor->PhysicalStart,
+            Hob.ResourceDescriptor->ResourceLength
+            );
         }
 
-        BuildResourceDescriptorHob (
-          ResourceType,
-          ResourceAttribute,
-          Hob.ResourceDescriptor->PhysicalStart,
-          Hob.ResourceDescriptor->ResourceLength
-          );
         break;
       case EFI_HOB_TYPE_MEMORY_ALLOCATION:
         BuildMemoryAllocationHob (
diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index c28d7601f87e..2bb346f31d44 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -41,6 +41,8 @@ Module Name:
 
 #include <Library/PlatformInitLib.h>
 
+#define MEGABYTE_SHIFT  20
+
 VOID
 EFIAPI
 PlatformQemuUc32BaseInitialization (
@@ -288,6 +290,31 @@ GetHighestSystemMemoryAddressFromPvhMemmap (
   return HighestAddress;
 }
 
+UINT32
+EFIAPI
+PlatformAdjustSystemMemorySizeBelow4gbForLazyAccept (
+  IN UINT32  LowerMemorySize
+  )
+{
+ #ifdef MDE_CPU_X64
+  UINT64  LazyAcceptMemSize;
+
+  LazyAcceptMemSize = FixedPcdGet64 (PcdLazyAcceptPartialMemorySize);
+  //
+  // If specified accept size is not zero,
+  // transfer the size in megabyte to the number in byte.
+  //
+  if (LazyAcceptMemSize != 0) {
+    LazyAcceptMemSize <<= MEGABYTE_SHIFT;
+    if (LazyAcceptMemSize < LowerMemorySize) {
+      LowerMemorySize = (UINT32)(UINTN)LazyAcceptMemSize;
+    }
+  }
+
+ #endif
+  return LowerMemorySize;
+}
+
 UINT32
 EFIAPI
 PlatformGetSystemMemorySizeBelow4gb (
diff --git a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
index d2a0bec43452..affb2fb90408 100644
--- a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
+++ b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
@@ -95,6 +95,7 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdGuidedExtractHandlerTableSize
 
   gUefiOvmfPkgTokenSpaceGuid.PcdTdxAcceptPageSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdLazyAcceptPartialMemorySize
 
 [FeaturePcd]
   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode
diff --git a/OvmfPkg/PlatformPei/MemDetect.c b/OvmfPkg/PlatformPei/MemDetect.c
index 2e47b1322990..acc1d7f63ee8 100644
--- a/OvmfPkg/PlatformPei/MemDetect.c
+++ b/OvmfPkg/PlatformPei/MemDetect.c
@@ -279,6 +279,11 @@ PublishPeiMemory (
     LowerMemorySize -= mPlatformInfoHob.Q35TsegMbytes * SIZE_1MB;
   }
 
+  //
+  // Adjustment for Lazy accept because it may accept part of the memory.
+  //
+  LowerMemorySize = PlatformAdjustSystemMemorySizeBelow4gbForLazyAccept (LowerMemorySize);
+
   S3AcpiReservedMemoryBase = 0;
   S3AcpiReservedMemorySize = 0;
 
-- 
2.29.2.windows.2

[PATCH 11/14] OvmfPkg: Update ConstructFwHobList for lazy accept

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

In TDVF the hob list is constructed at the memory region which is the
largest one below 4GB. After lazy accept is introduced, the
MaxAcceptedMemoryAddress (which is tha max accpeted memory address in
lazy accept) should be considered.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/Library/PeilessStartupLib/Hob.c       | 23 ++++++++++++++++++-
 .../PeilessStartupLib/PeilessStartupLib.inf   |  1 +
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/OvmfPkg/Library/PeilessStartupLib/Hob.c b/OvmfPkg/Library/PeilessStartupLib/Hob.c
index a9b92b5fbaba..884490af68fd 100644
--- a/OvmfPkg/Library/PeilessStartupLib/Hob.c
+++ b/OvmfPkg/Library/PeilessStartupLib/Hob.c
@@ -21,6 +21,7 @@
 #include <Library/PlatformInitLib.h>
 #include <OvmfPlatforms.h>
 #include <Pi/PrePiHob.h>
+#include <WorkArea.h>
 #include "PeilessStartupInternal.h"
 
 /**
@@ -74,10 +75,13 @@ ConstructFwHobList (
   )
 {
   EFI_PEI_HOB_POINTERS  Hob;
+  EFI_PHYSICAL_ADDRESS  PhysicalStart;
   EFI_PHYSICAL_ADDRESS  PhysicalEnd;
   UINT64                ResourceLength;
   EFI_PHYSICAL_ADDRESS  LowMemoryStart;
   UINT64                LowMemoryLength;
+  UINT64                MaxAcceptedMemoryAddress;
+  TDX_WORK_AREA         *WorkArea;
 
   ASSERT (VmmHobList != NULL);
 
@@ -86,14 +90,31 @@ ConstructFwHobList (
   LowMemoryLength = 0;
   LowMemoryStart  = 0;
 
+  WorkArea = (TDX_WORK_AREA *)FixedPcdGet32 (PcdOvmfWorkAreaBase);
+  ASSERT (WorkArea != NULL);
+  ASSERT (WorkArea->Header.GuestType == CcGuestTypeIntelTdx);
+  MaxAcceptedMemoryAddress = WorkArea->SecTdxWorkArea.MaxAcceptedMemoryAddress;
+  if (MaxAcceptedMemoryAddress == 0) {
+    MaxAcceptedMemoryAddress = MAX_UINT64;
+  }
+
   //
   // Parse the HOB list until end of list or matching type is found.
   //
   while (!END_OF_HOB_LIST (Hob)) {
     if (Hob.Header->HobType == EFI_HOB_TYPE_RESOURCE_DESCRIPTOR) {
-      if (Hob.ResourceDescriptor->ResourceType == EFI_RESOURCE_MEMORY_UNACCEPTED) {
+      if ((Hob.ResourceDescriptor->ResourceType == EFI_RESOURCE_MEMORY_UNACCEPTED) && (Hob.ResourceDescriptor->PhysicalStart < MaxAcceptedMemoryAddress)) {
         PhysicalEnd    = Hob.ResourceDescriptor->PhysicalStart + Hob.ResourceDescriptor->ResourceLength;
         ResourceLength = Hob.ResourceDescriptor->ResourceLength;
+        PhysicalStart  = Hob.ResourceDescriptor->PhysicalStart;
+
+        if ((PhysicalEnd >= MaxAcceptedMemoryAddress) && (PhysicalStart < MaxAcceptedMemoryAddress)) {
+          //
+          // This memory region is split into 2 parts. The left part is accepted.
+          //
+          PhysicalEnd    = MaxAcceptedMemoryAddress;
+          ResourceLength = PhysicalEnd - PhysicalStart;
+        }
 
         if (PhysicalEnd <= BASE_4GB) {
           if (ResourceLength > LowMemoryLength) {
diff --git a/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf b/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
index c5d291f02bcd..2ca49324d8f9 100644
--- a/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
+++ b/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
@@ -88,3 +88,4 @@
   gEfiMdeModulePkgTokenSpaceGuid.PcdNullPointerDetectionPropertyMask    ## CONSUMES
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvBase
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdOvmfWorkAreaBase
-- 
2.29.2.windows.2

[PATCH 12/14] MdePkg: The prototype definition of EfiMemoryAcceptProtocol

[Min Xu]

From: Jiaqi Gao <jiaqi.gao@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

EfiMemoryAcceptProtocol is defined in MdePkg, the method AcceptMemory()
can be called when memory needs to be accepted.

EfiMemoryAcceptProtocol can be installed by architecture-specific drivers
such as TdxDxe.This allows different isolation architectures to realize
their own low-level methods to accept memory.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdePkg/Include/Protocol/MemoryAccept.h | 37 ++++++++++++++++++++++++++
 MdePkg/MdePkg.dec                      |  3 +++
 2 files changed, 40 insertions(+)
 create mode 100644 MdePkg/Include/Protocol/MemoryAccept.h

diff --git a/MdePkg/Include/Protocol/MemoryAccept.h b/MdePkg/Include/Protocol/MemoryAccept.h
new file mode 100644
index 000000000000..27b43ce81126
--- /dev/null
+++ b/MdePkg/Include/Protocol/MemoryAccept.h
@@ -0,0 +1,37 @@
+/** @file
+  The file provides the protocol to provide interface to accept memory.
+
+  Copyright (c) 2021 - 2022, Intel Corporation. All rights reserved.<BR>
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#ifndef MEMORY_ACCEPT_H_
+#define MEMORY_ACCEPT_H_
+
+#define EFI_MEMORY_ACCEPT_PROTOCOL_GUID \
+  { 0x38c74800, 0x5590, 0x4db4, { 0xa0, 0xf3, 0x67, 0x5d, 0x9b, 0x8e, 0x80, 0x26 } };
+
+typedef struct _EFI_MEMORY_ACCEPT_PROTOCOL EFI_MEMORY_ACCEPT_PROTOCOL;
+
+/**
+  @param This                   A pointer to a MEMORY_ACCEPT_PROTOCOL.
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_ACCEPT_MEMORY)(
+  IN  EFI_MEMORY_ACCEPT_PROTOCOL    *This,
+  IN  EFI_PHYSICAL_ADDRESS          StartAddress,
+  IN  UINTN                         Size
+  );
+
+///
+/// The MEMORY_ACCEPT_PROTOCOL provides the ability for memory services
+/// to accept memory.
+///
+struct _EFI_MEMORY_ACCEPT_PROTOCOL {
+  EFI_ACCEPT_MEMORY    AcceptMemory;
+};
+
+extern EFI_GUID  gEfiMemoryAcceptProtocolGuid;
+
+#endif
diff --git a/MdePkg/MdePkg.dec b/MdePkg/MdePkg.dec
index f1ebf9e251c1..de3c56758b95 100644
--- a/MdePkg/MdePkg.dec
+++ b/MdePkg/MdePkg.dec
@@ -1019,6 +1019,9 @@
   gEfiPeiDelayedDispatchPpiGuid  = { 0x869c711d, 0x649c, 0x44fe, { 0x8b, 0x9e, 0x2c, 0xbb, 0x29, 0x11, 0xc3, 0xe6 }}
 
 [Protocols]
+  ## Include/Protocol/MemoryAccept.h
+  gEfiMemoryAcceptProtocolGuid   = { 0x38c74800, 0x5590, 0x4db4, { 0xa0, 0xf3, 0x67, 0x5d, 0x9b, 0x8e, 0x80, 0x26 }}
+
   ## Include/Protocol/Pcd.h
   gPcdProtocolGuid               = { 0x11B34006, 0xD85B, 0x4D0A, { 0xA2, 0x90, 0xD5, 0xA5, 0x71, 0x31, 0x0E, 0xF7 }}
 
-- 
2.29.2.windows.2

[PATCH 13/14] OvmfPkg: Realize EfiMemoryAcceptProtocol in TdxDxe

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

Memory usage may exceed the amount accepted at the begining (SEC), TDVF
needs to accept memory dynamically when OUT_OF_RESOURCE occurs.

EfiMemoryAcceptProtocol is defined in MdePkg and is implementated /
Installed in TdxDxe for Intel TDX memory acceptance.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/TdxDxe/TdxDxe.c   | 103 ++++++++++++++++++++++++++++++++++++++
 OvmfPkg/TdxDxe/TdxDxe.inf |   2 +
 2 files changed, 105 insertions(+)

diff --git a/OvmfPkg/TdxDxe/TdxDxe.c b/OvmfPkg/TdxDxe/TdxDxe.c
index 2318db989792..7cc9bf5f8c98 100644
--- a/OvmfPkg/TdxDxe/TdxDxe.c
+++ b/OvmfPkg/TdxDxe/TdxDxe.c
@@ -24,6 +24,7 @@
 #include <Library/HobLib.h>
 #include <Protocol/Cpu.h>
 #include <Protocol/MpInitLibDepProtocols.h>
+#include <Protocol/MemoryAccept.h>
 #include <Library/UefiBootServicesTableLib.h>
 #include <ConfidentialComputingGuestAttr.h>
 #include <IndustryStandard/Tdx.h>
@@ -32,6 +33,95 @@
 #include <TdxAcpiTable.h>
 #include <Library/MemEncryptTdxLib.h>
 
+#define ALIGNED_2MB_MASK  0x1fffff
+EFI_HANDLE  mTdxDxeHandle = NULL;
+
+EFI_STATUS
+EFIAPI
+TdxMemoryAccept (
+  IN EFI_MEMORY_ACCEPT_PROTOCOL  *This,
+  IN EFI_PHYSICAL_ADDRESS        StartAddress,
+  IN UINTN                       Size
+  )
+{
+  EFI_STATUS  Status;
+  UINT32      AcceptPageSize;
+  UINT64      StartAddress1;
+  UINT64      StartAddress2;
+  UINT64      StartAddress3;
+  UINT64      Length1;
+  UINT64      Length2;
+  UINT64      Length3;
+  UINT64      Pages;
+
+  AcceptPageSize = FixedPcdGet32 (PcdTdxAcceptPageSize);
+  StartAddress1  = 0;
+  StartAddress2  = 0;
+  StartAddress3  = 0;
+  Length1        = 0;
+  Length2        = 0;
+  Length3        = 0;
+
+  if (Size == 0) {
+    return EFI_SUCCESS;
+  }
+
+  if (ALIGN_VALUE (StartAddress, SIZE_2MB) != StartAddress) {
+    StartAddress1 = StartAddress;
+    Length1       = ALIGN_VALUE (StartAddress, SIZE_2MB) - StartAddress;
+    if (Length1 >= Size) {
+      Length1 = Size;
+    }
+
+    StartAddress += Length1;
+    Size         -= Length1;
+  }
+
+  if (Size > SIZE_2MB) {
+    StartAddress2 = StartAddress;
+    Length2       = Size & ~(UINT64)ALIGNED_2MB_MASK;
+    StartAddress += Length2;
+    Size         -= Length2;
+  }
+
+  if (Size) {
+    StartAddress3 = StartAddress;
+    Length3       = Size;
+  }
+
+  Status = EFI_SUCCESS;
+  if (Length1 > 0) {
+    Pages  = Length1 / SIZE_4KB;
+    Status = TdAcceptPages (StartAddress1, Pages, SIZE_4KB);
+    if (EFI_ERROR (Status)) {
+      return Status;
+    }
+  }
+
+  if (Length2 > 0) {
+    Pages  = Length2 / AcceptPageSize;
+    Status = TdAcceptPages (StartAddress2, Pages, AcceptPageSize);
+    if (EFI_ERROR (Status)) {
+      return Status;
+    }
+  }
+
+  if (Length3 > 0) {
+    Pages  = Length3 / SIZE_4KB;
+    Status = TdAcceptPages (StartAddress3, Pages, SIZE_4KB);
+    ASSERT (!EFI_ERROR (Status));
+    if (EFI_ERROR (Status)) {
+      return Status;
+    }
+  }
+
+  return Status;
+}
+
+EFI_MEMORY_ACCEPT_PROTOCOL  mMemoryAcceptProtocol = {
+  TdxMemoryAccept
+};
+
 VOID
 SetPcdSettings (
   EFI_HOB_PLATFORM_INFO  *PlatformInfoHob
@@ -277,6 +367,19 @@ TdxDxeEntryPoint (
          NULL
          );
 
+  //
+  // Install MemoryAccept protocol for TDX
+  //
+  Status = gBS->InstallProtocolInterface (
+                  &mTdxDxeHandle,
+                  &gEfiMemoryAcceptProtocolGuid,
+                  EFI_NATIVE_INTERFACE,
+                  &mMemoryAcceptProtocol
+                  );
+  if (EFI_ERROR (Status)) {
+    DEBUG ((DEBUG_ERROR, "Install EfiMemoryAcceptProtocol failed.\n"));
+  }
+
   //
   // Call TDINFO to get actual number of cpus in domain
   //
diff --git a/OvmfPkg/TdxDxe/TdxDxe.inf b/OvmfPkg/TdxDxe/TdxDxe.inf
index a7e0abda1522..e24a000f768c 100644
--- a/OvmfPkg/TdxDxe/TdxDxe.inf
+++ b/OvmfPkg/TdxDxe/TdxDxe.inf
@@ -52,6 +52,7 @@
   gEfiAcpiTableProtocolGuid                        ## CONSUMES
   gEfiMpInitLibMpDepProtocolGuid
   gEfiMpInitLibUpDepProtocolGuid
+  gEfiMemoryAcceptProtocolGuid
 
 [Pcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdPciIoBase
@@ -68,3 +69,4 @@
   gEfiMdePkgTokenSpaceGuid.PcdConfidentialComputingGuestAttr
   gEfiMdeModulePkgTokenSpaceGuid.PcdTdxSharedBitMask
   gEfiMdeModulePkgTokenSpaceGuid.PcdSetNxForStack
+  gUefiOvmfPkgTokenSpaceGuid.PcdTdxAcceptPageSize
-- 
2.29.2.windows.2

[PATCH 14/14] MdeModulePkg: Pool and page functions accept memory when OOM occurs

[Min Xu]

From: Jiaqi Gao <jiaqi.gao@intel.com>

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937

When CoreAllocatePages()/CoreAllocatePool() meets error of
EFI_OUT_OF_RESOURCES, locate the EfiMemoryAcceptProtocol and accept extra
memory dynamically.

Firstly, find the unaccpeted memory region with enough size in GCD
entries. Then locate the EfiMemoryAcceptProtocol and accept the memory.
Finally, update the GCD memory and gMemoryMap entries.

After updating the memory infomation, CoreInternalAllocatePages()/
CoreInternalAllocatePool() will be recalled to allocate pages/pool.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Dandan Bi <dandan.bi@intel.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 MdeModulePkg/Core/Dxe/DxeMain.inf |   1 +
 MdeModulePkg/Core/Dxe/Mem/Imem.h  |  16 +++
 MdeModulePkg/Core/Dxe/Mem/Page.c  | 190 ++++++++++++++++++++++++++++++
 MdeModulePkg/Core/Dxe/Mem/Pool.c  |  14 +++
 4 files changed, 221 insertions(+)

diff --git a/MdeModulePkg/Core/Dxe/DxeMain.inf b/MdeModulePkg/Core/Dxe/DxeMain.inf
index e4bca895773d..4b6c627a2cde 100644
--- a/MdeModulePkg/Core/Dxe/DxeMain.inf
+++ b/MdeModulePkg/Core/Dxe/DxeMain.inf
@@ -169,6 +169,7 @@
   gEfiVariableArchProtocolGuid                  ## CONSUMES
   gEfiCapsuleArchProtocolGuid                   ## CONSUMES
   gEfiWatchdogTimerArchProtocolGuid             ## CONSUMES
+  gEfiMemoryAcceptProtocolGuid                  ## CONSUMES
 
 [Pcd]
   gEfiMdeModulePkgTokenSpaceGuid.PcdLoadFixAddressBootTimeCodePageNumber    ## SOMETIMES_CONSUMES
diff --git a/MdeModulePkg/Core/Dxe/Mem/Imem.h b/MdeModulePkg/Core/Dxe/Mem/Imem.h
index 2f0bf2bf631f..22e0d0e44030 100644
--- a/MdeModulePkg/Core/Dxe/Mem/Imem.h
+++ b/MdeModulePkg/Core/Dxe/Mem/Imem.h
@@ -47,6 +47,22 @@ typedef struct {
 // Internal prototypes
 //
 
+/**
+  Internal function.  Used by the pool and page functions to accept memory
+  when OOM occurs.
+
+  @param  Type                   The type of allocation to perform.
+  @param  AcceptSize             Size of memory to be accepted.
+  @param  Memory                 Accept memory address
+
+**/
+EFI_STATUS
+AcceptMemoryResource (
+  IN EFI_ALLOCATE_TYPE         Type,
+  IN UINTN                     AcceptSize,
+  IN OUT EFI_PHYSICAL_ADDRESS  *Memory
+  );
+
 /**
   Internal function.  Used by the pool functions to allocate pages
   to back pool allocation requests.
diff --git a/MdeModulePkg/Core/Dxe/Mem/Page.c b/MdeModulePkg/Core/Dxe/Mem/Page.c
index 008efdf397e0..0584373d68be 100644
--- a/MdeModulePkg/Core/Dxe/Mem/Page.c
+++ b/MdeModulePkg/Core/Dxe/Mem/Page.c
@@ -9,6 +9,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
 #include "DxeMain.h"
 #include "Imem.h"
 #include "HeapGuard.h"
+#include <Protocol/MemoryAccept.h>
 
 //
 // Entry for tracking the memory regions for each memory type to coalesce similar memory types
@@ -378,6 +379,176 @@ CoreFreeMemoryMapStack (
   mFreeMapStack -= 1;
 }
 
+/**
+  Used to accept memory when OOM occurs.
+
+  @param  Type                   The type of allocation to perform.
+  @param  AcceptSize             Size of memory to be accepted.
+  @param  Memory                 Accept memory address
+
+**/
+EFI_STATUS
+AcceptMemoryResource (
+  IN EFI_ALLOCATE_TYPE         Type,
+  IN UINTN                     AcceptSize,
+  IN OUT EFI_PHYSICAL_ADDRESS  *Memory
+  )
+{
+ #ifdef MDE_CPU_X64
+
+  LIST_ENTRY                  *Link;
+  EFI_GCD_MAP_ENTRY           *GcdEntry;
+  EFI_GCD_MAP_ENTRY           UnacceptedEntry;
+  EFI_MEMORY_ACCEPT_PROTOCOL  *MemoryAcceptProtocol;
+  UINTN                       Start;
+  UINTN                       End;
+  EFI_STATUS                  Status;
+
+  //
+  // We accept n*32MB at one time to improve the efficiency.
+  //
+  AcceptSize = (AcceptSize + SIZE_32MB - 1) & ~(SIZE_32MB - 1);
+
+  if (AcceptSize == 0) {
+    return EFI_INVALID_PARAMETER;
+  }
+
+  Status = gBS->LocateProtocol (&gEfiMemoryAcceptProtocolGuid, NULL, (VOID **)&MemoryAcceptProtocol);
+  if (EFI_ERROR (Status)) {
+    return EFI_UNSUPPORTED;
+  }
+
+  if (Type == AllocateAddress) {
+    Start = *Memory;
+    End   = *Memory + AcceptSize;
+  }
+
+  if (Type == AllocateMaxAddress) {
+    if (*Memory < EFI_PAGE_MASK) {
+      return EFI_INVALID_PARAMETER;
+    }
+
+    if ((*Memory & EFI_PAGE_MASK) != EFI_PAGE_MASK) {
+      //
+      // Change MaxAddress to be 1 page lower
+      //
+      *Memory -= EFI_PAGE_SIZE;
+
+      //
+      // Set MaxAddress to a page boundary
+      //
+      *Memory &= ~(UINT64)EFI_PAGE_MASK;
+
+      //
+      // Set MaxAddress to end of the page
+      //
+      *Memory |= EFI_PAGE_MASK;
+    }
+  }
+
+  //
+  // Traverse the mGcdMemorySpaceMap to find out the unaccepted
+  // memory entry with big enough size.
+  //
+  Link = mGcdMemorySpaceMap.ForwardLink;
+  while (Link != &mGcdMemorySpaceMap) {
+    GcdEntry = CR (Link, EFI_GCD_MAP_ENTRY, Link, EFI_GCD_MAP_SIGNATURE);
+    if (GcdEntry->GcdMemoryType == EfiGcdMemoryTypeUnaccepted) {
+      if (Type == AllocateMaxAddress) {
+        if (GcdEntry->BaseAddress + AcceptSize - 1 > *Memory) {
+          Link = Link->ForwardLink;
+          continue;
+        }
+      } else if (Type == AllocateAddress) {
+        if ((GcdEntry->BaseAddress > *Memory) || (GcdEntry->EndAddress < *Memory + AcceptSize - 1)) {
+          Link = Link->ForwardLink;
+          continue;
+        }
+      }
+
+      //
+      // Is the entry big enough?
+      //
+      if (AcceptSize <= GcdEntry->EndAddress - GcdEntry->BaseAddress + 1) {
+        UnacceptedEntry = *GcdEntry;
+        if (Type != AllocateAddress) {
+          Start = UnacceptedEntry.BaseAddress;
+          End   = UnacceptedEntry.BaseAddress + AcceptSize - 1;
+        }
+
+        break;
+      }
+    }
+
+    Link = Link->ForwardLink;
+  }
+
+  if (Link == &mGcdMemorySpaceMap) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+
+  //
+  // Accept memory using the interface provide by the protocol.
+  //
+  Status = MemoryAcceptProtocol->AcceptMemory (MemoryAcceptProtocol, Start, AcceptSize);
+  if (EFI_ERROR (Status)) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+
+  //
+  // If memory is accepted successfully, remove the target memory space from GCD.
+  //
+  CoreRemoveMemorySpace (UnacceptedEntry.BaseAddress, UnacceptedEntry.EndAddress - UnacceptedEntry.BaseAddress + 1);
+
+  //
+  // Add the remain lower part of unaccepted memory to the
+  // Gcd memory space and memory map.
+  //
+  if (Start > UnacceptedEntry.BaseAddress) {
+    CoreAddMemorySpace (
+      EfiGcdMemoryTypeUnaccepted,
+      UnacceptedEntry.BaseAddress,
+      Start - UnacceptedEntry.BaseAddress,
+      UnacceptedEntry.Capabilities
+      );
+  }
+
+  //
+  // Update accepted part of the memory entry to type of EfiGcdMemoryTypeSystemMemory
+  // and add the range to the memory map.
+  //
+  CoreAddMemorySpace (
+    EfiGcdMemoryTypeSystemMemory,
+    Start,
+    AcceptSize,
+    //
+    // Hardcode memory space attributes.
+    //
+    EFI_MEMORY_CPU_CRYPTO | EFI_MEMORY_XP | EFI_MEMORY_RO | EFI_MEMORY_RP
+    );
+
+  //
+  // Add the remain higher part of unaccepted memory to the
+  // Gcd memory space and memory map.
+  //
+  if (UnacceptedEntry.EndAddress > End) {
+    CoreAddMemorySpace (
+      EfiGcdMemoryTypeUnaccepted,
+      End + 1,
+      UnacceptedEntry.EndAddress - End,
+      UnacceptedEntry.Capabilities
+      );
+  }
+
+  return EFI_SUCCESS;
+
+ #else
+
+  return EFI_UNSUPPORTED;
+
+ #endif
+}
+
 /**
   Find untested but initialized memory regions in GCD map and convert them to be DXE allocatable.
 
@@ -1471,6 +1642,25 @@ CoreAllocatePages (
                 Memory,
                 NeedGuard
                 );
+ #ifdef MDE_CPU_X64
+
+  if (Status == EFI_OUT_OF_RESOURCES) {
+    Status = AcceptMemoryResource (Type, NumberOfPages << EFI_PAGE_SHIFT, Memory);
+    if (!EFI_ERROR (Status)) {
+      Status = CoreInternalAllocatePages (
+                 Type,
+                 MemoryType,
+                 NumberOfPages,
+                 Memory,
+                 NeedGuard
+                 );
+    } else {
+      Status = EFI_OUT_OF_RESOURCES;
+    }
+  }
+
+ #endif
+
   if (!EFI_ERROR (Status)) {
     CoreUpdateProfile (
       (EFI_PHYSICAL_ADDRESS)(UINTN)RETURN_ADDRESS (0),
diff --git a/MdeModulePkg/Core/Dxe/Mem/Pool.c b/MdeModulePkg/Core/Dxe/Mem/Pool.c
index 7aaf501600cf..9e8c8611c1ef 100644
--- a/MdeModulePkg/Core/Dxe/Mem/Pool.c
+++ b/MdeModulePkg/Core/Dxe/Mem/Pool.c
@@ -273,6 +273,20 @@ CoreAllocatePool (
   EFI_STATUS  Status;
 
   Status = CoreInternalAllocatePool (PoolType, Size, Buffer);
+
+ #ifdef MDE_CPU_X64
+
+  if (Status == EFI_OUT_OF_RESOURCES) {
+    Status = AcceptMemoryResource (AllocateAnyPages, Size, NULL);
+    if (!EFI_ERROR (Status)) {
+      Status = CoreInternalAllocatePool (PoolType, Size, Buffer);
+    } else {
+      Status = EFI_OUT_OF_RESOURCES;
+    }
+  }
+
+ #endif
+
   if (!EFI_ERROR (Status)) {
     CoreUpdateProfile (
       (EFI_PHYSICAL_ADDRESS)(UINTN)RETURN_ADDRESS (0),
-- 
2.29.2.windows.2

Re: [PATCH 06/14] ShellPkg: Update shell command memmap to show unaccepted memory

[Ni, Ray]

Reviewed-by: Ray Ni <ray.ni@intel.com>

> -----Original Message-----
> From: Xu, Min M <min.m.xu@intel.com>
> Sent: Monday, June 6, 2022 11:00 AM
> To: devel@edk2.groups.io
> Cc: Xu, Min M <min.m.xu@intel.com>; Ni, Ray <ray.ni@intel.com>; Gao, Zhichao <zhichao.gao@intel.com>; Aktas, Erdem
> <erdemaktas@google.com>; Gerd Hoffmann <kraxel@redhat.com>; James Bottomley <jejb@linux.ibm.com>; Yao, Jiewen
> <jiewen.yao@intel.com>; Tom Lendacky <thomas.lendacky@amd.com>
> Subject: [PATCH 06/14] ShellPkg: Update shell command memmap to show unaccepted memory
> 
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> ShellCommandRunMemMap() is updated to handle the case of unaccepted
> memory type.
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Zhichao Gao <zhichao.gao@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> ---
>  ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
> b/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
> index 72f3c58b0970..41994ae709ce 100644
> --- a/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
> +++ b/ShellPkg/Library/UefiShellDebug1CommandsLib/MemMap.c
> @@ -26,6 +26,7 @@ STATIC CONST CHAR16  NameEfiACPIMemoryNVS[]           = L"ACPIMemoryNVS";
>  STATIC CONST CHAR16  NameEfiMemoryMappedIO[]          = L"MemoryMappedIO";
>  STATIC CONST CHAR16  NameEfiMemoryMappedIOPortSpace[] = L"MemoryMappedIOPortSpace";
>  STATIC CONST CHAR16  NameEfiPalCode[]                 = L"PalCode";
> +STATIC CONST CHAR16  NameEfiUnacceptedMemory[]        = L"Unaccepted";
> 
>  //
>  // Need short names for some memory types
> @@ -303,6 +304,11 @@ ShellCommandRunMemMap (
>                TotalPages   += Walker->NumberOfPages;
>                PalCodePages += Walker->NumberOfPages;
>                break;
> +            case EfiUnacceptedMemory:
> +              ShellPrintHiiEx (-1, -1, NULL, (EFI_STRING_ID)(!Sfo ? STRING_TOKEN (STR_MEMMAP_LIST_ITEM) : STRING_TOKEN
> (STR_MEMMAP_LIST_ITEM_SFO)), gShellDebug1HiiHandle, NameEfiUnacceptedMemory, Walker->PhysicalStart, Walker-
> >PhysicalStart+MultU64x64 (SIZE_4KB, Walker->NumberOfPages)-1, Walker->NumberOfPages, Walker->Attribute);
> +              TotalPages   += Walker->NumberOfPages;
> +              PalCodePages += Walker->NumberOfPages;
> +              break;
>              default:
>                //
>                // Shell Spec defines the SFO format.
> --
> 2.29.2.windows.2

Re: [PATCH 06/14] ShellPkg: Update shell command memmap to show unaccepted memory

[Min Xu]

On June 6, 2022 11:02 AM, Ni Ray wrote:
> 
> Reviewed-by: Ray Ni <ray.ni@intel.com>
> 
Ah my bad, there is another ShellPkg patch that I forgot to submit. I will submit it in v2.

Re: [PATCH 12/14] MdePkg: The prototype definition of EfiMemoryAcceptProtocol

[Yao, Jiewen]

Hi
I don't remember it is accepted in UEFI protocol.
As such, we cannot use EFI_ prefix, and we cannot put it to MdePkg.

I recommend we use EDKII_ prefix, and move it to MdeModulePkg as first step.

Thank you
Yao Jiewen


> -----Original Message-----
> From: Xu, Min M <min.m.xu@intel.com>
> Sent: Monday, June 6, 2022 11:00 AM
> To: devel@edk2.groups.io
> Cc: Gao, Jiaqi <jiaqi.gao@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Liu,
> Zhiguang <zhiguang.liu@intel.com>; Aktas, Erdem <erdemaktas@google.com>;
> Gerd Hoffmann <kraxel@redhat.com>; James Bottomley <jejb@linux.ibm.com>;
> Yao, Jiewen <jiewen.yao@intel.com>; Tom Lendacky
> <thomas.lendacky@amd.com>; Xu, Min M <min.m.xu@intel.com>
> Subject: [PATCH 12/14] MdePkg: The prototype definition of
> EfiMemoryAcceptProtocol
> 
> From: Jiaqi Gao <jiaqi.gao@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> EfiMemoryAcceptProtocol is defined in MdePkg, the method AcceptMemory()
> can be called when memory needs to be accepted.
> 
> EfiMemoryAcceptProtocol can be installed by architecture-specific drivers
> such as TdxDxe.This allows different isolation architectures to realize
> their own low-level methods to accept memory.
> 
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Zhiguang Liu <zhiguang.liu@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> ---
>  MdePkg/Include/Protocol/MemoryAccept.h | 37
> ++++++++++++++++++++++++++
>  MdePkg/MdePkg.dec                      |  3 +++
>  2 files changed, 40 insertions(+)
>  create mode 100644 MdePkg/Include/Protocol/MemoryAccept.h
> 
> diff --git a/MdePkg/Include/Protocol/MemoryAccept.h
> b/MdePkg/Include/Protocol/MemoryAccept.h
> new file mode 100644
> index 000000000000..27b43ce81126
> --- /dev/null
> +++ b/MdePkg/Include/Protocol/MemoryAccept.h
> @@ -0,0 +1,37 @@
> +/** @file
> +  The file provides the protocol to provide interface to accept memory.
> +
> +  Copyright (c) 2021 - 2022, Intel Corporation. All rights reserved.<BR>
> +  SPDX-License-Identifier: BSD-2-Clause-Patent
> +**/
> +
> +#ifndef MEMORY_ACCEPT_H_
> +#define MEMORY_ACCEPT_H_
> +
> +#define EFI_MEMORY_ACCEPT_PROTOCOL_GUID \
> +  { 0x38c74800, 0x5590, 0x4db4, { 0xa0, 0xf3, 0x67, 0x5d, 0x9b, 0x8e, 0x80,
> 0x26 } };
> +
> +typedef struct _EFI_MEMORY_ACCEPT_PROTOCOL
> EFI_MEMORY_ACCEPT_PROTOCOL;
> +
> +/**
> +  @param This                   A pointer to a MEMORY_ACCEPT_PROTOCOL.
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_ACCEPT_MEMORY)(
> +  IN  EFI_MEMORY_ACCEPT_PROTOCOL    *This,
> +  IN  EFI_PHYSICAL_ADDRESS          StartAddress,
> +  IN  UINTN                         Size
> +  );
> +
> +///
> +/// The MEMORY_ACCEPT_PROTOCOL provides the ability for memory services
> +/// to accept memory.
> +///
> +struct _EFI_MEMORY_ACCEPT_PROTOCOL {
> +  EFI_ACCEPT_MEMORY    AcceptMemory;
> +};
> +
> +extern EFI_GUID  gEfiMemoryAcceptProtocolGuid;
> +
> +#endif
> diff --git a/MdePkg/MdePkg.dec b/MdePkg/MdePkg.dec
> index f1ebf9e251c1..de3c56758b95 100644
> --- a/MdePkg/MdePkg.dec
> +++ b/MdePkg/MdePkg.dec
> @@ -1019,6 +1019,9 @@
>    gEfiPeiDelayedDispatchPpiGuid  = { 0x869c711d, 0x649c, 0x44fe, { 0x8b, 0x9e,
> 0x2c, 0xbb, 0x29, 0x11, 0xc3, 0xe6 }}
> 
>  [Protocols]
> +  ## Include/Protocol/MemoryAccept.h
> +  gEfiMemoryAcceptProtocolGuid   = { 0x38c74800, 0x5590, 0x4db4, { 0xa0,
> 0xf3, 0x67, 0x5d, 0x9b, 0x8e, 0x80, 0x26 }}
> +
>    ## Include/Protocol/Pcd.h
>    gPcdProtocolGuid               = { 0x11B34006, 0xD85B, 0x4D0A, { 0xA2, 0x90,
> 0xD5, 0xA5, 0x71, 0x31, 0x0E, 0xF7 }}
> 
> --
> 2.29.2.windows.2

Re: [PATCH 02/14] MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE

[Yao, Jiewen]

Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>

> -----Original Message-----
> From: Xu, Min M <min.m.xu@intel.com>
> Sent: Monday, June 6, 2022 11:00 AM
> To: devel@edk2.groups.io
> Cc: Xu, Min M <min.m.xu@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Liu,
> Zhiguang <zhiguang.liu@intel.com>; Aktas, Erdem <erdemaktas@google.com>;
> Gerd Hoffmann <kraxel@redhat.com>; James Bottomley <jejb@linux.ibm.com>;
> Yao, Jiewen <jiewen.yao@intel.com>; Tom Lendacky
> <thomas.lendacky@amd.com>
> Subject: [PATCH 02/14] MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE
> 
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
> But this defitinion has not been officially in the PI spec. Base
> on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
> MdeModulePkg/Include/Pi/PrePiHob.h and update
> EFI_RESOURCE_MAX_MEMORY_TYPE
> to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
> in PI spec, we will re-visit here.
> 
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Zhiguang Liu <zhiguang.liu@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> ---
>  MdePkg/Include/Pi/PiHob.h | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/MdePkg/Include/Pi/PiHob.h b/MdePkg/Include/Pi/PiHob.h
> index e9f0ab4309d1..9af2e957fee5 100644
> --- a/MdePkg/Include/Pi/PiHob.h
> +++ b/MdePkg/Include/Pi/PiHob.h
> @@ -232,7 +232,16 @@ typedef UINT32 EFI_RESOURCE_TYPE;
>  #define EFI_RESOURCE_MEMORY_MAPPED_IO_PORT  0x00000004
>  #define EFI_RESOURCE_MEMORY_RESERVED        0x00000005
>  #define EFI_RESOURCE_IO_RESERVED            0x00000006
> -#define EFI_RESOURCE_MAX_MEMORY_TYPE        0x00000007
> +//
> +// EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
> +// But this defitinion has not been officially in the PI spec. Base
> +// on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
> +// MdeModulePkg/Include/Pi/PrePiHob.h and update
> EFI_RESOURCE_MAX_MEMORY_TYPE
> +// to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
> +// in PI spec, we will re-visit here.
> +//
> +// #define EFI_RESOURCE_MEMORY_UNACCEPTED      0x00000007
> +#define EFI_RESOURCE_MAX_MEMORY_TYPE  0x00000008
> 
>  ///
>  /// A type of recount attribute type.
> --
> 2.29.2.windows.2

Re: [PATCH 04/14] MdePkg: Add UEFI Unaccepted memory definition

[Yao, Jiewen]

Reviewed-by: Jiewen Yao <Jiewen.yao@intel.com>

> -----Original Message-----
> From: Xu, Min M <min.m.xu@intel.com>
> Sent: Monday, June 6, 2022 11:00 AM
> To: devel@edk2.groups.io
> Cc: Xu, Min M <min.m.xu@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Liu,
> Zhiguang <zhiguang.liu@intel.com>; Aktas, Erdem <erdemaktas@google.com>;
> Gerd Hoffmann <kraxel@redhat.com>; James Bottomley <jejb@linux.ibm.com>;
> Yao, Jiewen <jiewen.yao@intel.com>; Tom Lendacky
> <thomas.lendacky@amd.com>
> Subject: [PATCH 04/14] MdePkg: Add UEFI Unaccepted memory definition
> 
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> Plase refer to:
>   UEFI Spec v2.9 Table 7-5 Memory Type Usage before ExitBootServices()
> 
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Zhiguang Liu <zhiguang.liu@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> ---
>  MdePkg/Include/Pi/PiDxeCis.h         | 5 +++++
>  MdePkg/Include/Uefi/UefiMultiPhase.h | 5 +++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/MdePkg/Include/Pi/PiDxeCis.h b/MdePkg/Include/Pi/PiDxeCis.h
> index d0f2ed0e58df..67c99801b00d 100644
> --- a/MdePkg/Include/Pi/PiDxeCis.h
> +++ b/MdePkg/Include/Pi/PiDxeCis.h
> @@ -56,6 +56,11 @@ typedef enum {
>    /// system. If all memory has the same reliability, then this bit is not used.
>    ///
>    EfiGcdMemoryTypeMoreReliable,
> +  ///
> +  /// A memory region that describes system memory that has not been
> accepted
> +  /// by a corresponding call to the underlying isolation architecture.
> +  ///
> +  EfiGcdMemoryTypeUnaccepted,
>    EfiGcdMemoryTypeMaximum
>  } EFI_GCD_MEMORY_TYPE;
> 
> diff --git a/MdePkg/Include/Uefi/UefiMultiPhase.h
> b/MdePkg/Include/Uefi/UefiMultiPhase.h
> index 22bae43e36e8..29aabab5948e 100644
> --- a/MdePkg/Include/Uefi/UefiMultiPhase.h
> +++ b/MdePkg/Include/Uefi/UefiMultiPhase.h
> @@ -103,6 +103,11 @@ typedef enum {
>    /// however it happens to also support byte-addressable non-volatility.
>    ///
>    EfiPersistentMemory,
> +  ///
> +  /// A memory region that describes system memory that has not been
> accepted
> +  /// by a corresponding call to the underlying isolation architecture.
> +  ///
> +  EfiUnacceptedMemory,
>    EfiMaxMemoryType
>  } EFI_MEMORY_TYPE;
> 
> --
> 2.29.2.windows.2

Re: [PATCH 02/14] MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE

[Gerd Hoffmann]

On Mon, Jun 06, 2022 at 10:59:50AM +0800, Min Xu wrote:
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
> But this defitinion has not been officially in the PI spec. Base
> on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
> MdeModulePkg/Include/Pi/PrePiHob.h and update EFI_RESOURCE_MAX_MEMORY_TYPE
> to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
> in PI spec, we will re-visit here.
> 
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Zhiguang Liu <zhiguang.liu@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> ---
>  MdePkg/Include/Pi/PiHob.h | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/MdePkg/Include/Pi/PiHob.h b/MdePkg/Include/Pi/PiHob.h
> index e9f0ab4309d1..9af2e957fee5 100644
> --- a/MdePkg/Include/Pi/PiHob.h
> +++ b/MdePkg/Include/Pi/PiHob.h
> @@ -232,7 +232,16 @@ typedef UINT32 EFI_RESOURCE_TYPE;
>  #define EFI_RESOURCE_MEMORY_MAPPED_IO_PORT  0x00000004
>  #define EFI_RESOURCE_MEMORY_RESERVED        0x00000005
>  #define EFI_RESOURCE_IO_RESERVED            0x00000006
> -#define EFI_RESOURCE_MAX_MEMORY_TYPE        0x00000007
> +//
> +// EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted memory.
> +// But this defitinion has not been officially in the PI spec. Base
> +// on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
> +// MdeModulePkg/Include/Pi/PrePiHob.h and update EFI_RESOURCE_MAX_MEMORY_TYPE
> +// to 8. After EFI_RESOURCE_MEMORY_UNACCEPTED is officially published
> +// in PI spec, we will re-visit here.
> +//
> +// #define EFI_RESOURCE_MEMORY_UNACCEPTED      0x00000007
> +#define EFI_RESOURCE_MAX_MEMORY_TYPE  0x00000008

Shouldn't this be handled in PrePiHob.h too?
i.e. update EFI_RESOURCE_MAX_MEMORY_TYPE with #undef + #define there?

take care,
  Gerd

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Gerd Hoffmann]

On Mon, Jun 06, 2022 at 10:59:55AM +0800, Min Xu wrote:
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> Lazy accept page can be controlled in build time like below:
>   -D LAZY_ACCEPT_PARTIAL_MEM=512
> 
> The unit is MB. If it is 0 then it means Lazy-accept is turned off.
> 
> Lazy-accept is turned off by default in OvmfPkgX64.
> Lazy-accept is turned on with 512MB by default in IntelTdxX64.

Care to explain?  What is the point in not using lazy accept in OvmfPkgX64?
Also what exactly does this option mean?  Is this the minimum amount of
memory accepted by the firmware?

take care,
  Gerd

Re: [PATCH 08/14] OvmfPkg: Use PcdOvmfWorkAreaBase instead of PcdSevEsWorkAreaBase

[Gerd Hoffmann]

On Mon, Jun 06, 2022 at 10:59:56AM +0800, Min Xu wrote:
> From: Min M Xu <min.m.xu@intel.com>
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> It is an typo error that HobList pointer should be stored at
> PcdOvmfWorkAreaBase, not PcdSevEsWorkAreaBase.

How does that relate to lazy accept?

To me this looks like an independent bugfix which should be submitted
separately.

take care,
  Gerd

Re: [edk2-devel] [PATCH 02/14] MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE

[Min Xu]

On June 7, 2022 6:40 PM, Gerd Hoffmann wrote:
> On Mon, Jun 06, 2022 at 10:59:50AM +0800, Min Xu wrote:
> > --- a/MdePkg/Include/Pi/PiHob.h
> > +++ b/MdePkg/Include/Pi/PiHob.h
> > @@ -232,7 +232,16 @@ typedef UINT32 EFI_RESOURCE_TYPE;  #define
> > EFI_RESOURCE_MEMORY_MAPPED_IO_PORT  0x00000004
> >  #define EFI_RESOURCE_MEMORY_RESERVED        0x00000005
> >  #define EFI_RESOURCE_IO_RESERVED            0x00000006
> > -#define EFI_RESOURCE_MAX_MEMORY_TYPE        0x00000007
> > +//
> > +// EFI_RESOURCE_MEMORY_UNACCEPTED is defined for unaccepted
> memory.
> > +// But this defitinion has not been officially in the PI spec. Base
> > +// on the code-first we define EFI_RESOURCE_MEMORY_UNACCEPTED at
> //
> > +MdeModulePkg/Include/Pi/PrePiHob.h and update
> > +EFI_RESOURCE_MAX_MEMORY_TYPE // to 8. After
> > +EFI_RESOURCE_MEMORY_UNACCEPTED is officially published // in PI
> spec, we will re-visit here.
> > +//
> > +// #define EFI_RESOURCE_MEMORY_UNACCEPTED      0x00000007
> > +#define EFI_RESOURCE_MAX_MEMORY_TYPE  0x00000008
> 
> Shouldn't this be handled in PrePiHob.h too?
> i.e. update EFI_RESOURCE_MAX_MEMORY_TYPE with #undef + #define
> there?
> 
@Gao, Liming is the MdePkg maintainer. What's your thought, Liming?

Thanks
Min

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

On June 7, 2022 6:46 PM, Gerd Hoffmann wrote:
> On Mon, Jun 06, 2022 at 10:59:55AM +0800, Min Xu wrote:
> > From: Min M Xu <min.m.xu@intel.com>
> >
> > RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> >
> > Lazy accept page can be controlled in build time like below:
> >   -D LAZY_ACCEPT_PARTIAL_MEM=512
> >
> > The unit is MB. If it is 0 then it means Lazy-accept is turned off.
> >
> > Lazy-accept is turned off by default in OvmfPkgX64.
> > Lazy-accept is turned on with 512MB by default in IntelTdxX64.
> 
> Care to explain?  What is the point in not using lazy accept in OvmfPkgX64?
It's an Open to the community that if lazy accept should be enabled in OvmfPkgX64. I would like to hear the suggestions/comments from you guys. So as the first step, it is disabled.
> Also what exactly does this option mean?  Is this the minimum amount of
> memory accepted by the firmware?
Yes, this option defines the minimum amount of memory accepted by the firmware. For example, if it is 512MB, then there will be such amount memory accepted by the firmware before jump to OS.

Thanks 
Min

Re: [PATCH 08/14] OvmfPkg: Use PcdOvmfWorkAreaBase instead of PcdSevEsWorkAreaBase

[Min Xu]

On June 7, 2022 6:48 PM, Gerd Hoffmann wrote:
> On Mon, Jun 06, 2022 at 10:59:56AM +0800, Min Xu wrote:
> > From: Min M Xu <min.m.xu@intel.com>
> >
> > RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> >
> > It is an typo error that HobList pointer should be stored at
> > PcdOvmfWorkAreaBase, not PcdSevEsWorkAreaBase.
> 
> How does that relate to lazy accept?
> 
> To me this looks like an independent bugfix which should be submitted
> separately.
You're right. It is an independent bugfix. I found it during the lazy-accept integration test. I will submit a separate patch. Thanks for the reminder.

Min

Re: [PATCH 12/14] MdePkg: The prototype definition of EfiMemoryAcceptProtocol

[Min Xu]

On June 6, 2022 11:22 AM, Yao Jiewen wrote:
> 
> Hi
> I don't remember it is accepted in UEFI protocol.
> As such, we cannot use EFI_ prefix, and we cannot put it to MdePkg.
> 
> I recommend we use EDKII_ prefix, and move it to MdeModulePkg as first
> step.
> 
Thanks for the reminder. It will be updated in the next version.

Thanks
Min

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Gerd Hoffmann]

On Wed, Jun 08, 2022 at 12:06:28AM +0000, Xu, Min M wrote:
> On June 7, 2022 6:46 PM, Gerd Hoffmann wrote:
> > On Mon, Jun 06, 2022 at 10:59:55AM +0800, Min Xu wrote:
> > > From: Min M Xu <min.m.xu@intel.com>
> > >
> > > RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> > >
> > > Lazy accept page can be controlled in build time like below:
> > >   -D LAZY_ACCEPT_PARTIAL_MEM=512
> > >
> > > The unit is MB. If it is 0 then it means Lazy-accept is turned off.
> > >
> > > Lazy-accept is turned off by default in OvmfPkgX64.
> > > Lazy-accept is turned on with 512MB by default in IntelTdxX64.
> > 
> > Care to explain?  What is the point in not using lazy accept in OvmfPkgX64?

> It's an Open to the community that if lazy accept should be enabled in
> OvmfPkgX64. I would like to hear the suggestions/comments from you
> guys. So as the first step, it is disabled.

I think OvmfPkgX64 and IntelTdxX64 configuration should be identical,
and I'm wondering whenever this should be a compile time option in the
first place.  Can we simply use lazy accept unconditionally?

With the edk2 memory management code being able to handle unaccepted
memory I just don't see the point in supporting different options in
early firmware setup.  I think sec/pei should accept the memory needed
to run dxe and not more.  Anything beyond that can be accepted later on
demand via TdxDxe.

> > Also what exactly does this option mean?  Is this the minimum amount of
> > memory accepted by the firmware?

> Yes, this option defines the minimum amount of memory accepted by the
> firmware. For example, if it is 512MB, then there will be such amount
> memory accepted by the firmware before jump to OS.

Where does the 512MB figure come from?

Using a fixed amount of memory doesn't look very robust to me.  Is it
possible to accept memory when the guest calls ExitBootServices?  That
way we could guarantee a minimum amount of *free* accepted memory being
available, to make sure the linux kernel has enough memory to decompress
itself, allocate memory management data structures and get its own lazy
accept support going.

For guests which don't support lazy accept we could offer a (runtime)
option to simply accept all memory in ExitBootServices.

take care,
  Gerd

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Dionna Glaze]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

Since there will be guest OSes that don't support unaccepted memory type (at least initially), and what will be run won't be known at firmware build time, can PcdLazyAcceptPartialMemorySize be initialized by a QemuFwCfg option instead of a FixedPcd? I started a different thread on this, but perhaps it was lost.

[-- Attachment #2: Type: text/html, Size: 368 bytes --]

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

On June 8, 2022 2:18 PM, Gerd Hoffmann wrote:
> On Wed, Jun 08, 2022 at 12:06:28AM +0000, Xu, Min M wrote:
> > On June 7, 2022 6:46 PM, Gerd Hoffmann wrote:
> > > On Mon, Jun 06, 2022 at 10:59:55AM +0800, Min Xu wrote:
> > > > From: Min M Xu <min.m.xu@intel.com>
> > > >
> > > > RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> > > >
> > > > Lazy accept page can be controlled in build time like below:
> > > >   -D LAZY_ACCEPT_PARTIAL_MEM=512
> > > >
> > > > The unit is MB. If it is 0 then it means Lazy-accept is turned off.
> > > >
> > > > Lazy-accept is turned off by default in OvmfPkgX64.
> > > > Lazy-accept is turned on with 512MB by default in IntelTdxX64.
> > >
> > > Care to explain?  What is the point in not using lazy accept in
> OvmfPkgX64?
> 
> > It's an Open to the community that if lazy accept should be enabled in
> > OvmfPkgX64. I would like to hear the suggestions/comments from you
> > guys. So as the first step, it is disabled.
> 
> I think OvmfPkgX64 and IntelTdxX64 configuration should be identical, and
> I'm wondering whenever this should be a compile time option in the first
> place.  Can we simply use lazy accept unconditionally?
Not all the guest OS support lazy accept. That's why there is a PCD which is used to control this feature.
IntelTdxX64 is a newly added pkg and it hasn't been widely used. So it is low risk to enable the lazy-accept by default.
But OvmfPkgX64 is more general. So at the first stage I think it may be better to disable lazy-accept by default. If someone understand what lazy-accept is and what guest os should be used, then he can enable lazy-accept in OvmfPkgX64 by adding the compile time option.
Dionna Glaze has proposed a new fw_cfg to control the lazy-accept.
> 
> With the edk2 memory management code being able to handle unaccepted
> memory I just don't see the point in supporting different options in early
> firmware setup.  I think sec/pei should accept the memory needed to run
> dxe and not more.  Anything beyond that can be accepted later on demand
> via TdxDxe.
In current implementation by Tdx guest, we accept 512MB in SEC phase and install EfiMemoryAcceptProtocol TdxDxe driver. 512MB is more memories needed by DXE. 
That is because: Tdx guest now accepts memory in 2MB page size. Accepting 512MB is not much slower than accepting 96MB. And 512MB memory can satisfy most of the usage scenarios without accepting more memories later on demand.
Of course other architecture CPUs can have their own implementation. 
> 
> > > Also what exactly does this option mean?  Is this the minimum amount
> > > of memory accepted by the firmware?
> 
> > Yes, this option defines the minimum amount of memory accepted by the
> > firmware. For example, if it is 512MB, then there will be such amount
> > memory accepted by the firmware before jump to OS.
> 
> Where does the 512MB figure come from?
512MB is an experiment value. It includes the minimum DXE memories (~96MB) and the memories to load a normal kernel-image/initrd. In our internal test, 512MB can satisfy most of the usage scenarios without accepting more memories later. And it has no serious performance impact.
> 
> Using a fixed amount of memory doesn't look very robust to me.  Is it
> possible to accept memory when the guest calls ExitBootServices?  That way
> we could guarantee a minimum amount of *free* accepted memory being
> available, to make sure the linux kernel has enough memory to decompress
> itself, allocate memory management data structures and get its own lazy
> accept support going.
With the help of fw_cfg, we can know the size of QemuFwCfgItemKernelSize/ QemuFwCfgItemInitrdSize/QemuFwCfgItemKernelSetupSize. So I am thinking we can calculate the minimum size of accepted memory (PEI/DXE/KernelSize/InitrdSize/KernelSetupSize) before it jumps to OS.  If we can calculate the minimum memory size before jumping to OS, does it mean the on-demand accept is not needed? Actually I am not conformable to change code of AllocatePages/AllocatePools.
What's your thought?
> 
> For guests which don't support lazy accept we could offer a (runtime) option
> to simply accept all memory in ExitBootServices.
What is the runtime option? Some new fw_cfg?

Thanks
Min

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Hi, Glaze
We’re discussing it in another thread. I have added you into that mail thread.
Your comments is always welcomed.

Min

From: dionnaglaze via groups.io <dionnaglaze=google.com@groups.io>
Sent: Wednesday, June 15, 2022 12:10 AM
To: Xu, Min M <min.m.xu@intel.com>; devel@edk2.groups.io
Subject: Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

Since there will be guest OSes that don't support unaccepted memory type (at least initially), and what will be run won't be known at firmware build time, can PcdLazyAcceptPartialMemorySize be initialized by a QemuFwCfg option instead of a FixedPcd? I started a different thread on this, but perhaps it was lost.

[-- Attachment #2: Type: text/html, Size: 2886 bytes --]

Re: [PATCH 04/14] MdePkg: Add UEFI Unaccepted memory definition

[Min Xu]

On June 13, 2022 9:40 AM, Gao Liming wrote:
> 
> Min:
>   I have two comments for this change.
> 1. UEFI spec 2.9 defines EfiUnacceptedMemoryType, not
> EfiUnacceptedMemory.
Ah, you're right. Thanks for reminder. It will be updated in the next version.
> Please confirm.
> 2. I don't find EfiGcdMemoryTypeUnaccepted in PI spec 1.7. Where is it
> defined?
EfiGcdMemoryTypeUnaccepted has not been published in PI spec. So should it be defined in MdeModulePkg, just like EFI_RESOURCE_MEMORY_UNACCEPTED (which is defined in MdeModulePkg/Include/Pi/PrePiHob.h)?

Thanks
Min

Re: [edk2-devel] [PATCH 05/14] MdeModulePkg: Update Dxe to handle unaccepted memory type

[Min Xu]

On June 13, 2022 9:54 AM, Gao Liming wrote:
> Min:
>   I add my comments below.
> 
> >
> > @@ -83,6 +84,7 @@ EFI_MEMORY_TYPE_INFORMATION
> > gMemoryTypeInformation[EfiMaxMemoryType + 1] = {
> >    { EfiMemoryMappedIOPortSpace, 0 },
> >    { EfiPalCode,                 0 },
> >    { EfiPersistentMemory,        0 },
> > +  { EfiGcdMemoryTypeUnaccepted, 0 },
> >    { EfiMaxMemoryType,           0 }
> >  };
> 
> Here should be EfiUnacceptedMemoryType instead of
> EfiGcdMemoryTypeUnaccepted.
Sure. It will be updated in the next version.
> 
> And, based on UEFI spec 2.9, AllocatePages should return
> EFI_INVALID_PARAMETER when EfiUnacceptedMemoryType is specified.
> Please add this check.
Thanks for reminder. It will be fixed in the next version.

Thanks
Min

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Gerd Hoffmann]

  Hi,

> > Where does the 512MB figure come from?

> 512MB is an experiment value. It includes the minimum DXE memories
> (~96MB) and the memories to load a normal kernel-image/initrd. In our
> internal test, 512MB can satisfy most of the usage scenarios without
> accepting more memories later. And it has no serious performance
> impact.

Yes, it'll most likely work for > 90% of the use cases.

The remaining cases will come some day though, so having a plan for that
would be good IMHO.

> > Using a fixed amount of memory doesn't look very robust to me.  Is it
> > possible to accept memory when the guest calls ExitBootServices?  That way
> > we could guarantee a minimum amount of *free* accepted memory being
> > available, to make sure the linux kernel has enough memory to decompress
> > itself, allocate memory management data structures and get its own lazy
> > accept support going.

> With the help of fw_cfg, we can know the size of
> QemuFwCfgItemKernelSize/
> QemuFwCfgItemInitrdSize/QemuFwCfgItemKernelSetupSize.

That'll only work with direct kernel boot, when some boot loader
is in your boot workflow this will not work.

> > For guests which don't support lazy accept we could offer a
> > (runtime) option to simply accept all memory in ExitBootServices.
> What is the runtime option? Some new fw_cfg?

Ideally without manual configuration (see also the reply in Dionna's thread).

I think ovmf should start in lazy accept mode unconditionally.  Accept
enough memory to reach dxe, but not more.  Accept additional memory
needed on demand.

Whenever we pass unaccepted memory to the guest os or not can be decided
rather late, I think we can wait with that until the first
BS->GetMemoryMap() call comes in.  Doing it late gives is more options
to handle things automatically and it also simplifies SEC code.

take care,
  Gerd

Re: [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Dionna Glaze]

>
>
> > With the help of fw_cfg, we can know the size of
> > QemuFwCfgItemKernelSize/
> > QemuFwCfgItemInitrdSize/QemuFwCfgItemKernelSetupSize.
>
> That'll only work with direct kernel boot, when some boot loader
> is in your boot workflow this will not work.

Indeed, the TDX startup library does things very differently from how
we'd handle this for SEV-SNP. There are already 2 ranges of memory
prevalidated in SEC that have to be specially skipped over in PEI.
I tried the lazy accept changes from edk2-staging with SEV-SNP changes
to prevalidate only the PEI installed memory
https://github.com/AMDESE/ovmf/pull/4 and I just could not manage to
get the guest to boot. It'd crash in the QemuTryBootKernel part of
BDS.

Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory
under 4GB to make all that complexity go away. Only this approach
worked in my own testing. With the MMIO hole it's just validating 3GB
of memory.

>
> > > For guests which don't support lazy accept we could offer a
> > > (runtime) option to simply accept all memory in ExitBootServices.
> > What is the runtime option? Some new fw_cfg?
>
> Ideally without manual configuration (see also the reply in Dionna's thread).
>
> I think ovmf should start in lazy accept mode unconditionally.  Accept
> enough memory to reach dxe, but not more.  Accept additional memory
> needed on demand.

This will probably be complicated for SEV-SNP. I couldn't get it to
work as discussed above, but I wouldn't count myself a UEFI expert.

>
> Whenever we pass unaccepted memory to the guest os or not can be decided
> rather late, I think we can wait with that until the first
> BS->GetMemoryMap() call comes in.  Doing it late gives is more options
> to handle things automatically and it also simplifies SEC code.
>
> take care,
>   Gerd
>


-- 
-Dionna Glaze, PhD (she/her)

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

On June 16, 2022 4:52 AM, Dionna Glaze wrote:
> >
> >
> > > With the help of fw_cfg, we can know the size of
> > > QemuFwCfgItemKernelSize/
> > > QemuFwCfgItemInitrdSize/QemuFwCfgItemKernelSetupSize.
> >
> > That'll only work with direct kernel boot, when some boot loader is in
> > your boot workflow this will not work.
> 
> Indeed, the TDX startup library does things very differently from how we'd
> handle this for SEV-SNP. There are already 2 ranges of memory prevalidated
> in SEC that have to be specially skipped over in PEI.
> I tried the lazy accept changes from edk2-staging with SEV-SNP changes to
> prevalidate only the PEI installed memory
> https://github.com/AMDESE/ovmf/pull/4 and I just could not manage to get
> the guest to boot. It'd crash in the QemuTryBootKernel part of BDS.
> 
> Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory under
> 4GB to make all that complexity go away. Only this approach worked in my
> own testing. With the MMIO hole it's just validating 3GB of memory.
Accepting all memory under 4GB will make the things much simpler. In this way I think the accept-on-demand maybe not needed. 
A question: is there some performance impact when accepting all memory under 4GB?

Thanks
Min

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Gerd Hoffmann]

  Hi,

> > Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory under
> > 4GB to make all that complexity go away. Only this approach worked in my
> > own testing. With the MMIO hole it's just validating 3GB of memory.
> Accepting all memory under 4GB will make the things much simpler. In this way I think the accept-on-demand maybe not needed. 
> A question: is there some performance impact when accepting all memory under 4GB?

That would certainly be easiest when it is acceptable from a performance
point of view.  Will also simplify the code because you don't have to
split the low memory block into accepted/unaccepted parts.

It'll be 3G (-machine pc) or 2G (-machine q35) of memory.
Is it possible to accept gigabyte pages btw?

And, yes, this might be enough that accept-on-demand is not needed any
more.

take care,
  Gerd

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Min Xu]

On June 16, 2022 1:51 PM, Gerd Hoffmann wrote:
> > > Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory
> > > under 4GB to make all that complexity go away. Only this approach
> > > worked in my own testing. With the MMIO hole it's just validating 3GB of
> memory.
> > Accepting all memory under 4GB will make the things much simpler. In this
> way I think the accept-on-demand maybe not needed.
> > A question: is there some performance impact when accepting all memory
> under 4GB?
> 
> That would certainly be easiest when it is acceptable from a performance
> point of view.  Will also simplify the code because you don't have to split the
> low memory block into accepted/unaccepted parts.
> 
> It'll be 3G (-machine pc) or 2G (-machine q35) of memory.
> Is it possible to accept gigabyte pages btw?
Currently the supported page size are 4K and 2M. Gigabyte page size has not been supported.
> 
> And, yes, this might be enough that accept-on-demand is not needed any
> more.

Thanks
Min

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Lendacky, Thomas]

On 6/16/22 00:51, Gerd Hoffmann wrote:
>    Hi,
> 
>>> Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory under
>>> 4GB to make all that complexity go away. Only this approach worked in my
>>> own testing. With the MMIO hole it's just validating 3GB of memory.
>> Accepting all memory under 4GB will make the things much simpler. In this way I think the accept-on-demand maybe not needed.
>> A question: is there some performance impact when accepting all memory under 4GB?
> 
> That would certainly be easiest when it is acceptable from a performance
> point of view.  Will also simplify the code because you don't have to
> split the low memory block into accepted/unaccepted parts.
> 
> It'll be 3G (-machine pc) or 2G (-machine q35) of memory.
> Is it possible to accept gigabyte pages btw?

+Ashish/Mike - I thought I had added them earlier...

SNP supports 4K and 2MB. However, SNP can support multiple pages in a 
single Page State Change GHCB request. This might be better than having to 
exit once for every on-demand page.

Thanks,
Tom

> 
> And, yes, this might be enough that accept-on-demand is not needed any
> more.
> 
> take care,
>    Gerd
> 

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Dionna Glaze]

> A question: is there some performance impact when accepting all memory under 4GB?

On SEV-SNP, we accept the HOBs 0-0xA000 and 0x10_0000-0xC00_0000,
which takes a couple seconds, which is non-negligible.
For VMs that want to boot very fast and still have access to a lot of
memory in the long run (e.g., a UEFI app as an enclave or sandbox), I
admit it's not my favorite solution.
That being said, supporting unaccepted memory in the guest means that
the 4GB solution doesn't preclude an accept-on-demand solution in the
future in the case that there is demand.

-- 
-Dionna Glaze, PhD (she/her)

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Lendacky, Thomas]

On 6/16/22 11:44, Dionna Amalie Glaze wrote:
>> A question: is there some performance impact when accepting all memory under 4GB?
> 
> On SEV-SNP, we accept the HOBs 0-0xA000 and 0x10_0000-0xC00_0000,
> which takes a couple seconds, which is non-negligible.

Are you possibly including the memory pinning time (that happens in the 
hypervisor before the guest is started) in your measurement?

For example, I added from RDTSCP instructions in before and after the 
calls to MemEncryptSevSnpPreValidateSystemRam() and got:

*** DEBUG: SEC PreValidating RAM from 820000 to 1710000
*** DEBUG: TSC1=3057966080, TSC2=3070503712, delta = 12537632
*** DEBUG: PEI PreValidating RAM from 0 to A0000
*** DEBUG: TSC1=3430088800, TSC2=3432589504, delta = 2500704
*** DEBUG: PEI PreValidating RAM from 100000 to 80000000
*** DEBUG: TSC1=3436557536, TSC2=3496324336, delta = 59766800


This is for 2GB (0x80000000) of RAM. If I've done my calculations 
correctly for a 1600MHz TSC, that comes out to about 45 milliseconds.

Thanks,
Tom

> For VMs that want to boot very fast and still have access to a lot of
> memory in the long run (e.g., a UEFI app as an enclave or sandbox), I
> admit it's not my favorite solution.
> That being said, supporting unaccepted memory in the guest means that
> the 4GB solution doesn't preclude an accept-on-demand solution in the
> future in the case that there is demand.
> 

Re: [edk2-devel] [PATCH 07/14] OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.

[Dionna Glaze]

> Are you possibly including the memory pinning time (that happens in the
> hypervisor before the guest is started) in your measurement?
>

I might be, sorry. I'd go with your numbers.

> This is for 2GB (0x80000000) of RAM. If I've done my calculations
> correctly for a 1600MHz TSC, that comes out to about 45 milliseconds.
>


-- 
-Dionna Glaze, PhD (she/her)

Re: [PATCH 00/14] Introduce Lazy-accept for Tdx guest

[Min Xu]

Hi, All
Since the lazy-accept patch-set was sent for review, there are a lot of discussions in multiple OpenSource-Communities (EDK2/QEMU/LinuxKernel). 

Below are the summaries of the discussions/proposals
1. Accept memory under 4G
  - Tom Lendacky's suggestion for SEV-SNP is to pre-accept all memory under 4GB to make all complexity go way. 
     https://edk2.groups.io/g/devel/message/90539 
  - Gerd comments that before the automatic negotiation is available, we use some config option and then accept all memory below 4G or accept all memory.
     https://www.mail-archive.com/qemu-devel@nongnu.org/msg894393.html

2. GetMemoryMapEx / OsIndications
  - https://bugzilla.tianocore.org/show_bug.cgi?id=3987
  - GetMemoryMapEx: cannot resolve the boot-loader because it doesn't know about the kernel's capability.
  - OsIndications & self-report kernel header bit could be a full solution.
 
3.  fw_cfg
  - Add new fw_cfg item (opt/ovmf/AcceptAllMemory) to indicate how to handle the unaccepted memory.
      > True - accept all the memory
      > False - don't accept the memory
      > Default - It allows the firmware to choose depending on various factors.
  - Glaze has submit the patch https://lore.kernel.org/all/20220620223300.1555849-1-dionnaglaze@google.com/
 
4.Put some information in kernel image.  (Peter Gonda)
  - Put information into UTS_VERSION which can be read from the buld bzImage. https://www.spinics.net/lists/linux-mm/msg302506.html
  - Glaze denied it. https://www.spinics.net/lists/linux-mm/msg304973.html
 
5. A protocol kernel call from EFI stub before EBS().(Ard)
  - It doesn't work if boot loader is in the boot flow. 
  - https://www.spinics.net/lists/linux-mm/msg304896.html

Among the above 5 proposals, Proposal 1) is different from the other 4 proposals.
Proposal 1) uses a config option and hence there are 2 kinds of OVMF: OVMF-with-LazyAccept and OVMF-with-FullMemory. End user should be aware of it and choose the correct OVMF for the linux kernel (LazyAccept enlightened or not).
Proposal 3) only works for QEMU because of fw_cfg.

So essentially there are 2 questions about the lazy-accept.
1. If Lazy-accept capability should be automatically negotiated?
2. How to implement the Lazy-Accept in OVMF? 

I wonder if lazy-accept feature can be split into 2 stages. 
1. In first stage there is a config option to indicate if lazy-accept is enabled or not.
2. In the second stage the automatic negotiation is introduced so that lazy-accept is enabled or not by the negotiation result.

Thanks
Min

> -----Original Message-----
> From: Min Xu <min.m.xu@intel.com>
> Sent: Monday, June 6, 2022 11:00 AM
> To: devel@edk2.groups.io
> Cc: Xu, Min M <min.m.xu@intel.com>; Gao, Zhichao
> <zhichao.gao@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>;
> Liu, Zhiguang <zhiguang.liu@intel.com>; Wang, Jian J
> <jian.j.wang@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Ni, Ray
> <ray.ni@intel.com>; Aktas, Erdem <erdemaktas@google.com>; Gerd
> Hoffmann <kraxel@redhat.com>; James Bottomley <jejb@linux.ibm.com>;
> Yao, Jiewen <jiewen.yao@intel.com>; Tom Lendacky
> <thomas.lendacky@amd.com>; Gao, Jiaqi <jiaqi.gao@intel.com>
> Subject: [PATCH 00/14] Introduce Lazy-accept for Tdx guest
> 
> RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3937
> 
> UnacceptedMemory is one of the four defined types of TD memory in Intel
> TDX guest. TDVF must invoke TDCALL [TDG.MEM.PAGE.ACCEPT] the
> unaccepted memory before use it. See [TDVF] Section 7.1.
> TDVF: https://www.intel.com/content/dam/develop/external/us/en/
> documents/tdx-virtual-firmware-design-guide-rev-1.01.pdf
> 
> It is a time-consuming task which impacts the boot performance badly.
> One of the mitigation is the lazy-accept mechanism. That the whole system
> memory is divided into 2 parts, one is accepted in bios phase, the other is
> tagged as EfiGcdMemoryTypeUnaccepted and OS will handle these
> "unaccepted" memories.
> See "UEFI Spec v2.9 Table 7-5 Memory Type Usage before
> ExitBootServices()"
> 
> Patch 1-4:
>   Introduce lazy-accept related definitions.
> 
> Patch 5-6:
>   Update Dxe and shell for unaccepted memory.
> 
> Patch 7 - 11:
>   Update OvmfPkg for unaccepted memory.
> 
> Patch 12 - 13:
>   Introduce EfiMemoryAcceptProtocol and realize it in TdxDxe.
> 
> Patch 14:
>   Update Pool and Page functions to accept memory when OOM occurs.
> 
> Code: https://github.com/mxu9/edk2/tree/lazyaccept.v1
> 
> Cc: Zhichao Gao <zhichao.gao@intel.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Zhiguang Liu <zhiguang.liu@intel.com>
> Cc: Jian J Wang <jian.j.wang@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Jiaqi Gao <jiaqi.gao@intel.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> 
> Jiaqi Gao (2):
>   MdePkg: The prototype definition of EfiMemoryAcceptProtocol
>   MdeModulePkg: Pool and page functions accept memory when OOM
> occurs
> 
> Min M Xu (12):
>   MdeModulePkg: Add PrePiHob.h
>   MdePkg: Increase EFI_RESOURCE_MAX_MEMORY_TYPE
>   OvmfPkg: Use EFI_RESOURCE_MEMORY_UNACCEPTED which defined in
>     MdeModulePkg
>   MdePkg: Add UEFI Unaccepted memory definition
>   MdeModulePkg: Update Dxe to handle unaccepted memory type
>   ShellPkg: Update shell command memmap to show unaccepted memory
>   OvmfPkg: Add PCD and DEFINEs for Lazy Accept page.
>   OvmfPkg: Use PcdOvmfWorkAreaBase instead of PcdSevEsWorkAreaBase
>   OvmfPkg: Add MaxAcceptedMemoryAddress in TDX work area
>   OvmfPkg: Introduce lazy accept in PlatformInitLib and PlatformPei
>   OvmfPkg: Update ConstructFwHobList for lazy accept
>   OvmfPkg: Realize EfiMemoryAcceptProtocol in TdxDxe
> 
>  MdeModulePkg/Core/Dxe/DxeMain.inf             |   1 +
>  MdeModulePkg/Core/Dxe/Gcd/Gcd.c               |   5 +
>  MdeModulePkg/Core/Dxe/Mem/Imem.h              |  16 ++
>  MdeModulePkg/Core/Dxe/Mem/Page.c              | 218 ++++++++++++++++++
>  MdeModulePkg/Core/Dxe/Mem/Pool.c              |  14 ++
>  MdeModulePkg/Include/Pi/PrePiHob.h            |  20 ++
>  MdePkg/Include/Pi/PiDxeCis.h                  |   5 +
>  MdePkg/Include/Pi/PiHob.h                     |  11 +-
>  MdePkg/Include/Protocol/MemoryAccept.h        |  37 +++
>  MdePkg/Include/Uefi/UefiMultiPhase.h          |   5 +
>  MdePkg/MdePkg.dec                             |   3 +
>  OvmfPkg/Include/Library/PlatformInitLib.h     |   6 +
>  OvmfPkg/Include/WorkArea.h                    |   1 +
>  OvmfPkg/IntelTdx/IntelTdxX64.dsc              |   8 +
>  .../PrePiHobListPointer.c                     |   4 +-
>  .../PrePiHobListPointerLibTdx.inf             |   2 +-
>  OvmfPkg/Library/PeilessStartupLib/Hob.c       |  26 ++-
>  .../PeilessStartupLib/PeilessStartupLib.inf   |   1 +
>  OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 145 +++++++++++-
>  OvmfPkg/Library/PlatformInitLib/MemDetect.c   |  27 +++
>  .../PlatformInitLib/PlatformInitLib.inf       |   1 +
>  OvmfPkg/OvmfPkg.dec                           |   4 +
>  OvmfPkg/OvmfPkgX64.dsc                        |   9 +
>  OvmfPkg/PlatformPei/MemDetect.c               |   5 +
>  OvmfPkg/TdxDxe/TdxDxe.c                       | 103 +++++++++
>  OvmfPkg/TdxDxe/TdxDxe.inf                     |   2 +
>  .../UefiShellDebug1CommandsLib/MemMap.c       |   6 +
>  27 files changed, 666 insertions(+), 19 deletions(-)  create mode 100644
> MdeModulePkg/Include/Pi/PrePiHob.h
>  create mode 100644 MdePkg/Include/Protocol/MemoryAccept.h
> 
> --
> 2.29.2.windows.2

Re: [PATCH 00/14] Introduce Lazy-accept for Tdx guest

[Gerd Hoffmann]

  Hi,

> 3.  fw_cfg
>   - Add new fw_cfg item (opt/ovmf/AcceptAllMemory) to indicate how to handle the unaccepted memory.
>       > True - accept all the memory
>       > False - don't accept the memory
>       > Default - It allows the firmware to choose depending on various factors.
>   - Glaze has submit the patch https://lore.kernel.org/all/20220620223300.1555849-1-dionnaglaze@google.com/

> Proposal 3) only works for QEMU because of fw_cfg.

Well, while that is true for the patch at hand it doesn't have to be
that way.  We can also simply store the config option in a EFI variable.
Wire up a HII configuration so it can be changed via firmware setup.
Allow setting the EFI variable from fw_cfg, so qemu users can set that
on the qemu command line too (and possibly have similar mechanisms for
other hypervisors, hello cloudhv).

> I wonder if lazy-accept feature can be split into 2 stages. 
> 1. In first stage there is a config option to indicate if lazy-accept is enabled or not.
> 2. In the second stage the automatic negotiation is introduced so that lazy-accept is enabled or not by the negotiation result.

Absolutely.  That is one of the reasons why I suggested to have a
true/false/default config option instead of just true/false.

When the first stage is implemented "default" behavior would be fixed
(either hard-coded or a compile-time option).

When the second stage is implemented "default" behavior would be
dynamic, depending on the negotiation result.

take care,
  Gerd

Re: [PATCH 00/14] Introduce Lazy-accept for Tdx guest

[Dionna Glaze]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

I have a working stack with proposal 5. I was using a version of grub that
didn't use Linux's EFI handover protocol, and I wasn't signalling the
unaccepted memory behavior at the right place (before EBS).

Many thanks to everyone who contributed to this discussion and brought some
creative ideas to the table. Thanks especially to Ard, for multiple
consultations.
With proposal 5, we won't need any control plane annotation for disabling
unaccepted memory, and users will always get safe behavior.

I'll have patches out to OVMF (based on this patch series) and to Linux
shortly.

On Mon, Aug 15, 2022 at 11:47 PM Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
>
> > 3.  fw_cfg
> >   - Add new fw_cfg item (opt/ovmf/AcceptAllMemory) to indicate how to
> handle the unaccepted memory.
> >       > True - accept all the memory
> >       > False - don't accept the memory
> >       > Default - It allows the firmware to choose depending on various
> factors.
> >   - Glaze has submit the patch
> https://lore.kernel.org/all/20220620223300.1555849-1-dionnaglaze@google.com/
>
> > Proposal 3) only works for QEMU because of fw_cfg.
>
> Well, while that is true for the patch at hand it doesn't have to be
> that way.  We can also simply store the config option in a EFI variable.
> Wire up a HII configuration so it can be changed via firmware setup.
> Allow setting the EFI variable from fw_cfg, so qemu users can set that
> on the qemu command line too (and possibly have similar mechanisms for
> other hypervisors, hello cloudhv).
>
> > I wonder if lazy-accept feature can be split into 2 stages.
> > 1. In first stage there is a config option to indicate if lazy-accept is
> enabled or not.
> > 2. In the second stage the automatic negotiation is introduced so that
> lazy-accept is enabled or not by the negotiation result.
>
> Absolutely.  That is one of the reasons why I suggested to have a
> true/false/default config option instead of just true/false.
>
> When the first stage is implemented "default" behavior would be fixed
> (either hard-coded or a compile-time option).
>
> When the second stage is implemented "default" behavior would be
> dynamic, depending on the negotiation result.
>
> take care,
>   Gerd
>
>

-- 
-Dionna Glaze, PhD (she/her)

[-- Attachment #2: Type: text/html, Size: 3027 bytes --]