[PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[Min Xu]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181

Current lazy-accept accepts the memory under address of 4G. To improve
boot performance further more, we introduce the feature of customizing
the physical end address of lazy-accept in build time.

The end address is indicated by PcdAcceptMemoryEndAddress. It means it
accepts the memory under PcdAcceptMemoryEndAddress. The default value
and the max value is 4G. This is to be consistent with the current rule
(accept memory under 4G).

In IntelTdxX64 PcdAcceptMemoryEndAddress can be customized on-demand in
build-time by adding -D ACCEPT_MEMORY_END_ADDRESS=512 in build command.

Code: https://github.com/mxu9/edk2/tree/CustomizeLazyAcceptSize.v1

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>

Min M Xu (3):
  OvmfPkg: Customize lazy-accept's end address
  OvmfPkg/PeilessStartupLib: Update ConstructFwHobList for lazy-accept
  OvmfPkg/PlatformPei: Adjust LowerMemorySize in PublishPeiMemory

 OvmfPkg/IntelTdx/IntelTdxX64.dsc              |  8 ++++
 OvmfPkg/Library/PeilessStartupLib/Hob.c       | 25 +++++++++----
 .../PeilessStartupLib/PeilessStartupLib.inf   |  1 +
 OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 37 ++++++++++++++-----
 .../PlatformInitLib/PlatformInitLib.inf       |  1 +
 OvmfPkg/OvmfPkg.dec                           |  2 +
 OvmfPkg/PlatformPei/MemDetect.c               | 13 +++++++
 OvmfPkg/PlatformPei/PlatformPei.inf           |  1 +
 8 files changed, 72 insertions(+), 16 deletions(-)

-- 
2.29.2.windows.2

[PATCH V1 1/3] OvmfPkg: Customize lazy-accept's end address

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181

Current lazy-accept accepts the memory under address of 4G. To improve
boot performance further more, we introduce the feature of customizing
the physical end address of lazy-accept.

The end address is indicated by PcdAcceptMemoryEndAddress. It means it
accepts the memory under PcdAcceptMemoryEndAddress. The default value
is 4G.

In IntelTdxX64 PcdAcceptMemoryEndAddress can be customized on-demand in
build-time by adding -D ACCEPT_MEMORY_END_ADDRESS=512 in build command.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/IntelTdx/IntelTdxX64.dsc              |  8 ++++
 OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 37 ++++++++++++++-----
 .../PlatformInitLib/PlatformInitLib.inf       |  1 +
 OvmfPkg/OvmfPkg.dec                           |  2 +
 4 files changed, 39 insertions(+), 9 deletions(-)

diff --git a/OvmfPkg/IntelTdx/IntelTdxX64.dsc b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
index 6ec64df91871..46b0b96ad671 100644
--- a/OvmfPkg/IntelTdx/IntelTdxX64.dsc
+++ b/OvmfPkg/IntelTdx/IntelTdxX64.dsc
@@ -62,6 +62,11 @@
   #
   DEFINE UP_CPU_DXE_GUID  = 6490f1c5-ebcc-4665-8892-0075b9bb49b7
 
+  #
+  # Define the end of physical address of memory to be accepted. The unit is M.
+  #
+  DEFINE ACCEPT_MEMORY_END_ADDRESS = 512
+
 [BuildOptions]
   GCC:RELEASE_*_*_CC_FLAGS             = -DMDEPKG_NDEBUG
   INTEL:RELEASE_*_*_CC_FLAGS           = /D MDEPKG_NDEBUG
@@ -457,6 +462,9 @@
   # TDX need 1G PageTable support
   gEfiMdeModulePkgTokenSpaceGuid.PcdUse1GPageTable|TRUE
 
+  ## End of physical address of memory to be accepted.
+  gUefiOvmfPkgTokenSpaceGuid.PcdAcceptMemoryEndAddress|($(ACCEPT_MEMORY_END_ADDRESS)*0x100000)
+
   gEfiShellPkgTokenSpaceGuid.PcdShellFileOperationSize|0x20000
 
   # IRQs 5, 9, 10, 11 are level-triggered
diff --git a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
index 6cb63139cba0..9514badb8ef6 100644
--- a/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
+++ b/OvmfPkg/Library/PlatformInitLib/IntelTdx.c
@@ -375,7 +375,8 @@ AcceptMemoryForAPsStack (
 }
 
 /**
-  BSP and APs work togeter to accept memory which is under the address of 4G.
+  BSP and APs work togeter to accept memory which is under the address
+  indicated by PcdAcceptMemoryEndAddress.
 
   @param[in] VmmHobList           The Hoblist pass the firmware
   @param[in] CpusNum              Number of vCPUs
@@ -400,13 +401,22 @@ AcceptMemory (
   EFI_PHYSICAL_ADDRESS  PhysicalEnd;
   EFI_PHYSICAL_ADDRESS  AcceptMemoryEndAddress;
 
-  Status                 = EFI_SUCCESS;
-  AcceptMemoryEndAddress = BASE_4GB;
+  Status = EFI_SUCCESS;
 
   ASSERT (VmmHobList != NULL);
   Hob.Raw = (UINT8 *)VmmHobList;
 
-  DEBUG ((DEBUG_INFO, "AcceptMemory under address of 4G\n"));
+  AcceptMemoryEndAddress = (PHYSICAL_ADDRESS)FixedPcdGet64 (PcdAcceptMemoryEndAddress);
+  if (AcceptMemoryEndAddress == 0) {
+    AcceptMemoryEndAddress = MAX_UINT64;
+  }
+
+  if (AcceptMemoryEndAddress <= PhysicalAddressStart) {
+    ASSERT (FALSE);
+    return EFI_SUCCESS;
+  }
+
+  DEBUG ((DEBUG_INFO, "AcceptMemory till 0x%llx\n", AcceptMemoryEndAddress));
 
   //
   // Parse the HOB list until end of list or matching type is found.
@@ -816,11 +826,7 @@ BuildResourceDescriptorHobForUnacceptedMemory (
   ResourceLength    = Hob->ResourceLength;
   PhysicalEnd       = PhysicalStart + ResourceLength;
 
-  //
-  // In the first stage of lazy-accept, all the memory under 4G will be accepted.
-  // The memory above 4G will not be accepted.
-  //
-  MaxAcceptedMemoryAddress = BASE_4GB;
+  MaxAcceptedMemoryAddress = FixedPcdGet64 (PcdAcceptMemoryEndAddress);
 
   if (PhysicalEnd <= MaxAcceptedMemoryAddress) {
     //
@@ -833,6 +839,19 @@ BuildResourceDescriptorHobForUnacceptedMemory (
     // This memory region hasn't been accepted.
     // So keep the ResourceType and ResourceAttribute unchange.
     //
+  } else if ((PhysicalStart < MaxAcceptedMemoryAddress) && (PhysicalEnd > MaxAcceptedMemoryAddress)) {
+    //
+    // Left part of the memory region is accepted. The right part is unaccepted.
+    //
+    BuildResourceDescriptorHob (
+      EFI_RESOURCE_SYSTEM_MEMORY,
+      ResourceAttribute | (EFI_RESOURCE_ATTRIBUTE_PRESENT | EFI_RESOURCE_ATTRIBUTE_INITIALIZED | EFI_RESOURCE_ATTRIBUTE_TESTED),
+      PhysicalStart,
+      MaxAcceptedMemoryAddress - PhysicalStart
+      );
+
+    PhysicalStart  = MaxAcceptedMemoryAddress;
+    ResourceLength = PhysicalEnd - MaxAcceptedMemoryAddress;
   }
 
   BuildResourceDescriptorHob (
diff --git a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
index 140216979a54..2a909ade895b 100644
--- a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
+++ b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
@@ -100,6 +100,7 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdTdxAcceptPageSize
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashNvStorageVariableBase
   gUefiOvmfPkgTokenSpaceGuid.PcdCfvRawDataSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdAcceptMemoryEndAddress
 
 [FeaturePcd]
   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 693925a1dc7a..e6cc524e0f7f 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -408,6 +408,8 @@
 
   ## The Tdx accept page size. 0x1000(4k),0x200000(2M)
   gUefiOvmfPkgTokenSpaceGuid.PcdTdxAcceptPageSize|0x200000|UINT32|0x65
+  ## End of physical address of memory to be accepted
+  gUefiOvmfPkgTokenSpaceGuid.PcdAcceptMemoryEndAddress|0x100000000|UINT64|0x69
 
   ## The QEMU fw_cfg variable that UefiDriverEntryPointFwCfgOverrideLib will
   #  check to decide whether to abort dispatch of the driver it is linked into.
-- 
2.29.2.windows.2

[PATCH V1 2/3] OvmfPkg/PeilessStartupLib: Update ConstructFwHobList for lazy-accept

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181

ConstructFwHobList once searched the accepted memory under 4G. This
need to be updated because of PcdAcceptMemoryEndAddress. If
PcdAcceptMemoryEndAddress is less than 4G, we should search the memory
under PcdAcceptMemoryEndAddress.

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/Library/PeilessStartupLib/Hob.c       | 25 +++++++++++++------
 .../PeilessStartupLib/PeilessStartupLib.inf   |  1 +
 2 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/OvmfPkg/Library/PeilessStartupLib/Hob.c b/OvmfPkg/Library/PeilessStartupLib/Hob.c
index 630ce445ebec..8ea7a68343ed 100644
--- a/OvmfPkg/Library/PeilessStartupLib/Hob.c
+++ b/OvmfPkg/Library/PeilessStartupLib/Hob.c
@@ -74,17 +74,23 @@ ConstructFwHobList (
   )
 {
   EFI_PEI_HOB_POINTERS  Hob;
+  EFI_PHYSICAL_ADDRESS  PhysicalStart;
   EFI_PHYSICAL_ADDRESS  PhysicalEnd;
   UINT64                ResourceLength;
   EFI_PHYSICAL_ADDRESS  LowMemoryStart;
   UINT64                LowMemoryLength;
+  EFI_PHYSICAL_ADDRESS  AcceptMemoryEndAddress;
 
   ASSERT (VmmHobList != NULL);
 
   Hob.Raw = (UINT8 *)VmmHobList;
 
-  LowMemoryLength = 0;
-  LowMemoryStart  = 0;
+  LowMemoryLength        = 0;
+  LowMemoryStart         = 0;
+  AcceptMemoryEndAddress = FixedPcdGet64 (PcdAcceptMemoryEndAddress);
+  if ((AcceptMemoryEndAddress == 0) || (AcceptMemoryEndAddress > SIZE_4GB)) {
+    AcceptMemoryEndAddress = SIZE_4GB;
+  }
 
   //
   // Parse the HOB list until end of list or matching type is found.
@@ -92,16 +98,21 @@ ConstructFwHobList (
   while (!END_OF_HOB_LIST (Hob)) {
     if (Hob.Header->HobType == EFI_HOB_TYPE_RESOURCE_DESCRIPTOR) {
       if (Hob.ResourceDescriptor->ResourceType == BZ3937_EFI_RESOURCE_MEMORY_UNACCEPTED) {
-        PhysicalEnd    = Hob.ResourceDescriptor->PhysicalStart + Hob.ResourceDescriptor->ResourceLength;
+        PhysicalStart  = Hob.ResourceDescriptor->PhysicalStart;
+        PhysicalEnd    = PhysicalStart + Hob.ResourceDescriptor->ResourceLength;
         ResourceLength = Hob.ResourceDescriptor->ResourceLength;
 
-        if (PhysicalEnd <= BASE_4GB) {
+        if (PhysicalEnd >= AcceptMemoryEndAddress) {
+          ResourceLength = AcceptMemoryEndAddress - PhysicalStart;
+        }
+
+        if (PhysicalStart >= AcceptMemoryEndAddress) {
+          break;
+        } else {
           if (ResourceLength > LowMemoryLength) {
             LowMemoryStart  = Hob.ResourceDescriptor->PhysicalStart;
             LowMemoryLength = ResourceLength;
           }
-        } else {
-          break;
         }
       }
     }
@@ -110,7 +121,7 @@ ConstructFwHobList (
   }
 
   if (LowMemoryLength == 0) {
-    DEBUG ((DEBUG_ERROR, "Cannot find a memory region under 4GB for Fw hoblist.\n"));
+    DEBUG ((DEBUG_ERROR, "Cannot find a memory region under 0x%llx for Fw hoblist.\n", AcceptMemoryEndAddress));
     return EFI_NOT_FOUND;
   }
 
diff --git a/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf b/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
index def50b4b019e..644facb60074 100644
--- a/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
+++ b/OvmfPkg/Library/PeilessStartupLib/PeilessStartupLib.inf
@@ -88,3 +88,4 @@
   gEfiMdeModulePkgTokenSpaceGuid.PcdNullPointerDetectionPropertyMask    ## CONSUMES
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvBase
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdAcceptMemoryEndAddress
-- 
2.29.2.windows.2

[PATCH V1 3/3] OvmfPkg/PlatformPei: Adjust LowerMemorySize in PublishPeiMemory

[Min Xu]

From: Min M Xu <min.m.xu@intel.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181

With the introduce of PcdAcceptMemoryEndAddress, TDVF may accept less
memory than LowerMemorySizse. So it should be adjusted in
PublishPeiMemory().

Cc: Erdem Aktas <erdemaktas@google.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/PlatformPei/MemDetect.c     | 13 +++++++++++++
 OvmfPkg/PlatformPei/PlatformPei.inf |  1 +
 2 files changed, 14 insertions(+)

diff --git a/OvmfPkg/PlatformPei/MemDetect.c b/OvmfPkg/PlatformPei/MemDetect.c
index 3d8375320dcb..12a1ad46f1ca 100644
--- a/OvmfPkg/PlatformPei/MemDetect.c
+++ b/OvmfPkg/PlatformPei/MemDetect.c
@@ -270,8 +270,21 @@ PublishPeiMemory (
   UINT32                PeiMemoryCap;
   UINT32                S3AcpiReservedMemoryBase;
   UINT32                S3AcpiReservedMemorySize;
+  UINT64                AcceptMemoryEndAddress;
 
   LowerMemorySize = PlatformGetSystemMemorySizeBelow4gb (PlatformInfoHob);
+
+  //
+  // In Tdx guest there may be less memory accepted than LowerMemorySize.
+  // So LowerMemorySize need to be adjusted accordingly.
+  //
+  if (TdIsEnabled ()) {
+    AcceptMemoryEndAddress = FixedPcdGet64 (PcdAcceptMemoryEndAddress);
+    if ((AcceptMemoryEndAddress < SIZE_4GB) && (LowerMemorySize > (UINT32)(UINTN)AcceptMemoryEndAddress)) {
+      LowerMemorySize = (UINT32)(UINTN)AcceptMemoryEndAddress;
+    }
+  }
+
   if (PlatformInfoHob->SmmSmramRequire) {
     //
     // TSEG is chipped from the end of low RAM
diff --git a/OvmfPkg/PlatformPei/PlatformPei.inf b/OvmfPkg/PlatformPei/PlatformPei.inf
index 1fadadeb5565..c87d2fa25c08 100644
--- a/OvmfPkg/PlatformPei/PlatformPei.inf
+++ b/OvmfPkg/PlatformPei/PlatformPei.inf
@@ -130,6 +130,7 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfWorkAreaSize
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSnpSecretsBase
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSnpSecretsSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdAcceptMemoryEndAddress
 
 [FeaturePcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdCsmEnable
-- 
2.29.2.windows.2

Re: [PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[Gerd Hoffmann]

On Mon, Dec 26, 2022 at 09:33:35AM +0800, Min Xu wrote:
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181
> 
> Current lazy-accept accepts the memory under address of 4G. To improve
> boot performance further more, we introduce the feature of customizing
> the physical end address of lazy-accept in build time.

Do you have numbers?  I'm wondering how much of a difference this
actually is, given that 2M pages is fast and tdx already uses all
processors to accept memory ...

What happens in case the firmware runs out of memory in DXE phase?

We have MemoryAcceptProtocol meanwhile, but I don't think the memory
core uses that to accept more memory on demand (for example when booting
linux with a large initrd).

take care,
  Gerd

Re: [PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[Lendacky, Thomas]

On 12/25/22 19:33, Min Xu wrote:
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181
> 
> Current lazy-accept accepts the memory under address of 4G. To improve
> boot performance further more, we introduce the feature of customizing
> the physical end address of lazy-accept in build time.
> 
> The end address is indicated by PcdAcceptMemoryEndAddress. It means it
> accepts the memory under PcdAcceptMemoryEndAddress. The default value
> and the max value is 4G. This is to be consistent with the current rule
> (accept memory under 4G).
> 
> In IntelTdxX64 PcdAcceptMemoryEndAddress can be customized on-demand in
> build-time by adding -D ACCEPT_MEMORY_END_ADDRESS=512 in build command.

Wasn't there an agreement against modifying the build environment like 
this for memory acceptance? I realize this is not an accept / no-accept 
build environment change, but still...

Thanks,
Tom

> 
> Code: https://github.com/mxu9/edk2/tree/CustomizeLazyAcceptSize.v1
> 
> Cc: Erdem Aktas <erdemaktas@google.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: James Bottomley <jejb@linux.ibm.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Tom Lendacky <thomas.lendacky@amd.com>
> Signed-off-by: Min Xu <min.m.xu@intel.com>
> 
> Min M Xu (3):
>    OvmfPkg: Customize lazy-accept's end address
>    OvmfPkg/PeilessStartupLib: Update ConstructFwHobList for lazy-accept
>    OvmfPkg/PlatformPei: Adjust LowerMemorySize in PublishPeiMemory
> 
>   OvmfPkg/IntelTdx/IntelTdxX64.dsc              |  8 ++++
>   OvmfPkg/Library/PeilessStartupLib/Hob.c       | 25 +++++++++----
>   .../PeilessStartupLib/PeilessStartupLib.inf   |  1 +
>   OvmfPkg/Library/PlatformInitLib/IntelTdx.c    | 37 ++++++++++++++-----
>   .../PlatformInitLib/PlatformInitLib.inf       |  1 +
>   OvmfPkg/OvmfPkg.dec                           |  2 +
>   OvmfPkg/PlatformPei/MemDetect.c               | 13 +++++++
>   OvmfPkg/PlatformPei/PlatformPei.inf           |  1 +
>   8 files changed, 72 insertions(+), 16 deletions(-)
> 

Re: [edk2-devel] [PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[Min Xu]

On January 2, 2023 6:37 PM, Gerd Hoffmann wrote:
> On Mon, Dec 26, 2022 at 09:33:35AM +0800, Min Xu wrote:
> > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181
> >
> > Current lazy-accept accepts the memory under address of 4G. To improve
> > boot performance further more, we introduce the feature of customizing
> > the physical end address of lazy-accept in build time.
> 
> Do you have numbers?  I'm wondering how much of a difference this actually
> is, given that 2M pages is fast and tdx already uses all processors to accept
> memory ...
This feature is tested in Intel SPR platform (boot up a td guest configured with 4vCPU + 4G memory).
It costs about 91ms to accept memories under address of 0x20000000. As a comparison it costs about 240ms to accept memories under address of 0x100000000.

> 
> What happens in case the firmware runs out of memory in DXE phase?
We create an initrd which size is 881MB. The td guest is configured to accept memories under address of 0x20000000.
 1) Direct boot
If we set the boot mode as direct boot, then it will turn to next boot option. In our case it is a grub boot.
If the log message is turned on, then we can see below errors when trying to FetchBlob "initrd":
  AllocatePoolPages: failed to allocate 225423 pages
  AllocatePool: failed to allocate 923331624 bytes
  FetchBlob: failed to allocate 923331584 bytes for "initrd"
  Error: Image at 0001E152000 start failed: Out of Resources

2) Grub boot
If we set the boot mode as grub boot, then below error message is shown:
error: ../../grub-core/loader/i386/efi/linux.c:119:can't allocate initrd.
error: ../../grub-core/loader/i386/efi/linux.c:119:can't allocate initrd.
Press any key to continue...Press any key to continue...

After a while the boot process continued. Finally the td guest is successfully brought up.

Thanks
Min

Re: [edk2-devel] [PATCH V1 0/3] Customize lazy-accepted memory size for TDVF

[Gerd Hoffmann]

On Mon, Jan 16, 2023 at 12:01:40PM +0000, Xu, Min M wrote:
> On January 2, 2023 6:37 PM, Gerd Hoffmann wrote:
> > On Mon, Dec 26, 2022 at 09:33:35AM +0800, Min Xu wrote:
> > > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4181
> > >
> > > Current lazy-accept accepts the memory under address of 4G. To improve
> > > boot performance further more, we introduce the feature of customizing
> > > the physical end address of lazy-accept in build time.
> > 
> > Do you have numbers?  I'm wondering how much of a difference this actually
> > is, given that 2M pages is fast and tdx already uses all processors to accept
> > memory ...
> This feature is tested in Intel SPR platform (boot up a td guest configured with 4vCPU + 4G memory).
> It costs about 91ms to accept memories under address of 0x20000000. As a comparison it costs about 240ms to accept memories under address of 0x100000000.

Under 0x100000000 is 2G in practice with the default q35 memory layout.
Under 0x020000000 is 512M (implemented by this patch series).

Accepting 4x the memory takes less than 4x the time, probably because
some memory is not accepted in multiprocessor mode; or maybe other
constant overhead.

Accepting additional 1.5G needs ~150 ms, so we talk about ~0.1s per GB.

> > What happens in case the firmware runs out of memory in DXE phase?
> We create an initrd which size is 881MB. The td guest is configured to accept memories under address of 0x20000000.
>  1) Direct boot
> [ ... ]
>   Error: Image at 0001E152000 start failed: Out of Resources
> 
> 2) Grub boot
> error: ../../grub-core/loader/i386/efi/linux.c:119:can't allocate initrd.

> After a while the boot process continued. Finally the td guest is successfully brought up.

I wouldn't call not being able to load the initrd a success.

So we can't accept more memory on demand in case 512 MB is not enough.

IIRC the last time this was discussed we agreed on accepting all memory
below 4G for this reason, and also to keep things simple and give the
loaded kernel some wiggle room.

take care,
  Gerd