[PATCH 0/4] OvmfPkg: make better use of physical address space.

[PATCH 0/4] OvmfPkg: make better use of physical address space.

[Gerd Hoffmann]

For historical reasons ovmf is quite conservative on address space
usage, to play safe and avoid using more than 36 physical address bits
(the guaranteed minimum) if possible.  With devices (specifically GPUs)
becoming larger and larger pci memory bars this becomes increasingly
problematic.

This patch series address that by trying to figure what the physical
address space size is (which is a bit tricky, see patch #1 for details).
If that worked scale up the 64-bit mmio window and also pcie bridge
windows dynamically with the available address space.

Gerd Hoffmann (4):
  OvmfPkg/PlatformInitLib: qemu cpuid physbits detection
  OvmfPkg/PlatformInitLib: detect physical address space
  OvmfPkg/PlatformInitLib: dynamic mmio window size
  OvmfPkg/PciHotPlugInitDxe: reserve more mmio space

 OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf |   1 +
 OvmfPkg/Library/PlatformInitLib/MemDetect.c  | 143 ++++++++++++++++---
 OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c   |  13 ++
 3 files changed, 139 insertions(+), 18 deletions(-)

-- 
2.37.3

[PATCH 1/4] OvmfPkg/PlatformInitLib: qemu cpuid physbits detection

[Gerd Hoffmann]

Add some qemu specific quirks to PlatformAddressWidthFromCpuid()
to figure whenever the PhysBits value returned by CPUID is
something real we can work with or not.

See the source code comment for details on the logic.

Also apply some limits to the address space we are going to use:
 * Place a hard cap at 47 PhysBits (128 TB) to avoid using addresses
   which require 5-level paging support.
 * Cap at 40 PhysBits (1 TB) in case the CPU has no support for
   gigabyte pages, to avoid excessive amounts of pages being
   used for page tables.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Library/PlatformInitLib/MemDetect.c | 102 ++++++++++++++++----
 1 file changed, 84 insertions(+), 18 deletions(-)

diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index d1a4f4b20791..143a01ceb01e 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -503,39 +503,105 @@ PlatformGetFirstNonAddress (
 }
 
 /*
- * Use CPUID to figure physical address width.  Does *not* work
- * reliable on qemu.  For historical reasons qemu returns phys-bits=40
- * even in case the host machine supports less than that.
- *
- * qemu has a cpu property (host-phys-bits={on,off}) to change that
- * and make sure guest phys-bits are not larger than host phys-bits.,
- * but it is off by default.  Exception: microvm machine type
- * hard-wires that property to on.
+ * Use CPUID to figure physical address width.
+ *
+ * Does *not* work reliable on qemu.  For historical reasons qemu
+ * returns phys-bits=40 by default even in case the host machine
+ * supports less than that.
+ *
+ * So we apply the following rules (which can be enabled/disabled
+ * using the QemuQuirk parameter) to figure whenever we can work with
+ * the returned physical address width or not:
+ *
+ *   (1) If it is 41 or higher consider it valid.
+ *   (2) If it is 40 or lower consider it valid in case it matches a
+ *       known-good value for the CPU vendor, which is:
+ *         ->  36 or 39 for Intel
+ *         ->  40 for AMD
+ *   (3) Otherwise consider it invalid.
+ *
+ * Recommendation: Run qemu with host-phys-bits=on.  That will make
+ * sure guest phys-bits is not larger than host phys-bits.  Some
+ * distro builds do that by default.
  */
 VOID
 EFIAPI
 PlatformAddressWidthFromCpuid (
-  IN OUT EFI_HOB_PLATFORM_INFO  *PlatformInfoHob
+  IN OUT EFI_HOB_PLATFORM_INFO  *PlatformInfoHob,
+  IN     BOOLEAN                QemuQuirk
   )
 {
-  UINT32  RegEax;
+  UINT32   RegEax, RegEbx, RegEcx, RegEdx, Max;
+  UINT8    PhysBits;
+  CHAR8    Signature[13] = { 0 };
+  BOOLEAN  Valid         = FALSE;
+  BOOLEAN  Page1GSupport = FALSE;
 
-  AsmCpuid (0x80000000, &RegEax, NULL, NULL, NULL);
-  if (RegEax >= 0x80000008) {
+  AsmCpuid (0x80000000, &RegEax, &RegEbx, &RegEcx, &RegEdx);
+  *(UINT32 *)(Signature + 0) = RegEbx;
+  *(UINT32 *)(Signature + 4) = RegEdx;
+  *(UINT32 *)(Signature + 8) = RegEcx;
+  Max                        = RegEax;
+
+  if (Max >= 0x80000001) {
+    AsmCpuid (0x80000001, NULL, NULL, NULL, &RegEdx);
+    if ((RegEdx & BIT26) != 0) {
+      Page1GSupport = TRUE;
+    }
+  }
+
+  if (Max >= 0x80000008) {
     AsmCpuid (0x80000008, &RegEax, NULL, NULL, NULL);
-    PlatformInfoHob->PhysMemAddressWidth = (UINT8)RegEax;
+    PhysBits = (UINT8)RegEax;
   } else {
-    PlatformInfoHob->PhysMemAddressWidth = 36;
+    PhysBits = 36;
   }
 
-  PlatformInfoHob->FirstNonAddress = LShiftU64 (1, PlatformInfoHob->PhysMemAddressWidth);
+  if (!QemuQuirk) {
+    Valid = TRUE;
+  } else if (PhysBits >= 41) {
+    Valid = TRUE;
+  } else if (AsciiStrCmp (Signature, "GenuineIntel") == 0) {
+    if ((PhysBits == 36) || (PhysBits == 39)) {
+      Valid = TRUE;
+    }
+  } else if (AsciiStrCmp (Signature, "AuthenticAMD") == 0) {
+    if (PhysBits == 40) {
+      Valid = TRUE;
+    }
+  }
 
   DEBUG ((
     DEBUG_INFO,
-    "%a: cpuid: phys-bits is %d\n",
+    "%a: Signature: '%a', PhysBits: %d, QemuQuirk: %a, Valid: %a\n",
     __FUNCTION__,
-    PlatformInfoHob->PhysMemAddressWidth
+    Signature,
+    PhysBits,
+    QemuQuirk ? "On" : "Off",
+    Valid ? "Yes" : "No"
     ));
+
+  if (Valid) {
+    if (PhysBits > 47) {
+      /*
+       * Avoid 5-level paging altogether for now, which limits
+       * PhysBits to 48.  Also avoid using address bit 48, due to sign
+       * extension we can't identity-map these addresses (and lots of
+       * places in edk2 assume we have everything identity-mapped).
+       * So the actual limit is 47.
+       */
+      DEBUG ((DEBUG_INFO, "%a: limit PhysBits to 47 (avoid 5-level paging)\n", __func__));
+      PhysBits = 47;
+    }
+
+    if (!Page1GSupport && (PhysBits > 40)) {
+      DEBUG ((DEBUG_INFO, "%a: limit PhysBits to 40 (no 1G pages available)\n", __func__));
+      PhysBits = 40;
+    }
+
+    PlatformInfoHob->PhysMemAddressWidth = PhysBits;
+    PlatformInfoHob->FirstNonAddress     = LShiftU64 (1, PlatformInfoHob->PhysMemAddressWidth);
+  }
 }
 
 /**
@@ -672,7 +738,7 @@ PlatformAddressWidthInitialization (
   EFI_STATUS  Status;
 
   if (PlatformInfoHob->HostBridgeDevId == 0xffff /* microvm */) {
-    PlatformAddressWidthFromCpuid (PlatformInfoHob);
+    PlatformAddressWidthFromCpuid (PlatformInfoHob, FALSE);
     return;
   }
 
-- 
2.37.3

[PATCH 2/4] OvmfPkg/PlatformInitLib: detect physical address space

[Gerd Hoffmann]

Try detect physical address space, when successful use it.
Otherwise go continue using the current guesswork code path.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Library/PlatformInitLib/MemDetect.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index 143a01ceb01e..16ecbfadc30c 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -761,6 +761,19 @@ PlatformAddressWidthInitialization (
     FirstNonAddress = PlatformGetFirstNonAddress (PlatformInfoHob);
   }
 
+  PlatformAddressWidthFromCpuid (PlatformInfoHob, TRUE);
+  if (PlatformInfoHob->PhysMemAddressWidth != 0) {
+    // physical address width is known
+    PlatformInfoHob->FirstNonAddress = FirstNonAddress;
+    return;
+  }
+
+  //
+  // physical address width is NOT known
+  //   -> do some guess work, mostly based on installed memory
+  //   -> try be conservstibe to stay below the guaranteed minimum of
+  //      36 phys bits (aka 64 GB).
+  //
   PhysMemAddressWidth = (UINT8)HighBitSet64 (FirstNonAddress);
 
   //
-- 
2.37.3

[PATCH 3/4] OvmfPkg/PlatformInitLib: dynamic mmio window size

[Gerd Hoffmann]

In case we have a reliable PhysMemAddressWidth use that to dynamically
size the 64bit address window.  Allocate 1/8 of the physical address
space and place the window at the upper end of the address space.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Library/PlatformInitLib/MemDetect.c | 28 +++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index 16ecbfadc30c..ae217d0242ed 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -604,6 +604,33 @@ PlatformAddressWidthFromCpuid (
   }
 }
 
+VOID
+EFIAPI
+PlatformDynamicMmioWindow (
+  IN OUT EFI_HOB_PLATFORM_INFO  *PlatformInfoHob
+  )
+{
+  UINT64  AddrSpace, MmioSpace;
+
+  AddrSpace = LShiftU64 (1, PlatformInfoHob->PhysMemAddressWidth);
+  MmioSpace = LShiftU64 (1, PlatformInfoHob->PhysMemAddressWidth - 3);
+
+  if ((PlatformInfoHob->PcdPciMmio64Size < MmioSpace) &&
+      (PlatformInfoHob->PcdPciMmio64Base + MmioSpace < AddrSpace))
+  {
+    DEBUG ((DEBUG_INFO, "%a: using dynamic mmio window\n", __func__));
+    DEBUG ((DEBUG_INFO, "%a:   Addr Space 0x%Lx (%Ld GB)\n", __func__, AddrSpace, RShiftU64 (AddrSpace, 30)));
+    DEBUG ((DEBUG_INFO, "%a:   MMIO Space 0x%Lx (%Ld GB)\n", __func__, MmioSpace, RShiftU64 (MmioSpace, 30)));
+    PlatformInfoHob->PcdPciMmio64Size = MmioSpace;
+    PlatformInfoHob->PcdPciMmio64Base = AddrSpace - MmioSpace;
+  } else {
+    DEBUG ((DEBUG_INFO, "%a: using classic mmio window\n", __func__));
+  }
+
+  DEBUG ((DEBUG_INFO, "%a:   Pci64 Base 0x%Lx\n", __func__, PlatformInfoHob->PcdPciMmio64Base));
+  DEBUG ((DEBUG_INFO, "%a:   Pci64 Size 0x%Lx\n", __func__, PlatformInfoHob->PcdPciMmio64Size));
+}
+
 /**
   Iterate over the PCI host bridges resources information optionally provided
   in fw-cfg and find the highest address contained in the PCI MMIO windows. If
@@ -765,6 +792,7 @@ PlatformAddressWidthInitialization (
   if (PlatformInfoHob->PhysMemAddressWidth != 0) {
     // physical address width is known
     PlatformInfoHob->FirstNonAddress = FirstNonAddress;
+    PlatformDynamicMmioWindow (PlatformInfoHob);
     return;
   }
 
-- 
2.37.3

[PATCH 4/4] OvmfPkg/PciHotPlugInitDxe: reserve more mmio space

[Gerd Hoffmann]

In case the 64-bit pci mmio window is larger than the default size
of 32G be generous and hand out larger chunks of address space for
prefetchable mmio bridge windows.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf |  1 +
 OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c   | 13 +++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf
index 78b95faa7a7a..f56b1de6fd92 100644
--- a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf
+++ b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf
@@ -40,6 +40,7 @@ [Protocols]
 
 [Pcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId ## CONSUMES
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size          ## CONSUMES
 
 [Depex]
   TRUE
diff --git a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
index c122855b735d..08ea5b57bc18 100644
--- a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
+++ b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
@@ -744,12 +744,25 @@ GetResourcePadding (
     //
     // Request defaults.
     //
+    UINT64  Pci64Size = PcdGet64 (PcdPciMmio64Size);
+
+    // non-prefetchable
     SetMmioPadding (
       --FirstResource,
       FALSE,
       TRUE,
       (UINTN)HighBitSetRoundUp32 (SIZE_2MB)
       );
+
+    // prefetchable
+    if (Pci64Size > SIZE_32GB) {
+      SetMmioPadding (
+        --FirstResource,
+        TRUE,
+        FALSE,
+        (UINTN)HighBitSetRoundUp64 (RShiftU64 (Pci64Size, 8))
+        );
+    }
   }
 
   //
-- 
2.37.3