[PATCH v7 0/6] OvmfPkg/Microvm/pcie: add pcie support

[PATCH v7 0/6] OvmfPkg/Microvm/pcie: add pcie support

[Gerd Hoffmann]

Needs two little tweaks in PCI code because microvm supports mmio only.
Other than that just wire up the existing code (the PCIe host adapter
used by microvm is the same (virtual) hardware used by the arm/aarch64
virtual machines).

v7:
 - allow non-existing io address space only in case
   there are no io reservations (Mateusz Albecki)

v6:
 - codestyle fix (Abner Chang).

v5:
 - codestyle (uncrustify) fix.

v4:
 - update PciHostBridge check (Abner Chang).

v3:
 - rebase to latest master, adapt to PlatformInitLib.
 - rework PhysMemAddressWidth handling for microvm.

v2:
 - rebase to latest master
 - pick up review tags

Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3777

Gerd Hoffmann (6):
  MdeModulePkg/PciHostBridge: io range is not mandatory
  OvmfPkg/FdtPciHostBridgeLib: io range is not mandatory
  OvmfPkg/Platform: unfix PcdPciExpressBaseAddress
  OvmfPkg/Microvm/pcie: no vbeshim please
  OvmfPkg/Microvm/pcie: mPhysMemAddressWidth tweak
  OvmfPkg/Microvm/pcie: add pcie support

 OvmfPkg/Microvm/MicrovmX64.dsc                | 40 ++++++++++-------
 .../PlatformInitLib/PlatformInitLib.inf       |  4 +-
 OvmfPkg/PlatformPei/PlatformPei.inf           |  2 +-
 .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c  |  6 +++
 .../FdtPciHostBridgeLib/FdtPciHostBridgeLib.c | 45 ++++++++++---------
 OvmfPkg/Library/PlatformInitLib/MemDetect.c   | 45 ++++++++++++++++++-
 OvmfPkg/Library/PlatformInitLib/Platform.c    |  4 +-
 OvmfPkg/PlatformPei/Platform.c                |  2 +-
 OvmfPkg/QemuVideoDxe/VbeShim.c                |  2 +
 OvmfPkg/Microvm/README                        |  2 +-
 10 files changed, 107 insertions(+), 45 deletions(-)

-- 
2.36.1

[PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Gerd Hoffmann]

io range is not mandatory according to pcie spec, so allow
pcie host bridge configurations without io window in case
there are no io reservations.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
---
 MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
index b20bcd310ad5..354be6dbb313 100644
--- a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
+++ b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
@@ -1085,6 +1085,12 @@ NotifyPhase (
               RootBridge->ResAllocNode[Index].Base   = BaseAddress;
               RootBridge->ResAllocNode[Index].Status = ResAllocated;
               DEBUG ((DEBUG_INFO, "Success\n"));
+            } else if ((Index == TypeIo) &&
+                       (RootBridge->Io.Base == MAX_UINT64) &&
+                       (RootBridge->ResAllocNode[Index].Length == 0))
+            {
+              /* I/O is optional on PCIe */
+              DEBUG ((DEBUG_INFO, "Success (PCIe NoIO)\n"));
             } else {
               ReturnStatus = EFI_OUT_OF_RESOURCES;
               DEBUG ((DEBUG_ERROR, "Out Of Resource!\n"));
-- 
2.36.1

[PATCH v7 2/6] OvmfPkg/FdtPciHostBridgeLib: io range is not mandatory

[Gerd Hoffmann]

io range is not mandatory according to pcie spec,
so allow host bridges without io address space.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 .../FdtPciHostBridgeLib/FdtPciHostBridgeLib.c | 45 ++++++++++---------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/OvmfPkg/Fdt/FdtPciHostBridgeLib/FdtPciHostBridgeLib.c b/OvmfPkg/Fdt/FdtPciHostBridgeLib/FdtPciHostBridgeLib.c
index 98828e0b262b..14b41a533e96 100644
--- a/OvmfPkg/Fdt/FdtPciHostBridgeLib/FdtPciHostBridgeLib.c
+++ b/OvmfPkg/Fdt/FdtPciHostBridgeLib/FdtPciHostBridgeLib.c
@@ -292,13 +292,8 @@ ProcessPciHost (
     }
   }
 
-  if ((*IoSize == 0) || (*Mmio32Size == 0)) {
-    DEBUG ((
-      DEBUG_ERROR,
-      "%a: %a space empty\n",
-      __FUNCTION__,
-      (*IoSize == 0) ? "IO" : "MMIO32"
-      ));
+  if (*Mmio32Size == 0) {
+    DEBUG ((DEBUG_ERROR, "%a: MMIO32 space empty\n", __FUNCTION__));
     return EFI_PROTOCOL_ERROR;
   }
 
@@ -333,13 +328,15 @@ ProcessPciHost (
     return Status;
   }
 
-  //
-  // Map the MMIO window that provides I/O access - the PCI host bridge code
-  // is not aware of this translation and so it will only map the I/O view
-  // in the GCD I/O map.
-  //
-  Status = MapGcdMmioSpace (*IoBase + IoTranslation, *IoSize);
-  ASSERT_EFI_ERROR (Status);
+  if (*IoSize != 0) {
+    //
+    // Map the MMIO window that provides I/O access - the PCI host bridge code
+    // is not aware of this translation and so it will only map the I/O view
+    // in the GCD I/O map.
+    //
+    Status = MapGcdMmioSpace (*IoBase + IoTranslation, *IoSize);
+    ASSERT_EFI_ERROR (Status);
+  }
 
   return Status;
 }
@@ -413,17 +410,21 @@ PciHostBridgeGetRootBridges (
 
   AllocationAttributes = EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM;
 
-  Io.Base   = IoBase;
-  Io.Limit  = IoBase + IoSize - 1;
+  if (IoSize != 0) {
+    Io.Base  = IoBase;
+    Io.Limit = IoBase + IoSize - 1;
+  } else {
+    Io.Base  = MAX_UINT64;
+    Io.Limit = 0;
+  }
+
   Mem.Base  = Mmio32Base;
   Mem.Limit = Mmio32Base + Mmio32Size - 1;
 
-  if (sizeof (UINTN) == sizeof (UINT64)) {
-    MemAbove4G.Base  = Mmio64Base;
-    MemAbove4G.Limit = Mmio64Base + Mmio64Size - 1;
-    if (Mmio64Size > 0) {
-      AllocationAttributes |= EFI_PCI_HOST_BRIDGE_MEM64_DECODE;
-    }
+  if ((sizeof (UINTN) == sizeof (UINT64)) && (Mmio64Size != 0)) {
+    MemAbove4G.Base       = Mmio64Base;
+    MemAbove4G.Limit      = Mmio64Base + Mmio64Size - 1;
+    AllocationAttributes |= EFI_PCI_HOST_BRIDGE_MEM64_DECODE;
   } else {
     //
     // UEFI mandates a 1:1 virtual-to-physical mapping, so on a 32-bit
-- 
2.36.1

[PATCH v7 3/6] OvmfPkg/Platform: unfix PcdPciExpressBaseAddress

[Gerd Hoffmann]

Will be set by FdtPciHostBridgeLib, so it can't be an fixed when we
want use that library.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf | 4 +++-
 OvmfPkg/PlatformPei/PlatformPei.inf                 | 2 +-
 OvmfPkg/Library/PlatformInitLib/MemDetect.c         | 4 ++--
 OvmfPkg/Library/PlatformInitLib/Platform.c          | 4 ++--
 4 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
index d0fab5cc1f4f..d2a0bec43452 100644
--- a/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
+++ b/OvmfPkg/Library/PlatformInitLib/PlatformInitLib.inf
@@ -54,8 +54,10 @@ [LibraryClasses]
 [LibraryClasses.X64]
   TdxLib
 
-[FixedPcd]
+[Pcd]
   gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
+
+[FixedPcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfWorkAreaBase
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfWorkAreaSize
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfLockBoxStorageBase
diff --git a/OvmfPkg/PlatformPei/PlatformPei.inf b/OvmfPkg/PlatformPei/PlatformPei.inf
index 00372fa0ebb5..3cd83e6ec3e5 100644
--- a/OvmfPkg/PlatformPei/PlatformPei.inf
+++ b/OvmfPkg/PlatformPei/PlatformPei.inf
@@ -95,6 +95,7 @@ [Pcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdXenPvhStartOfDayStructPtr
   gUefiOvmfPkgTokenSpaceGuid.PcdXenPvhStartOfDayStructPtrSize
   gEfiMdePkgTokenSpaceGuid.PcdGuidedExtractHandlerTableAddress
+  gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareSize
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableSize
   gEfiMdeModulePkgTokenSpaceGuid.PcdEmuVariableNvStoreReserved
@@ -118,7 +119,6 @@ [Pcd]
 [FixedPcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfCpuidBase
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfCpuidSize
-  gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress
   gEmbeddedTokenSpaceGuid.PcdMemoryTypeEfiACPIMemoryNVS
   gEmbeddedTokenSpaceGuid.PcdMemoryTypeEfiACPIReclaimMemory
   gEmbeddedTokenSpaceGuid.PcdMemoryTypeEfiReservedMemoryType
diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index 4c1dedf863c3..83a7b6726bb7 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -61,8 +61,8 @@ PlatformQemuUc32BaseInitialization (
     // [PcdPciExpressBaseAddress, 4GB) range require a very small number of
     // variable MTRRs (preferably 1 or 2).
     //
-    ASSERT (FixedPcdGet64 (PcdPciExpressBaseAddress) <= MAX_UINT32);
-    PlatformInfoHob->Uc32Base = (UINT32)FixedPcdGet64 (PcdPciExpressBaseAddress);
+    ASSERT (PcdGet64 (PcdPciExpressBaseAddress) <= MAX_UINT32);
+    PlatformInfoHob->Uc32Base = (UINT32)PcdGet64 (PcdPciExpressBaseAddress);
     return;
   }
 
diff --git a/OvmfPkg/Library/PlatformInitLib/Platform.c b/OvmfPkg/Library/PlatformInitLib/Platform.c
index 101074f6100d..60a30a01f3b5 100644
--- a/OvmfPkg/Library/PlatformInitLib/Platform.c
+++ b/OvmfPkg/Library/PlatformInitLib/Platform.c
@@ -154,7 +154,7 @@ PlatformMemMapInitialization (
     // The MMCONFIG area is expected to fall between the top of low RAM and
     // the base of the 32-bit PCI host aperture.
     //
-    PciExBarBase = FixedPcdGet64 (PcdPciExpressBaseAddress);
+    PciExBarBase = PcdGet64 (PcdPciExpressBaseAddress);
     ASSERT (TopOfLowRam <= PciExBarBase);
     ASSERT (PciExBarBase <= MAX_UINT32 - SIZE_256MB);
     PciBase = (UINT32)(PciExBarBase + SIZE_256MB);
@@ -278,7 +278,7 @@ PciExBarInitialization (
   // determined in AddressWidthInitialization(), i.e., 36 bits, will suffice
   // for DXE's page tables to cover the MMCONFIG area.
   //
-  PciExBarBase.Uint64 = FixedPcdGet64 (PcdPciExpressBaseAddress);
+  PciExBarBase.Uint64 = PcdGet64 (PcdPciExpressBaseAddress);
   ASSERT ((PciExBarBase.Uint32[1] & MCH_PCIEXBAR_HIGHMASK) == 0);
   ASSERT ((PciExBarBase.Uint32[0] & MCH_PCIEXBAR_LOWMASK) == 0);
 
-- 
2.36.1

[PATCH v7 4/6] OvmfPkg/Microvm/pcie: no vbeshim please

[Gerd Hoffmann]

Those old windows versions which need the vbeshim hack
will not run on microvm anyway.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/QemuVideoDxe/VbeShim.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/OvmfPkg/QemuVideoDxe/VbeShim.c b/OvmfPkg/QemuVideoDxe/VbeShim.c
index 8faa146b6cce..2a048211a823 100644
--- a/OvmfPkg/QemuVideoDxe/VbeShim.c
+++ b/OvmfPkg/QemuVideoDxe/VbeShim.c
@@ -156,6 +156,8 @@ InstallVbeShim (
     case INTEL_Q35_MCH_DEVICE_ID:
       Pam1Address = DRAMC_REGISTER_Q35 (MCH_PAM1);
       break;
+    case MICROVM_PSEUDO_DEVICE_ID:
+      return;
     default:
       DEBUG ((
         DEBUG_ERROR,
-- 
2.36.1

[PATCH v7 5/6] OvmfPkg/Microvm/pcie: mPhysMemAddressWidth tweak

[Gerd Hoffmann]

microvm places the 64bit mmio space at the end of the physical address
space.  So mPhysMemAddressWidth must be correct, otherwise the pci host
bridge setup throws an error because it thinks the 64bit mmio window is
not addressable.

On microvm we can simply use standard cpuid to figure the address width
because the host-phys-bits option (-cpu ${name},host-phys-bits=on) is
forced to be enabled.  Side note: For 'pc' and 'q35' this is not the
case for backward compatibility reasons.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Library/PlatformInitLib/MemDetect.c | 41 +++++++++++++++++++++
 OvmfPkg/PlatformPei/Platform.c              |  2 +-
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/OvmfPkg/Library/PlatformInitLib/MemDetect.c b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
index 83a7b6726bb7..c28d7601f87e 100644
--- a/OvmfPkg/Library/PlatformInitLib/MemDetect.c
+++ b/OvmfPkg/Library/PlatformInitLib/MemDetect.c
@@ -491,6 +491,42 @@ PlatformGetFirstNonAddress (
   return FirstNonAddress;
 }
 
+/*
+ * Use CPUID to figure physical address width.  Does *not* work
+ * reliable on qemu.  For historical reasons qemu returns phys-bits=40
+ * even in case the host machine supports less than that.
+ *
+ * qemu has a cpu property (host-phys-bits={on,off}) to change that
+ * and make sure guest phys-bits are not larger than host phys-bits.,
+ * but it is off by default.  Exception: microvm machine type
+ * hard-wires that property to on.
+ */
+VOID
+EFIAPI
+PlatformAddressWidthFromCpuid (
+  IN OUT EFI_HOB_PLATFORM_INFO  *PlatformInfoHob
+  )
+{
+  UINT32  RegEax;
+
+  AsmCpuid (0x80000000, &RegEax, NULL, NULL, NULL);
+  if (RegEax >= 0x80000008) {
+    AsmCpuid (0x80000008, &RegEax, NULL, NULL, NULL);
+    PlatformInfoHob->PhysMemAddressWidth = (UINT8)RegEax;
+  } else {
+    PlatformInfoHob->PhysMemAddressWidth = 36;
+  }
+
+  PlatformInfoHob->FirstNonAddress = LShiftU64 (1, PlatformInfoHob->PhysMemAddressWidth);
+
+  DEBUG ((
+    DEBUG_INFO,
+    "%a: cpuid: phys-bits is %d\n",
+    __FUNCTION__,
+    PlatformInfoHob->PhysMemAddressWidth
+    ));
+}
+
 /**
   Initialize the PhysMemAddressWidth field in PlatformInfoHob based on guest RAM size.
 **/
@@ -503,6 +539,11 @@ PlatformAddressWidthInitialization (
   UINT64  FirstNonAddress;
   UINT8   PhysMemAddressWidth;
 
+  if (PlatformInfoHob->HostBridgeDevId == 0xffff /* microvm */) {
+    PlatformAddressWidthFromCpuid (PlatformInfoHob);
+    return;
+  }
+
   //
   // As guest-physical memory size grows, the permanent PEI RAM requirements
   // are dominated by the identity-mapping page tables built by the DXE IPL.
diff --git a/OvmfPkg/PlatformPei/Platform.c b/OvmfPkg/PlatformPei/Platform.c
index f006755d5fdb..009db67ee60a 100644
--- a/OvmfPkg/PlatformPei/Platform.c
+++ b/OvmfPkg/PlatformPei/Platform.c
@@ -357,12 +357,12 @@ InitializePlatform (
 
   S3Verification ();
   BootModeInitialization (&mPlatformInfoHob);
-  AddressWidthInitialization (&mPlatformInfoHob);
 
   //
   // Query Host Bridge DID
   //
   mPlatformInfoHob.HostBridgeDevId = PciRead16 (OVMF_HOSTBRIDGE_DID);
+  AddressWidthInitialization (&mPlatformInfoHob);
 
   MaxCpuCountInitialization (&mPlatformInfoHob);
 
-- 
2.36.1

[PATCH v7 6/6] OvmfPkg/Microvm/pcie: add pcie support

[Gerd Hoffmann]

Link in pcie and host bridge bits.  Enables support for PCIe in microvm
(qemu-system-x86_64 -M microvm,pcie=on).

Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3777
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 OvmfPkg/Microvm/MicrovmX64.dsc | 40 +++++++++++++++++++++-------------
 OvmfPkg/Microvm/README         |  2 +-
 2 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/OvmfPkg/Microvm/MicrovmX64.dsc b/OvmfPkg/Microvm/MicrovmX64.dsc
index f8fc977cb205..5b150a959c12 100644
--- a/OvmfPkg/Microvm/MicrovmX64.dsc
+++ b/OvmfPkg/Microvm/MicrovmX64.dsc
@@ -337,7 +337,9 @@ [LibraryClasses.common.DXE_RUNTIME_DRIVER]
 !endif
   UefiRuntimeLib|MdePkg/Library/UefiRuntimeLib/UefiRuntimeLib.inf
   BaseCryptLib|CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+#  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+#  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+#  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   VariablePolicyLib|MdeModulePkg/Library/VariablePolicyLib/VariablePolicyLibRuntimeDxe.inf
 
@@ -354,7 +356,9 @@ [LibraryClasses.common.UEFI_DRIVER]
   DebugLib|OvmfPkg/Library/PlatformDebugLibIoPort/PlatformDebugLibIoPort.inf
 !endif
   UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
 
 [LibraryClasses.common.DXE_DRIVER]
   PcdLib|MdePkg/Library/DxePcdLib/DxePcdLib.inf
@@ -376,7 +380,9 @@ [LibraryClasses.common.DXE_DRIVER]
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/DxeDebugAgentLib.inf
 !endif
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
   MpInitLib|UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf
   QemuFwCfgS3Lib|OvmfPkg/Library/QemuFwCfgS3Lib/DxeQemuFwCfgS3LibFwCfg.inf
   QemuLoadImageLib|OvmfPkg/Library/X86QemuLoadImageLib/X86QemuLoadImageLib.inf
@@ -392,7 +398,9 @@ [LibraryClasses.common.UEFI_APPLICATION]
 !else
   DebugLib|OvmfPkg/Library/PlatformDebugLibIoPort/PlatformDebugLibIoPort.inf
 !endif
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
 
 [LibraryClasses.common.DXE_SMM_DRIVER]
   PcdLib|MdePkg/Library/DxePcdLib/DxePcdLib.inf
@@ -413,7 +421,9 @@ [LibraryClasses.common.DXE_SMM_DRIVER]
   DebugAgentLib|SourceLevelDebugPkg/Library/DebugAgent/SmmDebugAgentLib.inf
 !endif
   BaseCryptLib|CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
 
 [LibraryClasses.common.SMM_CORE]
   PcdLib|MdePkg/Library/DxePcdLib/DxePcdLib.inf
@@ -429,7 +439,9 @@ [LibraryClasses.common.SMM_CORE]
 !else
   DebugLib|OvmfPkg/Library/PlatformDebugLibIoPort/PlatformDebugLibIoPort.inf
 !endif
-  PciLib|OvmfPkg/Library/DxePciLibI440FxQ35/DxePciLibI440FxQ35.inf
+  PciLib|MdePkg/Library/BasePciLibPciExpress/BasePciLibPciExpress.inf
+  PciPcdProducerLib|OvmfPkg/Fdt/FdtPciPcdProducerLib/FdtPciPcdProducerLib.inf
+  PciExpressLib|OvmfPkg/Library/BaseCachingPciExpressLib/BaseCachingPciExpressLib.inf
 
 ################################################################################
 #
@@ -504,14 +516,6 @@ [PcdsFixedAtBuild]
   gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask|0x2F
 !endif
 
-  # This PCD is used to set the base address of the PCI express hierarchy. It
-  # is only consulted when OVMF runs on Q35. In that case it is programmed into
-  # the PCIEXBAR register.
-  #
-  # On Q35 machine types that QEMU intends to support in the long term, QEMU
-  # never lets the RAM below 4 GB exceed 2816 MB.
-  gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress|0xB0000000
-
 !if $(SOURCE_DEBUG_ENABLE) == TRUE
   gEfiSourceLevelDebugPkgTokenSpaceGuid.PcdDebugLoadImageMethod|0x2
 !endif
@@ -579,6 +583,12 @@ [PcdsDynamicDefault]
   gEfiMdePkgTokenSpaceGuid.PcdFSBClock|1000000000
   gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut|0
 
+  # set PcdPciExpressBaseAddress to MAX_UINT64, which signifies that this
+  # PCD and PcdPciDisableBusEnumeration below have not been assigned yet
+  gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress|0xFFFFFFFFFFFFFFFF
+  gEfiMdePkgTokenSpaceGuid.PcdPciIoTranslation|0x0
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration|TRUE
+
   # Set video resolution for text setup.
   gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoHorizontalResolution|640
   gEfiMdeModulePkgTokenSpaceGuid.PcdSetupVideoVerticalResolution|480
@@ -679,7 +689,7 @@ [Components]
   OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.inf
   MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridgeDxe.inf {
     <LibraryClasses>
-      PciHostBridgeLib|MdeModulePkg/Library/PciHostBridgeLibNull/PciHostBridgeLibNull.inf
+      PciHostBridgeLib|OvmfPkg/Fdt/FdtPciHostBridgeLib/FdtPciHostBridgeLib.inf
       PciHostBridgeUtilityLib|OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.inf
       NULL|OvmfPkg/Library/PlatformHasIoMmuLib/PlatformHasIoMmuLib.inf
   }
diff --git a/OvmfPkg/Microvm/README b/OvmfPkg/Microvm/README
index 540d39f2ec21..813920d92a60 100644
--- a/OvmfPkg/Microvm/README
+++ b/OvmfPkg/Microvm/README
@@ -29,7 +29,7 @@ features
  [working] serial console
  [working] direct kernel boot
  [working] virtio-mmio support
- [in progress] pcie support
+ [working] pcie support
 
 known limitations
 -----------------
-- 
2.36.1

Re: [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Ni, Ray]

Gerd,
The fix should work. But I am curious why the if in blow is not hit.
https://github.com/TianoCore/edk2/blob/master/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c#L922

The RootBridge->ResAllocNode[TypeIo].Status is assigned as ResNone in:
https://github.com/TianoCore/edk2/blob/master/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c#L263

Or maybe your OVMF platform doesn't execute the above code path?

Then I guess maybe the Base/Limit of Aperture is not set properly so that below if is hit?
https://github.com/tianocore/edk2/blob/64706ef761273ba403f9cb3b7a986bfb804c0a87/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c#L249

Thanks,
Ray

> -----Original Message-----
> From: Gerd Hoffmann <kraxel@redhat.com>
> Sent: Thursday, June 2, 2022 4:42 PM
> To: devel@edk2.groups.io
> Cc: Wu, Hao A <hao.a.wu@intel.com>; Pawel Polawski <ppolawsk@redhat.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>;
> Albecki, Mateusz <mateusz.albecki@intel.com>; Chang, Abner <abner.chang@hpe.com>; Ni, Ray <ray.ni@intel.com>; Leif
> Lindholm <quic_llindhol@quicinc.com>; Yao, Jiewen <jiewen.yao@intel.com>; Oliver Steffen <osteffen@redhat.com>; Gao,
> Liming <gaoliming@byosoft.com.cn>; Gerd Hoffmann <kraxel@redhat.com>; Wang, Jian J <jian.j.wang@intel.com>; Justen,
> Jordan L <jordan.l.justen@intel.com>; Ard Biesheuvel <ardb@kernel.org>
> Subject: [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory
> 
> io range is not mandatory according to pcie spec, so allow
> pcie host bridge configurations without io window in case
> there are no io reservations.
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
> b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
> index b20bcd310ad5..354be6dbb313 100644
> --- a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
> +++ b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
> @@ -1085,6 +1085,12 @@ NotifyPhase (
>                RootBridge->ResAllocNode[Index].Base   = BaseAddress;
>                RootBridge->ResAllocNode[Index].Status = ResAllocated;
>                DEBUG ((DEBUG_INFO, "Success\n"));
> +            } else if ((Index == TypeIo) &&
> +                       (RootBridge->Io.Base == MAX_UINT64) &&
> +                       (RootBridge->ResAllocNode[Index].Length == 0))
> +            {
> +              /* I/O is optional on PCIe */
> +              DEBUG ((DEBUG_INFO, "Success (PCIe NoIO)\n"));
>              } else {
>                ReturnStatus = EFI_OUT_OF_RESOURCES;
>                DEBUG ((DEBUG_ERROR, "Out Of Resource!\n"));
> --
> 2.36.1

Re: [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Ard Biesheuvel]

On Thu, 2 Jun 2022 at 12:14, Ni, Ray <ray.ni@intel.com> wrote:
>
> Gerd,
> The fix should work. But I am curious why the if in blow is not hit.
> https://github.com/TianoCore/edk2/blob/master/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c#L922
>
> The RootBridge->ResAllocNode[TypeIo].Status is assigned as ResNone in:
> https://github.com/TianoCore/edk2/blob/master/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c#L263
>
> Or maybe your OVMF platform doesn't execute the above code path?
>
> Then I guess maybe the Base/Limit of Aperture is not set properly so that below if is hit?
> https://github.com/tianocore/edk2/blob/64706ef761273ba403f9cb3b7a986bfb804c0a87/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c#L249
>

I did a quick test with both ArmVirtQemu and microvm (using this
series but omitting the MdeModulePkg), and I can confirm that not
having a I/O resource window at all seems to work fine if none of the
PCI devices have I/O BARs.

Gerd, do you remember why exactly this patch is needed? Is it related
to devices that have I/O BARs but don't actually require them to
function correctly?

Re: [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Gerd Hoffmann]

  Hi,

> I did a quick test with both ArmVirtQemu and microvm (using this
> series but omitting the MdeModulePkg), and I can confirm that not
> having a I/O resource window at all seems to work fine if none of the
> PCI devices have I/O BARs.
> 
> Gerd, do you remember why exactly this patch is needed? Is it related
> to devices that have I/O BARs but don't actually require them to
> function correctly?

Well, the difference seem to be pcie root ports.  When plugging my
virtio device into the root bus everything is fine:

  PCI Bus First Scanning
  PciBus: Discovered PCI @ [00|00|00]

  PciBus: Discovered PCI @ [00|01|00]
     BAR[1]: Type =  Mem32; Alignment = 0xFFF;	Length = 0x1000;	Offset = 0x14
     BAR[4]: Type = PMem64; Alignment = 0x3FFF;	Length = 0x4000;	Offset = 0x20
  [ ... ]
  PciHostBridge: NotifyPhase (AllocateResources)
   RootBridge: PciRoot(0x0)
    Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
    Mem: Base/Length/Alignment = C0000000/100000/FFFFF - Success
  PciBus: HostBridge->NotifyPhase(AllocateResources) - Success

When plugging the virtio device into a pcie root port it doesn't work
and the log looks like this:

  PCI Bus First Scanning
  PciBus: Discovered PCI @ [00|00|00]

  PciBus: Discovered PPB @ [00|08|00]
     Padding: Type =  Mem32; Alignment = 0x1FFFFF;	Length = 0x200000
     Padding: Type =     Io; Alignment = 0x1FF;	Length = 0x200
     BAR[0]: Type =  Mem32; Alignment = 0xFFF;	Length = 0x1000;	Offset = 0x10

  PciBus: Discovered PCI @ [01|00|00]
     BAR[1]: Type =  Mem32; Alignment = 0xFFF;	Length = 0x1000;	Offset = 0x14
     BAR[4]: Type = PMem64; Alignment = 0x3FFF;	Length = 0x4000;	Offset = 0x20
  [ ... ]
  PciHostBridge: NotifyPhase (AllocateResources)
   RootBridge: PciRoot(0x0)
    Mem: Base/Length/Alignment = C0000000/300000/1FFFFF - Success
    Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
    I/O: Base/Length/Alignment = FFFFFFFFFFFFFFFF/1000/FFF - Out Of Resource!
  [ ... ]
  PciHostBridge: NotifyPhase (AllocateResources)
   RootBridge: PciRoot(0x0)
    Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
    Mem: Base/Length/Alignment = C0000000/200000/FFFFF - Success
    I/O: Base/Length/Alignment = FFFFFFFFFFFFFFFF/0/FFF - Out Of Resource!

So, it's apparently the io window of the pcie root port which causes
edk2 try allocate io resources.

take care,
  Gerd

Re: [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Ard Biesheuvel]

On Thu, 2 Jun 2022 at 15:15, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > I did a quick test with both ArmVirtQemu and microvm (using this
> > series but omitting the MdeModulePkg), and I can confirm that not
> > having a I/O resource window at all seems to work fine if none of the
> > PCI devices have I/O BARs.
> >
> > Gerd, do you remember why exactly this patch is needed? Is it related
> > to devices that have I/O BARs but don't actually require them to
> > function correctly?
>
> Well, the difference seem to be pcie root ports.  When plugging my
> virtio device into the root bus everything is fine:
>
>   PCI Bus First Scanning
>   PciBus: Discovered PCI @ [00|00|00]
>
>   PciBus: Discovered PCI @ [00|01|00]
>      BAR[1]: Type =  Mem32; Alignment = 0xFFF;  Length = 0x1000;        Offset = 0x14
>      BAR[4]: Type = PMem64; Alignment = 0x3FFF; Length = 0x4000;        Offset = 0x20
>   [ ... ]
>   PciHostBridge: NotifyPhase (AllocateResources)
>    RootBridge: PciRoot(0x0)
>     Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
>     Mem: Base/Length/Alignment = C0000000/100000/FFFFF - Success
>   PciBus: HostBridge->NotifyPhase(AllocateResources) - Success
>
> When plugging the virtio device into a pcie root port it doesn't work
> and the log looks like this:
>
>   PCI Bus First Scanning
>   PciBus: Discovered PCI @ [00|00|00]
>
>   PciBus: Discovered PPB @ [00|08|00]
>      Padding: Type =  Mem32; Alignment = 0x1FFFFF;      Length = 0x200000
>      Padding: Type =     Io; Alignment = 0x1FF; Length = 0x200
>      BAR[0]: Type =  Mem32; Alignment = 0xFFF;  Length = 0x1000;        Offset = 0x10
>
>   PciBus: Discovered PCI @ [01|00|00]
>      BAR[1]: Type =  Mem32; Alignment = 0xFFF;  Length = 0x1000;        Offset = 0x14
>      BAR[4]: Type = PMem64; Alignment = 0x3FFF; Length = 0x4000;        Offset = 0x20
>   [ ... ]
>   PciHostBridge: NotifyPhase (AllocateResources)
>    RootBridge: PciRoot(0x0)
>     Mem: Base/Length/Alignment = C0000000/300000/1FFFFF - Success
>     Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
>     I/O: Base/Length/Alignment = FFFFFFFFFFFFFFFF/1000/FFF - Out Of Resource!
>   [ ... ]
>   PciHostBridge: NotifyPhase (AllocateResources)
>    RootBridge: PciRoot(0x0)
>     Mem64: Base/Length/Alignment = 6000000000/100000/FFFFF - Success
>     Mem: Base/Length/Alignment = C0000000/200000/FFFFF - Success
>     I/O: Base/Length/Alignment = FFFFFFFFFFFFFFFF/0/FFF - Out Of Resource!
>
> So, it's apparently the io window of the pcie root port which causes
> edk2 try allocate io resources.
>

This seems to be related to the padding logic, i.e., we are trying to
preserve some extra I/O space for the root port in case we hotplug
something that might need it.

The hack below gets around this - we'll need something suitable check
here that avoids I/O padding when the root port has not I/O resource
window in the first place. Care to cook something up?



--- a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
+++ b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
@@ -733,7 +733,7 @@ GetResourcePadding (
     }
   }

-  if (DefaultIo) {
+  if (DefaultIo && FALSE) {
     //
     // Request defaults.
     //

Re: [edk2-devel] [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Gerd Hoffmann]

  Hi,
 
> This seems to be related to the padding logic, i.e., we are trying to
> preserve some extra I/O space for the root port in case we hotplug
> something that might need it.

> --- a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
> +++ b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
> @@ -733,7 +733,7 @@ GetResourcePadding (
>      }
>    }
> 
> -  if (DefaultIo) {
> +  if (DefaultIo && FALSE) {
>      //
>      // Request defaults.
>      //

Oh, *there* it comes from.  Given this is configurable already we can
fix that one in qemu with a microvm tweak:

diff --git a/hw/i386/microvm.c b/hw/i386/microvm.c
index 4b3b1dd262f1..f01d972f5d28 100644
--- a/hw/i386/microvm.c
+++ b/hw/i386/microvm.c
@@ -757,6 +757,12 @@ static void microvm_class_init(ObjectClass *oc, void *data)
         "Set off to disable adding virtio-mmio devices to the kernel cmdline");
 
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_RAMFB_DEVICE);
+
+    /*
+     * pcie host bridge (gpex) on microvm has no io address window,
+     * so reserving io space is not going to work.  Turn it off.
+     */
+    object_register_sugar_prop("pcie-root-port", "io-reserve", "0", true);
 }
 
 static const TypeInfo microvm_machine_info = {

So, I think we can drop patch #1.  Want me respin the series, or can you
simply drop the patch on merge?

thanks,
  Gerd

Re: [edk2-devel] [PATCH v7 1/6] MdeModulePkg/PciHostBridge: io range is not mandatory

[Ard Biesheuvel]

On Fri, 3 Jun 2022 at 10:29, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > This seems to be related to the padding logic, i.e., we are trying to
> > preserve some extra I/O space for the root port in case we hotplug
> > something that might need it.
>
> > --- a/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
> > +++ b/OvmfPkg/PciHotPlugInitDxe/PciHotPlugInit.c
> > @@ -733,7 +733,7 @@ GetResourcePadding (
> >      }
> >    }
> >
> > -  if (DefaultIo) {
> > +  if (DefaultIo && FALSE) {
> >      //
> >      // Request defaults.
> >      //
>
> Oh, *there* it comes from.  Given this is configurable already we can
> fix that one in qemu with a microvm tweak:
>
> diff --git a/hw/i386/microvm.c b/hw/i386/microvm.c
> index 4b3b1dd262f1..f01d972f5d28 100644
> --- a/hw/i386/microvm.c
> +++ b/hw/i386/microvm.c
> @@ -757,6 +757,12 @@ static void microvm_class_init(ObjectClass *oc, void *data)
>          "Set off to disable adding virtio-mmio devices to the kernel cmdline");
>
>      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_RAMFB_DEVICE);
> +
> +    /*
> +     * pcie host bridge (gpex) on microvm has no io address window,
> +     * so reserving io space is not going to work.  Turn it off.
> +     */
> +    object_register_sugar_prop("pcie-root-port", "io-reserve", "0", true);
>  }
>
>  static const TypeInfo microvm_machine_info = {
>
> So, I think we can drop patch #1.  Want me respin the series, or can you
> simply drop the patch on merge?
>

I've already queued it up.

Re: [PATCH v7 0/6] OvmfPkg/Microvm/pcie: add pcie support

[Ard Biesheuvel]

On Thu, 2 Jun 2022 at 10:42, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> Needs two little tweaks in PCI code because microvm supports mmio only.
> Other than that just wire up the existing code (the PCIe host adapter
> used by microvm is the same (virtual) hardware used by the arm/aarch64
> virtual machines).
>
> v7:
>  - allow non-existing io address space only in case
>    there are no io reservations (Mateusz Albecki)
>
> v6:
>  - codestyle fix (Abner Chang).
>
> v5:
>  - codestyle (uncrustify) fix.
>
> v4:
>  - update PciHostBridge check (Abner Chang).
>
> v3:
>  - rebase to latest master, adapt to PlatformInitLib.
>  - rework PhysMemAddressWidth handling for microvm.
>
> v2:
>  - rebase to latest master
>  - pick up review tags
>
> Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3777
>
> Gerd Hoffmann (6):
>   MdeModulePkg/PciHostBridge: io range is not mandatory

Dropped this one

>   OvmfPkg/FdtPciHostBridgeLib: io range is not mandatory
>   OvmfPkg/Platform: unfix PcdPciExpressBaseAddress
>   OvmfPkg/Microvm/pcie: no vbeshim please
>   OvmfPkg/Microvm/pcie: mPhysMemAddressWidth tweak
>   OvmfPkg/Microvm/pcie: add pcie support
>

... and merged the rest as #2941

Thanks,

>  OvmfPkg/Microvm/MicrovmX64.dsc                | 40 ++++++++++-------
>  .../PlatformInitLib/PlatformInitLib.inf       |  4 +-
>  OvmfPkg/PlatformPei/PlatformPei.inf           |  2 +-
>  .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c  |  6 +++
>  .../FdtPciHostBridgeLib/FdtPciHostBridgeLib.c | 45 ++++++++++---------
>  OvmfPkg/Library/PlatformInitLib/MemDetect.c   | 45 ++++++++++++++++++-
>  OvmfPkg/Library/PlatformInitLib/Platform.c    |  4 +-
>  OvmfPkg/PlatformPei/Platform.c                |  2 +-
>  OvmfPkg/QemuVideoDxe/VbeShim.c                |  2 +
>  OvmfPkg/Microvm/README                        |  2 +-
>  10 files changed, 107 insertions(+), 45 deletions(-)
>
> --
> 2.36.1
>