[PATCH 0/5] Handling of multiple PCI host bridges with pre-populated resources in Ovmf.

[PATCH 0/5] Handling of multiple PCI host bridges with pre-populated resources in Ovmf.

[Nicolas Ojeda Leon]

Increased control is provided in Ovmf platforms to define and configure
the specifications of multiple PCI host bridges in the hypervisor. The
host propagates this information to the guest through fw-cfg interface.

In some AWS EC2 platforms, we expose a PCI topology including several
root bridges portraying information about physical distribution
that enables the guest to optimize accesses. Current PCI driver for
Ovmf platforms supports a single host bridges and EDK2 contains the
logic to determine its MMIO resources. However, we need a way to define
more than 1 host bridge and control, from the hypervisor, how many and
which resources each of them can use. For this reason, this patch series
introduces a mechanism to provide bus number, pxm number, bus number
ranges as well as 32 and 64- bit prefetchable and non--prefetchable MMIO
ranges through a fw-cfg item created by the hypervisor and consumed by
the guest firmware.

Furthermore, in these kind of high performance platforms, we exploit
PCIe features like Access Control Services to configure peer-to-peer
channels between devices. This allows us to create direct communication
channels that do not require packets to reach the Root Complex but
instead can follow a direct path from source to target. To enable Guest
Virtual Machines to profit from this performance improvement, we
configure resources (BARs) of peer-to-peer intended devices with Host
Physical Addresses. In this scenario, devices can be instructed, from
the guest VM, to perform DMA operations targeting a peer address space,
and the PCIe fabric can take care of directly routing them. Therefore
long and busy links towards the Root Complex are avoided. When we
configure resources this way, the guest must respect the pre-populated
BARs so that devices preserve the address ranges configured in the
apertures of physical PCIe ports that enable routing at the hardware
level.

Nicolas Ojeda Leon (5):
  OvmfPkg/PlatformPei: Extend 64-bit PCI range for multiple host bridges
  OvmfPkg/PciHostBridgeUtilityLib: Initialize RootBridges apertures with
    spec
  MdeModulePkg OvmfPkg: Add Pcd token for PCI pre-populated BARs
  MdeModulePkg/Pci MdePkg: Create service to retrieve PCI base addresses
  MdeModulePkg/PciBusDxe: Handling of pre-populated PCI BARs

 MdeModulePkg/MdeModulePkg.dec                 |   6 +
 OvmfPkg/OvmfPkgX64.dsc                        |   1 +
 MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf  |   1 +
 .../PciHostBridgeUtilityLib.inf               |   4 +
 OvmfPkg/PlatformPei/PlatformPei.inf           |   1 +
 MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h       |   1 +
 .../Bus/Pci/PciBusDxe/PciResourceSupport.h    |  20 ++
 .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.h  |  29 ++
 .../PciHostBridgeResourceAllocation.h         |  33 ++
 .../Include/Library/PciHostBridgeInfoLib.h    |  43 +++
 .../Bus/Pci/PciBusDxe/PciEnumeratorSupport.c  |   1 +
 MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c       |  21 ++
 .../Bus/Pci/PciBusDxe/PciResourceSupport.c    | 281 +++++++++++++++++-
 .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c  |  80 +++++
 .../PciHostBridgeUtilityLib.c                 | 145 ++++++++-
 OvmfPkg/PlatformPei/MemDetect.c               |  83 ++++++
 16 files changed, 746 insertions(+), 4 deletions(-)
 create mode 100644 OvmfPkg/Include/Library/PciHostBridgeInfoLib.h

-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

[PATCH 1/5] OvmfPkg/PlatformPei: Extend 64-bit PCI range for multiple host bridges

[Nicolas Ojeda Leon]

Create host_bridge_info struct describing PCI host bridges' resources
so that one or more host bridges can be specified by the hypervisor
and their information propagated through fw-cfg. The host is able to
determine the bus number, the bus number range, the NUMA proximity and
all MMIO resources each root bridge uses (low and high memory both
prefetchable and non-prefetchable).

With the increased control we give for the host to define the resources
each root bridge can use, the case may arise in which one of the
ranges, particularly the 64-bit MMIO window, exceeds the usable range.
Taking into account that PcdPciMmio64Size token determines, together with
PcdPciMmio64Base, the usable 64-bit range, we verify if the MMIO windows
set in the specification fit inside the valid range. If one of the root
bridges is set to use a higher memory address space, PcdPciMmio64Size
token is increased so that the intended window lies inside the valid range.

For example, using PcdPciMmio64Size of 0x800000000 and a runtime calculated
base address also of 0x800000000, the high memory usable addresses are
restricted to 0x1000000000. With the increased flexibility to specify
host bridge resources, a feasible example would be:

struct host_bridge_info pci_hb_spec[] = {
        {
                .set_pxm                                    = true,
                .mark_prefetchable_resources_as_cacheable   = true,

                .root_bus_nr            = 0x00,
                .pxm                    = 1,
                .num_hot_plug_slots     = 32,

                .root_bur_nr_start      = 0x00,
                .root_bus_nr_end        = 0x7F,

                .lowmem_start           = 0xC0000000,
                .lowmem_end             = 0xD0000000 - 1,
                .lowmem_pref_start      = 0xFFFFFFFF,
                .lowmem_pref_end        = 0xFFFFFFFF,

                .highmem_start          = 0xFFFFFFFFFFFFFFFF,
                .highmem_end            = 0xFFFFFFFFFFFFFFFF,
                .highmem_pref_start     = 0x800000000,
                .highmem_pref_end       = 0x1000000000 - 1
        },
        {
                .set_pxm                                        = true,
                .mark_prefetchable_resources_as_cacheable       = true,

                .root_bus_nr            = 0x80,
                .pxm                    = 1,
                .num_hot_plug_slots     = 32,

                .root_bur_nr_start      = 0x80,
                .root_bus_nr_end        = 0xFF,

                .lowmem_start           = 0xD0000000,
                .lowmem_end             = 0xE0000000 - 1,
                .lowmem_pref_start      = 0xFFFFFFFF,
                .lowmem_pref_end        = 0xFFFFFFFF,

                .highmem_start          = 0xFFFFFFFFFFFFFFFF,
                .highmem_end            = 0xFFFFFFFFFFFFFFFF,
                .highmem_pref_start     = 0x1000000000,
                .highmem_pref_end       = 0x1800000000 - 1
        }
};

The first host bridge already uses all the available space. Consequently,
the 64-bit PCI size needs to be extended. For that, the resource spec
exemplified is read from fw-cfg and PcdPciMmio64Size extended to
0x1000000000 so that both host bridges fit inside the valid window.

Signed-off-by: Nicolas Ojeda Leon <ncoleon@amazon.com>
Cc: Alexander Graf <graf@amazon.de>
---
 .../Include/Library/PciHostBridgeInfoLib.h    | 43 ++++++++++
 OvmfPkg/PlatformPei/MemDetect.c               | 83 +++++++++++++++++++
 2 files changed, 126 insertions(+)
 create mode 100644 OvmfPkg/Include/Library/PciHostBridgeInfoLib.h

diff --git a/OvmfPkg/Include/Library/PciHostBridgeInfoLib.h b/OvmfPkg/Include/Library/PciHostBridgeInfoLib.h
new file mode 100644
index 0000000000..e09e6c496e
--- /dev/null
+++ b/OvmfPkg/Include/Library/PciHostBridgeInfoLib.h
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * This program and the accompanying materials
+ * are licensed and made available under the terms and conditions of the BSD License
+ * which accompanies this distribution.  The full text of the license may be found at
+ * http://opensource.org/licenses/bsd-license.php
+ *
+ * THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+ */
+
+#ifndef __PCI_HOST_BRIDGE_INFO_LIB_H__
+#define __PCI_HOST_BRIDGE_INFO_LIB_H__
+
+//
+// Root bridge resource information for parsing fw-cfg data
+//
+#pragma pack(1)
+typedef struct {
+  BOOLEAN set_pxm;
+  BOOLEAN mark_prefetchable_resources_as_cacheable;
+
+  UINT8 root_bus_nr;
+  UINT8 pxm;
+  UINT8 num_hot_plug_slots;
+
+  UINT8 root_bur_nr_start;
+  UINT8 root_bus_nr_end;
+
+  UINT32 lowmem_start;
+  UINT32 lowmem_end;
+  UINT32 lowmem_pref_start;
+  UINT32 lowmem_pref_end;
+
+  UINT64 highmem_start;
+  UINT64 highmem_end;
+  UINT64 highmem_pref_start;
+  UINT64 highmem_pref_end;
+} HOST_BRIDGE_INFO;
+#pragma pack()
+
+#endif
diff --git a/OvmfPkg/PlatformPei/MemDetect.c b/OvmfPkg/PlatformPei/MemDetect.c
index 2c2c4641ec..2872e805a4 100644
--- a/OvmfPkg/PlatformPei/MemDetect.c
+++ b/OvmfPkg/PlatformPei/MemDetect.c
@@ -30,6 +30,7 @@ Module Name:
 #include <Library/MemEncryptSevLib.h>
 #include <Library/PcdLib.h>
 #include <Library/PciLib.h>
+#include <Library/PciHostBridgeInfoLib.h>
 #include <Library/PeimEntryPoint.h>
 #include <Library/ResourcePublicationLib.h>
 #include <Library/MtrrLib.h>
@@ -323,6 +324,83 @@ GetSystemMemorySizeAbove4gb (
 }
 
 
+/**
+  Iterate over the PCI host bridges resources information optionally provided
+  in fw-cfg.
+
+  Find the highest address used by the PCI bridges in 64-bit MMIO space to
+  calculate and modify the PCI aperture size accordingly (PciMmio64Size)
+
+  @param[in] PciMmio64Base    Base address (start) of the 64-bit PCI MMIO
+                              address space.
+
+  @param[inout] PciMmio64Size Size of the PCI 64-bit MMIO aperture provided
+                              as input and modified (output) if the resources
+                              indicated by fw_cfg require a larger address
+                              space.
+
+  @retval EFI_SUCCESS         The fw_cfg host-bridges-info was found and
+                              processed.
+
+  @retval EFI_PROTOCOL_ERROR  The host bridges information file was found,
+                              but its size wasn't a whole multiple of
+                              sizeof(HOST_BRIDGE_INFO). No entry was processed.
+
+  @retval EFI_NOT_FOUND       fw-cfg file with host bridges information was not
+                              found. Does not constitute an errro since the file
+                              is optional and used in special cases.
+
+  @retval EFI_UNSUPPORTED     fw-cfg is unavailable
+
+**/
+STATIC
+EFI_STATUS
+ScanPci64BitApertureSize (
+  IN UINT64         PciMmio64Base,
+  IN OUT UINT64     *PciMmio64Size
+  )
+{
+  EFI_STATUS           Status;
+  FIRMWARE_CONFIG_ITEM FwCfgItem;
+  HOST_BRIDGE_INFO     HostBridge;
+  UINTN                FwCfgSize;
+  UINTN                Processed;
+  UINT64               PciEnd;
+  UINT64               PcdPciEnd;
+
+  Status = QemuFwCfgFindFile ("etc/host-bridge-info", &FwCfgItem, &FwCfgSize);
+  if (EFI_ERROR (Status)) {
+    return Status;
+  }
+  if (FwCfgSize % sizeof HostBridge != 0) {
+    return EFI_PROTOCOL_ERROR;
+  }
+
+  PciEnd = 0;
+  QemuFwCfgSelectItem (FwCfgItem);
+  for (Processed = 0; Processed < FwCfgSize; Processed += sizeof HostBridge) {
+    QemuFwCfgReadBytes (sizeof HostBridge, &HostBridge);
+
+    if (HostBridge.highmem_end != MAX_UINT64 &&
+        HostBridge.highmem_end > PciEnd) {
+      PciEnd = HostBridge.highmem_end;
+    }
+    if (HostBridge.highmem_pref_end != MAX_UINT64 &&
+        HostBridge.highmem_pref_end > PciEnd) {
+      PciEnd = HostBridge.highmem_pref_end;
+    }
+  }
+
+  PcdPciEnd = PciMmio64Base + *PciMmio64Size;
+  if (PciEnd > PcdPciEnd) {
+    *PciMmio64Size = PciEnd - PciMmio64Base;
+    *PciMmio64Size = ALIGN_VALUE (*PciMmio64Size, (UINT64)SIZE_1GB);
+  }
+
+  return EFI_SUCCESS;
+}
+
+
 /**
   Return the highest address that DXE could possibly use, plus one.
 **/
@@ -452,6 +530,11 @@ GetFirstNonAddress (
   //
   Pci64Base = ALIGN_VALUE (Pci64Base, GetPowerOfTwo64 (Pci64Size));
 
+  //
+  // Extend Pci64Size if fw_cfg Host bridges specification requires it
+  //
+  ScanPci64BitApertureSize (Pci64Base, &Pci64Size);
+
   if (mBootMode != BOOT_ON_S3_RESUME) {
     //
     // The core PciHostBridgeDxe driver will automatically add this range to
-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

[PATCH 2/5] OvmfPkg/PciHostBridgeUtilityLib: Initialize RootBridges apertures with spec

[Nicolas Ojeda Leon]

Considering the "host-bridge-info" item from fw-cfg, the apertures for
all root bridges are initialized and used in the creation of the
root bridge objects that determine the resources each host bridge
can use.

PCI MMIO apertures are merged to produce a single low memory and a
single high memory apertures matching current Ovmf allocation strategy
which includes the EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM attribute.

Prefetchable and non-prefetchable ranges are merged together to
produce a single aperture covering both ranges. Therefore, if both
prefetchable and non-prefetchable apertures are defined for one of
the address spaces (32-bit or 64-bit), these 2 must be contiguous.

For platforms having multiple PCI host bridges, hypervisor builds and
propagates the resource specification (MMIO apertures) for each of the
host bridges over fw-cfg. These specifications determine the usable
windows in low and high memory, overwriting any calculated range and
providing both control for the hypervisor to configure the apertures
as well as possibility to use multiple host bridges.

Signed-off-by: Nicolas Ojeda Leon <ncoleon@amazon.com>
Cc: Alexander Graf <graf@amazon.de>
---
 .../PciHostBridgeUtilityLib.c                 | 128 +++++++++++++++++-
 1 file changed, 126 insertions(+), 2 deletions(-)

diff --git a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
index d2296f3308..7567da8423 100644
--- a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
+++ b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
@@ -18,6 +18,7 @@
 #include <Library/MemoryAllocationLib.h>
 #include <Library/PciHostBridgeUtilityLib.h>
 #include <Library/PciLib.h>
+#include <Library/PciHostBridgeInfoLib.h>
 #include <Library/QemuFwCfgLib.h>
 
 
@@ -186,6 +187,66 @@ PciHostBridgeUtilityUninitRootBridge (
   FreePool (RootBus->DevicePath);
 }
 
+/**
+  Interpret the MMIO resources in HostBridge and set the apertures
+  in 32-bit space (Mem) and 64-bit space (MemABove4G) accordingly.
+
+  The 2 types of apertures in each space (prefetchable and
+  non-prefetchable) are merged into a single window, hence if both
+  types of apertures are defined for a root bridges, these must be
+  contiguous.
+
+  @param[in]  HostBridge   Root bridge's resources specification
+
+  @param[out] Mem          32-bit MMIO aperture
+
+  @param[out] MemAbove4G   64-bit MMIO aperture
+**/
+STATIC
+VOID
+InitRootBridgeApertures (
+  IN  HOST_BRIDGE_INFO           *HostBridge,
+  OUT PCI_ROOT_BRIDGE_APERTURE   *Mem,
+  OUT PCI_ROOT_BRIDGE_APERTURE   *MemAbove4G
+
+  )
+{
+  if (HostBridge == NULL || Mem == NULL || MemAbove4G == NULL) {
+    return;
+  }
+
+  MemAbove4G->Base = MAX_UINT64;
+  MemAbove4G->Limit = 0;
+
+  if (HostBridge->highmem_start != MAX_UINT64 &&
+      HostBridge->highmem_end != MAX_UINT64) {
+    MemAbove4G->Base = HostBridge->highmem_start;
+    MemAbove4G->Limit = HostBridge->highmem_end;
+  }
+
+  if (HostBridge->highmem_pref_start != MAX_UINT64 &&
+      HostBridge->highmem_pref_end != MAX_UINT64) {
+    if (HostBridge->highmem_pref_start < MemAbove4G->Base)
+      MemAbove4G->Base = HostBridge->highmem_pref_start;
+
+    if (HostBridge->highmem_pref_end > MemAbove4G->Limit)
+      MemAbove4G->Limit = HostBridge->highmem_pref_end;
+  }
+
+  //
+  // Check if prefetchable range is valid and merge both ranges into a single
+  // 32-bit aperture.
+  // In case both prefetchable and non-prefetchable ranges are defined,
+  // these must be contiguous.
+  //
+  Mem->Base = HostBridge->lowmem_start;
+  if (HostBridge->lowmem_pref_end != MAX_UINT32) {
+    Mem->Limit = HostBridge->lowmem_pref_end;
+  } else {
+    Mem->Limit = HostBridge->lowmem_end;
+  }
+}
+
 
 /**
   Utility function to return all the root bridge instances in an array.
@@ -241,6 +302,7 @@ PciHostBridgeUtilityGetRootBridges (
   UINTN                Initialized;
   UINTN                LastRootBridgeNumber;
   UINTN                RootBridgeNumber;
+  HOST_BRIDGE_INFO     *RootBridgesInfo;
 
   *Count = 0;
 
@@ -277,13 +339,54 @@ PciHostBridgeUtilityGetRootBridges (
       __FUNCTION__, ExtraRootBridges));
   }
 
+  //
+  // Initialize RootBridgeInfo pointer, so in case no fw-cfg item is found
+  // the default configuration is used.
+  //
+  RootBridgesInfo =  NULL;
+  Initialized = 0;
+  Bridges = NULL;
+
+  //
+  // Hypervisor can provide, over fw-cfg, resource specifications for one or
+  // more PCI host bridges.
+  //
+  Status = QemuFwCfgFindFile ("etc/host-bridge-info", &FwCfgItem, &FwCfgSize);
+
+  if (!EFI_ERROR (Status)) {
+    UINTN RootBridgesInfoSize = (1 + (UINTN)ExtraRootBridges) *
+                                 sizeof(HOST_BRIDGE_INFO);
+    RootBridgesInfo = AllocatePool (RootBridgesInfoSize);
+
+    if (RootBridgesInfo == NULL) {
+      DEBUG ((DEBUG_ERROR,
+        "%a: Failed to allocate memory for resources info\n", __FUNCTION__));
+      return NULL;
+    }
+
+    //
+    // Read the resource information of all Root Bridges from fw-cfg
+    //
+    QemuFwCfgSelectItem (FwCfgItem);
+    QemuFwCfgReadBytes (FwCfgSize, RootBridgesInfo);
+
+    if (FwCfgSize != RootBridgesInfoSize) {
+      DEBUG ((DEBUG_ERROR,
+        "%a: Invalid HostBridgeInfo fw-cfg item size. Expected %ld, got %ld\n",
+        __FUNCTION__, RootBridgesInfoSize, FwCfgSize));
+        goto FreeBridges;
+    }
+    DEBUG ((DEBUG_INFO, "%a: Resources for %Lu root buses found in fw-cfg\n",
+      __FUNCTION__, ExtraRootBridges + 1));
+  }
+
   //
   // Allocate the "main" root bridge, and any extra root bridges.
   //
   Bridges = AllocatePool ((1 + (UINTN)ExtraRootBridges) * sizeof *Bridges);
   if (Bridges == NULL) {
     DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, EFI_OUT_OF_RESOURCES));
-    return NULL;
+    goto FreeBridges;
   }
   Initialized = 0;
 
@@ -309,6 +412,11 @@ PciHostBridgeUtilityGetRootBridges (
       }
     }
     if (Device <= PCI_MAX_DEVICE) {
+      //
+      // If resources obtained from fw-cfg, use those values
+      //
+      InitRootBridgeApertures(&RootBridgesInfo[Initialized], Mem, MemAbove4G);
+
       //
       // Found the next root bus. We can now install the *previous* one,
       // because now we know how big a bus number range *that* one has, for any
@@ -341,6 +449,8 @@ PciHostBridgeUtilityGetRootBridges (
   // Install the last root bus (which might be the only, ie. main, root bus, if
   // we've found no extra root buses).
   //
+  InitRootBridgeApertures(&RootBridgesInfo[Initialized], Mem, MemAbove4G);
+
   Status = PciHostBridgeUtilityInitRootBridge (
     Attributes,
     Attributes,
@@ -362,6 +472,14 @@ PciHostBridgeUtilityGetRootBridges (
   ++Initialized;
 
   *Count = Initialized;
+
+  //
+  // If resources were allocated for host bridges info, release them
+  //
+  if (RootBridgesInfo) {
+    FreePool(RootBridgesInfo);
+  }
+
   return Bridges;
 
 FreeBridges:
@@ -370,7 +488,13 @@ FreeBridges:
     PciHostBridgeUtilityUninitRootBridge (&Bridges[Initialized]);
   }
 
-  FreePool (Bridges);
+  if (Bridges) {
+    FreePool (Bridges);
+  }
+
+  if (RootBridgesInfo) {
+    FreePool(RootBridgesInfo);
+  }
   return NULL;
 }
 
-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

[PATCH 3/5] MdeModulePkg OvmfPkg: Add Pcd token for PCI pre-populated BARs

[Nicolas Ojeda Leon]

Create a new PCD boolean token in MdeModulePkg for global use.
We use this token to indicate if the configuration, parsed from
fw-cfg, requires pre-populated BARs to be preserved.

During creation of root bridges configurations, the flag is set
according to the "pre-populated-bars" item in fw-cfg. The Pcd token
is created as a dynamic item for consumption in both Pei and Dxe PCI
modules.

The token provides a globally accessible configuration flag to
determine, during PCI BAR allocation, if pre-populated BARs must
be respected. The pre-allocated PCI BARs are used in platforms in
which MMIO resources are configured with host physical addresses so
that DMA transactions can happen between PCI devices without packets
going through the IOMMU. Performance is improved due to PCI packets
travelling shorter distances and avoiding links reaching the Root
Complex, which can get busy during I/O intensive periods.

Signed-off-by: Nicolas Ojeda Leon <ncoleon@amazon.com>
Cc: Alexander Graf <graf@amazon.de>
---
 MdeModulePkg/MdeModulePkg.dec                   |  6 ++++++
 OvmfPkg/OvmfPkgX64.dsc                          |  1 +
 MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf    |  1 +
 .../PciHostBridgeUtilityLib.inf                 |  4 ++++
 OvmfPkg/PlatformPei/PlatformPei.inf             |  1 +
 .../PciHostBridgeUtilityLib.c                   | 17 +++++++++++++++++
 6 files changed, 30 insertions(+)

diff --git a/MdeModulePkg/MdeModulePkg.dec b/MdeModulePkg/MdeModulePkg.dec
index 133e04ee86..44a2150628 100644
--- a/MdeModulePkg/MdeModulePkg.dec
+++ b/MdeModulePkg/MdeModulePkg.dec
@@ -1902,6 +1902,12 @@
   # @Prompt Disable full PCI enumeration.
   gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration|FALSE|BOOLEAN|0x10000048
 
+  ## The flag to control preservation of pre-populated PCI BARs
+  #   TRUE  - Respect pre-populated PCI BARs
+  #   FALSE - No pre-populated BARs, place all BARs
+  # @Prompt Enable preservsation of pre-populated PCI BARs
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciPreservePopulatedMappings|FALSE|BOOLEAN|0x10000050
+
   ## Disk I/O - Number of Data Buffer block.
   # Define the size in block of the pre-allocated buffer. It provide better
   # performance for large Disk I/O requests.
diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc
index e56b83d95e..fff50355b0 100644
--- a/OvmfPkg/OvmfPkgX64.dsc
+++ b/OvmfPkg/OvmfPkgX64.dsc
@@ -605,6 +605,7 @@
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwWorkingBase|0
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareBase|0
 !endif
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciPreservePopulatedMappings|FALSE
   gEfiMdeModulePkgTokenSpaceGuid.PcdVideoHorizontalResolution|800
   gEfiMdeModulePkgTokenSpaceGuid.PcdVideoVerticalResolution|600
   gEfiMdeModulePkgTokenSpaceGuid.PcdAcpiS3Enable|FALSE
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
index e317169d9c..046876bb3b 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
@@ -107,6 +107,7 @@
   gEfiMdeModulePkgTokenSpaceGuid.PcdMrIovSupport                ## CONSUMES
   gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration    ## SOMETIMES_CONSUMES
   gEfiMdeModulePkgTokenSpaceGuid.PcdPcieResizableBarSupport     ## CONSUMES
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciPreservePopulatedMappings## CONSUMES
 
 [UserExtensions.TianoCore."ExtraFiles"]
   PciBusDxeExtra.uni
diff --git a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.inf b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.inf
index 83a734c172..3643ffd79e 100644
--- a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.inf
+++ b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.inf
@@ -39,5 +39,9 @@
   DebugLib
   DevicePathLib
   MemoryAllocationLib
+  PcdLib
   PciLib
   QemuFwCfgLib
+
+[Pcd]
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciPreservePopulatedMappings
diff --git a/OvmfPkg/PlatformPei/PlatformPei.inf b/OvmfPkg/PlatformPei/PlatformPei.inf
index 67eb7aa716..61bcf25999 100644
--- a/OvmfPkg/PlatformPei/PlatformPei.inf
+++ b/OvmfPkg/PlatformPei/PlatformPei.inf
@@ -93,6 +93,7 @@
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareSize
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableSize
   gEfiMdeModulePkgTokenSpaceGuid.PcdEmuVariableNvStoreReserved
+  gEfiMdeModulePkgTokenSpaceGuid.PcdPciPreservePopulatedMappings
   gEfiMdeModulePkgTokenSpaceGuid.PcdDxeIplSwitchToLongMode
   gEfiMdeModulePkgTokenSpaceGuid.PcdUse1GPageTable
   gEfiMdeModulePkgTokenSpaceGuid.PcdSetNxForStack
diff --git a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
index 7567da8423..0a9aac5359 100644
--- a/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
+++ b/OvmfPkg/Library/PciHostBridgeUtilityLib/PciHostBridgeUtilityLib.c
@@ -16,6 +16,7 @@
 #include <Library/DebugLib.h>
 #include <Library/DevicePathLib.h>
 #include <Library/MemoryAllocationLib.h>
+#include <Library/PcdLib.h>
 #include <Library/PciHostBridgeUtilityLib.h>
 #include <Library/PciLib.h>
 #include <Library/PciHostBridgeInfoLib.h>
@@ -298,6 +299,7 @@ PciHostBridgeUtilityGetRootBridges (
   FIRMWARE_CONFIG_ITEM FwCfgItem;
   UINTN                FwCfgSize;
   UINT64               ExtraRootBridges;
+  UINT64               PrePopulatedBars;
   PCI_ROOT_BRIDGE      *Bridges;
   UINTN                Initialized;
   UINTN                LastRootBridgeNumber;
@@ -339,6 +341,21 @@ PciHostBridgeUtilityGetRootBridges (
       __FUNCTION__, ExtraRootBridges));
   }
 
+  //
+  // Find file for pre-populated bars and set Pcd token if enabled
+  //
+  Status = QemuFwCfgFindFile ("etc/pre-populated-bars", &FwCfgItem, &FwCfgSize);
+  PrePopulatedBars = 0;
+  if (!EFI_ERROR (Status) && FwCfgSize == sizeof (PrePopulatedBars)) {
+    QemuFwCfgSelectItem (FwCfgItem);
+    QemuFwCfgReadBytes (FwCfgSize, &PrePopulatedBars);
+
+    if (PrePopulatedBars) {
+      PcdSetBoolS (PcdPciPreservePopulatedMappings, TRUE);
+      DEBUG ((DEBUG_INFO, "%a: Pre-populated BARs present\n", __FUNCTION__));
+    }
+  }
+
   //
   // Initialize RootBridgeInfo pointer, so in case no fw-cfg item is found
   // the default configuration is used.
-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

[PATCH 4/5] MdeModulePkg/Pci MdePkg: Create service to retrieve PCI base addresses

[Nicolas Ojeda Leon]

Extend the PCI host bridge resource allocation protocol to include one
more service that retrieves the base addresses of all resources of a
given root bridge.
The service is defined to provide, on runtime, the possibility to fetch
the base addresses of a root bridge, replicating the address alignment
used when placing the host bridge's resources in the Gcd memory map.

The intention of this service, initially, is to allow the PCI allocation
process to get the base addresses before allocating the individual BARs
grouped under a root bridge. This enables the placing logic to be
enhanced to account and calculate offsets for pre-populated BARs (PCI
devices' BARs that are already configured and need to be respected).

Signed-off-by: Nicolas Ojeda Leon <ncoleon@amazon.com>
Cc: Alexander Graf <graf@amazon.de>
---
 .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.h  | 29 +++++++
 .../PciHostBridgeResourceAllocation.h         | 33 ++++++++
 MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c       | 10 +++
 .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c  | 80 +++++++++++++++++++
 4 files changed, 152 insertions(+)

diff --git a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.h b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.h
index 755ab75b19..3b8432fb29 100644
--- a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.h
+++ b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.h
@@ -239,6 +239,35 @@ PreprocessController (
   IN EFI_PCI_CONTROLLER_RESOURCE_ALLOCATION_PHASE              Phase
   );
 
+/**
+
+  Retrieve the aligned base addresses for all resources of a root bridge.
+
+  @param This              The EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL instance.
+  @param RootBridgeHandle  RootBridgeHandle returned by GetNextRootBridge to locate the
+                           root bridge of interest among the list of root bridges.
+  @param IoBase            Returns the PIO aperture base address.
+  @param Mem32Base         Returns the 32-bit aperture base address.
+  @param PMem32Base        Returns the 32-bit prefetchable aperture base address.
+  @param Mem64Base         Returns the 64-bit aperture base address.
+  @param PMem64Base        Returns the 64-bit prefetchable aperture base address.
+
+  @retval EFI_SUCCESS      Succeed.
+  @retval EFI_NOT_FOUND    Root bridge was not found.
+
+**/
+EFI_STATUS
+EFIAPI
+GetResourcesBases(
+  IN EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL *This,
+  IN EFI_HANDLE                                       RootBridgeHandle,
+  OUT UINT64                                          *IoBase,
+  OUT UINT64                                          *Mem32Base,
+  OUT UINT64                                          *PMem32Base,
+  OUT UINT64                                          *Mem64Base,
+  OUT UINT64                                          *PMem64Base
+  );
+
 /**
   This routine constructs the resource descriptors for all root bridges and call PciHostBridgeResourceConflict().
 
diff --git a/MdePkg/Include/Protocol/PciHostBridgeResourceAllocation.h b/MdePkg/Include/Protocol/PciHostBridgeResourceAllocation.h
index 17b1b5a8d8..f42521a348 100644
--- a/MdePkg/Include/Protocol/PciHostBridgeResourceAllocation.h
+++ b/MdePkg/Include/Protocol/PciHostBridgeResourceAllocation.h
@@ -367,6 +367,33 @@ EFI_STATUS
   IN EFI_PCI_CONTROLLER_RESOURCE_ALLOCATION_PHASE      Phase
   );
 
+/**
+  Retrieves the base addresses of ost bridge resources.
+
+  @param This              The pointer to the EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL instance.
+  @param RootBridgeHandle  The PCI root bridge handle.
+  @param IoBase            The pointer to PIO aperture base address.
+  @param Mem32Base         The pointer to 32-bit aperture base address.
+  @param PMem32Base        The pointer to 32-bit prefetchable aperture base address.
+  @param Mem64Base         The pointer to 64-bit aperture base address.
+  @param PMem64Base        The pointer to 64-bit prefetchable aperture base address.
+
+  @retval EFI_SUCCESS      Succeed.
+  @retval EFI_NOT_FOUND    Root bridge was not found.
+
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL_GET_RESOURCES_BASES) (
+  IN EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL *This,
+  IN EFI_HANDLE                                       RootBridgeHandle,
+  OUT UINT64                                          *IoBase,
+  OUT UINT64                                          *Mem32Base,
+  OUT UINT64                                          *PMem32Base,
+  OUT UINT64                                          *Mem64Base,
+  OUT UINT64                                          *PMem64Base
+  );
+
 ///
 /// Provides the basic interfaces to abstract a PCI host bridge resource allocation.
 ///
@@ -415,6 +442,12 @@ struct _EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL {
   /// before enumeration.
   ///
   EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL_PREPROCESS_CONTROLLER  PreprocessController;
+
+  ///
+  /// Returns the aligned base addresses of the different resource windows
+  /// of the host bridge. Intended for use before resources are submitted.
+  ///
+  EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL_GET_RESOURCES_BASES    GetResourcesBases;
 };
 
 extern EFI_GUID gEfiPciHostBridgeResourceAllocationProtocolGuid;
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
index 4caac56f1d..82147a4946 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
@@ -561,6 +561,16 @@ PciHostBridgeResourceAllocator (
                        PciResUsageTypical
                        );
 
+      Status = PciResAlloc->GetResourcesBases(
+                              PciResAlloc,
+                              RootBridgeDev->Handle,
+                              &IoBase,
+                              &Mem32Base,
+                              &PMem32Base,
+                              &Mem64Base,
+                              &PMem64Base
+                              );
+
       //
       // Get the max ROM size that the root bridge can process
       // Insert to resource map so that there will be dedicate MEM32 resource range for Option ROM.
diff --git a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
index 4ab9415c96..0ba9e100fc 100644
--- a/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
+++ b/MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciHostBridge.c
@@ -527,6 +527,7 @@ InitializePciHostBridge (
     HostBridge->ResAlloc.SubmitResources = SubmitResources;
     HostBridge->ResAlloc.GetProposedResources = GetProposedResources;
     HostBridge->ResAlloc.PreprocessController = PreprocessController;
+    HostBridge->ResAlloc.GetResourcesBases = GetResourcesBases;
 
     Status = gBS->InstallMultipleProtocolInterfaces (
                     &HostBridge->Handle,
@@ -1594,3 +1595,82 @@ PreprocessController (
 
   return EFI_INVALID_PARAMETER;
 }
+
+/**
+
+  Retrieve the aligned base addresses for all resources of a root bridge.
+
+  @param This              The EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL instance.
+  @param RootBridgeHandle  RootBridgeHandle returned by GetNextRootBridge to locate the
+                           root bridge of interest among the list of root bridges.
+  @param IoBase            Returns the PIO aperture base address.
+  @param Mem32Base         Returns the 32-bit aperture base address.
+  @param PMem32Base        Returns the 32-bit prefetchable aperture base address.
+  @param Mem64Base         Returns the 64-bit aperture base address.
+  @param PMem64Base        Returns the 64-bit prefetchable aperture base address.
+
+  @retval EFI_SUCCESS      Succeed.
+  @retval EFI_NOT_FOUND    Root bridge was not found.
+
+**/
+EFI_STATUS
+EFIAPI
+GetResourcesBases(
+  IN EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL *This,
+  IN EFI_HANDLE                                       RootBridgeHandle,
+  OUT UINT64                                          *IoBase,
+  OUT UINT64                                          *Mem32Base,
+  OUT UINT64                                          *PMem32Base,
+  OUT UINT64                                          *Mem64Base,
+  OUT UINT64                                          *PMem64Base
+  )
+{
+  PCI_HOST_BRIDGE_INSTANCE              *HostBridge;
+  PCI_ROOT_BRIDGE_INSTANCE              *RootBridge;
+  LIST_ENTRY                            *Link;
+  UINT64                                Alignment;
+  UINTN                                 BitsOfAlignment;
+
+  HostBridge = PCI_HOST_BRIDGE_FROM_THIS(This);
+
+  for (Link = GetFirstNode (&HostBridge->RootBridges)
+          ; !IsNull (&HostBridge->RootBridges, Link)
+          ; Link = GetNextNode (&HostBridge->RootBridges, Link)
+          ) {
+      RootBridge = ROOT_BRIDGE_FROM_LINK (Link);
+
+      if (RootBridgeHandle == RootBridge->Handle) {
+        //
+        // Have to make sure Alignment is handled since we are doing direct address allocation
+        //
+        Alignment       = RootBridge->ResAllocNode[TypeIo].Alignment;
+        BitsOfAlignment = MIN (15, LowBitSet64 (Alignment + 1));
+        *IoBase         = ALIGN_VALUE (RootBridge->Io.Base, Alignment + 1);
+        *IoBase         = ALIGN_VALUE (*IoBase, LShiftU64 (1, BitsOfAlignment));
+
+        Alignment       = RootBridge->ResAllocNode[TypeMem32].Alignment;
+        BitsOfAlignment = MIN (31, LowBitSet64 (Alignment + 1));
+        *Mem32Base      = ALIGN_VALUE (RootBridge->Mem.Base, Alignment + 1);
+        *Mem32Base      = ALIGN_VALUE (*Mem32Base, LShiftU64 (1, BitsOfAlignment));
+
+        Alignment       = RootBridge->ResAllocNode[TypePMem32].Alignment;
+        BitsOfAlignment = MIN (31, LowBitSet64 (Alignment + 1));
+        *PMem32Base     = ALIGN_VALUE(RootBridge->PMem.Base, Alignment + 1);
+        *PMem32Base     = ALIGN_VALUE (*PMem32Base, LShiftU64 (1, BitsOfAlignment));
+
+        Alignment       = RootBridge->ResAllocNode[TypeMem64].Alignment;
+        BitsOfAlignment = MIN (63, LowBitSet64 (Alignment + 1));
+        *Mem64Base      = ALIGN_VALUE(RootBridge->MemAbove4G.Base, Alignment + 1);
+        *Mem64Base      = ALIGN_VALUE (*Mem64Base, LShiftU64 (1, BitsOfAlignment));
+
+        Alignment       = RootBridge->ResAllocNode[TypePMem64].Alignment;
+        BitsOfAlignment = MIN (63, LowBitSet64 (Alignment + 1));
+        *PMem64Base     = ALIGN_VALUE(RootBridge->PMemAbove4G.Base, Alignment + 1);
+        *PMem64Base     = ALIGN_VALUE (*PMem64Base, LShiftU64 (1, BitsOfAlignment));
+
+        return EFI_SUCCESS;
+      }
+    }
+
+  return EFI_NOT_FOUND;
+}
-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

[PATCH 5/5] MdeModulePkg/PciBusDxe: Handling of pre-populated PCI BARs

[Nicolas Ojeda Leon]

Extend the PCI BAR placement logic in order to consider pre-populated
resources first, if indicated by the Pcd token
PcdPciPreservePopulatedMappings.

The PCI_BAR type is augmented by one field for mapping the absolute
address of prepopulated BARs into a root bridge relative offset.
As part of the CreateResourceMap stage of the PCI resource allocation
process, all the resources belonging to a root bridge are analyzed and
if a BaseAddress (absolute address read from the PCI device during
enumeration) is present, it is translated into a root-bridge relative
offset and the space occupied by it is marked as used in the root
bridge's aperture. This process is performed before the regular
placement logic is executed.

The remaining unassigned BARs are then placed according to default
logic starting after the last pre-populated resource.

The motivation behind respecting pre-populated BARs is that the
hypervisor can decide on placement of some resources and the guest must
use the same values to benefit from performance optimizations or
specific requirements.

Signed-off-by: Nicolas Ojeda Leon <ncoleon@amazon.com>
Cc: Alexander Graf <graf@amazon.de>
---
 MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h       |   1 +
 .../Bus/Pci/PciBusDxe/PciResourceSupport.h    |  20 ++
 .../Bus/Pci/PciBusDxe/PciEnumeratorSupport.c  |   1 +
 MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c       |  11 +
 .../Bus/Pci/PciBusDxe/PciResourceSupport.c    | 281 +++++++++++++++++-
 5 files changed, 312 insertions(+), 2 deletions(-)

diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
index a619a68526..06e78ac40b 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
@@ -95,6 +95,7 @@ typedef enum {
 //
 struct _PCI_BAR {
   UINT64        BaseAddress;
+  UINT64        BaseAddressOffset;
   UINT64        Length;
   UINT64        Alignment;
   PCI_BAR_TYPE  BarType;
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h
index 9f2de291ba..5f4886d6ba 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h
@@ -111,6 +111,26 @@ CalculateApertureIo16 (
   IN PCI_RESOURCE_NODE    *Bridge
   );
 
+/**
+  Populate a root bridge's resource aperture according to
+  initial conditions ruled by individual pre-populated
+  resources
+
+   @note Pre-populated PIO BARs are not supported.
+
+   @param Bridge    PCI resource node for given bridge device.
+   @param Base      Resource aperture base address
+
+**/
+VOID
+PopulateResourceAperture (
+  IN     UINT64           Mem32Base,
+  IN     UINT64           PMem32Base,
+  IN     UINT64           Mem64Base,
+  IN     UINT64           PMem64Base,
+  IN OUT PCI_IO_DEVICE    *Bridge
+  );
+
 /**
   This function is used to calculate the resource aperture
   for a given bridge device.
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c
index db1b35f8ef..c76b37ab05 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c
@@ -1789,6 +1789,7 @@ PciParseBar (
 
   PciIoDevice->PciBar[BarIndex].BarTypeFixed = FALSE;
   PciIoDevice->PciBar[BarIndex].Offset = (UINT8) Offset;
+  PciIoDevice->PciBar[BarIndex].BaseAddressOffset = 0;
   if ((Value & 0x01) != 0) {
     //
     // Device I/Os
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
index 82147a4946..bf82d8fdaf 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c
@@ -571,6 +571,17 @@ PciHostBridgeResourceAllocator (
                               &PMem64Base
                               );
 
+      //
+      // Handle pre-populated resources
+      //
+      PopulateResourceAperture (
+        Mem32Base,
+        PMem32Base,
+        Mem64Base,
+        PMem64Base,
+        RootBridgeDev
+      );
+
       //
       // Get the max ROM size that the root bridge can process
       // Insert to resource map so that there will be dedicate MEM32 resource range for Option ROM.
diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c
index 4969ee0f64..ad8db9a0ca 100644
--- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c
+++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c
@@ -325,6 +325,270 @@ CalculateApertureIo16 (
   Bridge->Length = MAX (Bridge->Length, PaddingAperture);
 }
 
+/**
+  Analyze a single BAR and if it is pre-populated (its baseAddress is already
+  set), calculate the offset inside the root bridge.
+
+  @param Bar    PCI BAR to be placed if pre-populated
+  @param Base   Base address of the root bridge
+
+**/
+STATIC
+VOID
+PrePopulateBAR (
+  IN OUT PCI_BAR    *Bar,
+  IN     UINT64     Base
+  )
+{
+  if (Base == gAllOne) {
+    return;
+  }
+
+  //
+  // Pre-populated IO space BARs are not supported
+  //
+  if (Bar->BarType == PciBarTypeIo16 || Bar->BarType == PciBarTypeIo32) {
+    return ;
+  }
+
+  if (Bar->BaseAddress < Base) {
+    return ;
+  }
+
+  if (Bar->BaseAddress != 0) {
+    Bar->BaseAddressOffset = Bar->BaseAddress - Base;
+  }
+}
+
+
+
+/**
+  Calculate and set the address offset inside the root bridge of device
+  BARs that are pre-populated.
+
+  @param Dev      PCI device whose BARs are to be analyzed
+  @param Base     Base address of the root bridge under which the device
+                  is located.
+
+ */
+STATIC
+VOID
+PopulateDeviceBars (
+  IN UINT64           Mem32Base,
+  IN UINT64           PMem32Base,
+  IN UINT64           Mem64Base,
+  IN UINT64           PMem64Base,
+  IN PCI_IO_DEVICE    *Dev
+  )
+{
+  UINTN             Index;
+
+  for (Index = 0; Index < PCI_MAX_BAR; Index++)
+  {
+    switch (Dev->PciBar[Index].BarType) {
+      case PciBarTypeMem32:
+        PrePopulateBAR(&Dev->PciBar[Index], Mem32Base);
+        break;
+
+      case PciBarTypePMem32:
+        PrePopulateBAR(&Dev->PciBar[Index], PMem32Base);
+        break;
+
+      case PciBarTypeMem64:
+        PrePopulateBAR(&Dev->PciBar[Index], Mem64Base);
+        break;
+
+      case PciBarTypePMem64:
+        PrePopulateBAR(&Dev->PciBar[Index], PMem64Base);
+        break;
+
+      default:
+        break;
+    }
+  }
+}
+
+/**
+  Considering the Bridge attributes specifying the supported resources,
+  merge the base addresses to fake the non-suported types belonging to
+  the alternative type.
+
+  @param Mem32Base      32-bit base address
+  @param PMem32Base     32-bit prefetchable base address
+  @param Mem64Base      64-bit base address
+  @param PMem64Base     64-bit prefetchable base address
+  @param Bridge         PCI resource node for given bridge device.
+
+ */
+STATIC
+VOID
+MergeBridgeResourceBases (
+  IN OUT UINT64             *Mem32Base,
+  IN OUT UINT64             *PMem32Base,
+  IN OUT UINT64             *Mem64Base,
+  IN OUT UINT64             *PMem64Base,
+  IN     PCI_IO_DEVICE      *Bridge
+  )
+{
+  //
+  // if root bridge supports combined Pmem Mem decoding
+  // merge these two type of resource
+  //
+  if (BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM_MEM_COMBINE_SUPPORTED)) {
+    *PMem32Base = *Mem32Base;
+    *PMem64Base = *Mem64Base;
+  }
+
+  //
+  // If bridge doesn't support Pmem32
+  // degrade it to mem32
+  //
+  if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM32_DECODE_SUPPORTED)) {
+    *PMem32Base = *Mem32Base;
+  }
+
+  //
+  // If firmware is in 32-bit mode,
+  // then degrade PMEM64/MEM64 requests
+  //
+  if (sizeof (UINTN) <= 4) {
+    *Mem64Base = *Mem32Base ;
+    *PMem64Base = *PMem32Base;
+  } else {
+    //
+    // if the bridge does not support MEM64, degrade MEM64 to MEM32
+    //
+    if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_MEM64_DECODE_SUPPORTED)) {
+      *Mem64Base = *Mem32Base;
+    }
+
+    //
+    // if the bridge does not support PMEM64, degrade PMEM64 to PMEM32
+    //
+    if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM64_DECODE_SUPPORTED)) {
+      *PMem64Base = *PMem32Base;
+    }
+  }
+}
+
+/**
+  Populate a root bridge's resource aperture according to
+  initial conditions ruled by individual pre-populated
+  resources
+
+   @param Bridge    PCI resource node for given bridge device.
+   @param Base      Resource aperture base address
+
+**/
+VOID
+PopulateResourceAperture (
+  IN     UINT64           Mem32Base,
+  IN     UINT64           PMem32Base,
+  IN     UINT64           Mem64Base,
+  IN     UINT64           PMem64Base,
+  IN OUT PCI_IO_DEVICE    *Bridge
+  )
+{
+  PCI_IO_DEVICE     *Temp;
+  LIST_ENTRY        *CurrentLink;
+
+  //
+  // Verify if pre-populated BARs need to be respected, otherwise
+  // there is no need to pre-populate any resource
+  //
+  if (! PcdGetBool(PcdPciPreservePopulatedMappings)) {
+    return;
+  }
+
+  CurrentLink = Bridge->ChildList.ForwardLink;
+
+  //
+  // Merge base addresses of the different types depending on resource
+  // decoding supported by the bridge
+  //
+  MergeBridgeResourceBases(
+    &Mem32Base,
+    &PMem32Base,
+    &Mem64Base,
+    &PMem64Base,
+    Bridge
+  );
+
+  while (CurrentLink != NULL && CurrentLink != &Bridge->ChildList) {
+
+    Temp = PCI_IO_DEVICE_FROM_LINK (CurrentLink);
+
+    PopulateDeviceBars(
+      Mem32Base,
+      PMem32Base,
+      Mem64Base,
+      PMem64Base,
+      Temp
+    );
+
+    //
+    // Recursive call to analyze the hierarchical tree under a bridge
+    //
+    if (IS_PCI_BRIDGE (&Temp->Pci)) {
+      PopulateResourceAperture (
+        Mem32Base,
+        PMem32Base,
+        Mem64Base,
+        PMem64Base,
+        Temp
+      );
+    }
+
+    CurrentLink = CurrentLink->ForwardLink;
+  }
+}
+
+/**
+  Calculate the current used resource of the bridge provided that some
+  of the resources under it might have been pre-populated.
+
+  @param Bridge     PCI resource node for given bridge device.
+
+**/
+STATIC
+UINT64
+GetBridgePopulatedAperture (
+  IN PCI_RESOURCE_NODE    *Bridge
+  )
+{
+  UINT64            Aperture;
+  UINT64            EndAddress;
+  LIST_ENTRY        *CurrentLink;
+  PCI_RESOURCE_NODE *Node;
+  PCI_BAR           *Bar;
+
+  Aperture = 0;
+
+  //
+  // Analyze all resource nodes looking for the prepopulated with the
+  // highest address used.
+  //
+  for ( CurrentLink = GetFirstNode (&Bridge->ChildList)
+      ; !IsNull (&Bridge->ChildList, CurrentLink)
+      ; CurrentLink = GetNextNode (&Bridge->ChildList, CurrentLink)
+      ) {
+    Node = RESOURCE_NODE_FROM_LINK (CurrentLink);
+
+    Bar = &Node->PciDev->PciBar[Node->Bar];
+
+    if (Bar->BaseAddress != 0) {
+      EndAddress = Bar->BaseAddressOffset + Bar->Length;
+
+      if (EndAddress > Aperture) {
+        Aperture = EndAddress;
+      }
+    }
+
+  }
+
+  return Aperture;
+}
+
 /**
   This function is used to calculate the resource aperture
   for a given bridge device.
@@ -338,6 +602,7 @@ CalculateResourceAperture (
   )
 {
   UINT64            Aperture[2];
+  UINT64            InitialAperture;
   LIST_ENTRY        *CurrentLink;
   PCI_RESOURCE_NODE *Node;
 
@@ -351,8 +616,14 @@ CalculateResourceAperture (
     return ;
   }
 
-  Aperture[PciResUsageTypical] = 0;
-  Aperture[PciResUsagePadding] = 0;
+  //
+  // Initialize apertures at bridge's current resource usage
+  // which might be occupied by pre-populated resources.
+  //
+  InitialAperture = GetBridgePopulatedAperture(Bridge);
+  Aperture[PciResUsageTypical] = InitialAperture;
+  Aperture[PciResUsagePadding] = InitialAperture;
+
   //
   // Assume the bridge is aligned
   //
@@ -362,6 +633,12 @@ CalculateResourceAperture (
       ) {
     Node = RESOURCE_NODE_FROM_LINK (CurrentLink);
 
+    PCI_BAR *Bar = &Node->PciDev->PciBar[Node->Bar];
+    if (PcdGetBool(PcdPciPreservePopulatedMappings) && Bar->BaseAddress) {
+      Node->Offset = Bar->BaseAddressOffset;
+      continue;
+    }
+
     //
     // It's possible for a bridge to contain multiple padding resource
     // nodes due to DegradeResource().
-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [edk2-devel] [PATCH 1/5] OvmfPkg/PlatformPei: Extend 64-bit PCI range for multiple host bridges

[Gerd Hoffmann]

> +//
> +// Root bridge resource information for parsing fw-cfg data
> +//
> +#pragma pack(1)
> +typedef struct {
> +  BOOLEAN set_pxm;
> +  BOOLEAN mark_prefetchable_resources_as_cacheable;
> +
> +  UINT8 root_bus_nr;
> +  UINT8 pxm;
> +  UINT8 num_hot_plug_slots;
> +
> +  UINT8 root_bur_nr_start;
> +  UINT8 root_bus_nr_end;
> +
> +  UINT32 lowmem_start;
> +  UINT32 lowmem_end;
> +  UINT32 lowmem_pref_start;
> +  UINT32 lowmem_pref_end;
> +
> +  UINT64 highmem_start;
> +  UINT64 highmem_end;
> +  UINT64 highmem_pref_start;
> +  UINT64 highmem_pref_end;
> +} HOST_BRIDGE_INFO;
> +#pragma pack()

Hmm.  A number of fields is not used by this patch series, pxm for
example.  On the other hand it is incomplete, ioports are not there.
It looks like you are simply throwing some internal data structure
into fw_cfg.  This is *not* how things should be done.

Is the host-side code for this open source btw?  If so, do you
have a pointer?

I see two reasonable options going forward:

  (1) Create a format which contains *everything* needed to describe
      a pci(e) host bridge, not only those parts which you happen to
      need for your use case.  Basically all information needed by
      PciHostBridgeUtilityInitRootBridge(), and the pcie config space
      location.  Drop fields not needed.  And add comments documenting
      the struct.
  (2) Use an existing format.  Obvious choice coming to mind is
      device-tree.  We already have a parser for that because the
      arm platform (and riscv too soon) needs it to find the pcie
      host bridge.

take care,
  Gerd