From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-fw-9103.amazon.com (smtp-fw-9103.amazon.com [207.171.188.200]) by mx.groups.io with SMTP id smtpd.web11.208.1643922104410309147 for ; Thu, 03 Feb 2022 13:01:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=IBzoCnjV; spf=pass (domain: amazon.de, ip: 207.171.188.200, mailfrom: prvs=026b83387=ncoleon@amazon.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1643922104; x=1675458104; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=8V55TTdSDxT4kod6mSnzK3bpRml31spGiXcG+EmZ0Ic=; b=IBzoCnjVvYzcLnbBR36U3AfHwkvECF7MWVA+FDYllSaGcMZLAgTIcPnB QEUH3yOzWmbzOU6X3sYIUKbGrJlt+UbxvHBHBSHd/642l2nrnYW6/T4eZ tOpYxFMLm13FGhKOlsZ/C9CitJQTJ9G3ZppSQtSo3XxxI2DOlYQhs6b2M E=; X-IronPort-AV: E=Sophos;i="5.88,340,1635206400"; d="scan'208";a="989485476" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-iad-1a-828bd003.us-east-1.amazon.com) ([10.25.36.210]) by smtp-border-fw-9103.sea19.amazon.com with ESMTP; 03 Feb 2022 21:01:43 +0000 Received: from EX13D49EUC003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1a-828bd003.us-east-1.amazon.com (Postfix) with ESMTPS id F33B381618; Thu, 3 Feb 2022 21:01:42 +0000 (UTC) Received: from ub4014a598e6c52.ant.amazon.com (10.43.160.114) by EX13D49EUC003.ant.amazon.com (10.43.164.91) with Microsoft SMTP Server (TLS) id 15.0.1497.28; Thu, 3 Feb 2022 21:01:39 +0000 From: "Ojeda Leon, Nicolas" To: CC: , Nicolas Ojeda Leon , Alexander Graf , Gerd Hoffmann Subject: [PATCH v4 8/8] MdeModulePkg/PciBusDxe: Handling of pre-populated PCI BARs Date: Thu, 3 Feb 2022 21:56:28 +0100 Message-ID: <776ac740fddedd1f58f7ebbad6fba8837ef06600.1643919692.git.ncoleon@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.43.160.114] X-ClientProxiedBy: EX13D27UWB004.ant.amazon.com (10.43.161.101) To EX13D49EUC003.ant.amazon.com (10.43.164.91) Content-Type: text/plain Extend the PCI BAR placement logic in order to consider pre-populated resources first, if indicated by the Pcd token for such purpose. The PCI_BAR type is augmented by one field for mapping the absolute address of prepopulated BARs into a root bridge relative offset. As part of the CreateResourceMap stage of the PCI resource allocation process, all the resources belonging to a root bridge are analyzed and if a BaseAddress (absolute address read from the PCI device during enumeration) is present, it is translated into a root-bridge relative offset and the space occupied by it is marked as used in the root bridge's aperture. This process is performed before the regular placement logic is executed. The remaining unassigned BARs are then placed according to default logic starting after the last pre-populated resource. The motivation behind respecting pre-populated BARs is that the hypervisor can decide on placement of some resources and the guest must use the same values to benefit from performance optimizations or specific requirements. Cc: Alexander Graf Cc: Gerd Hoffmann Signed-off-by: Nicolas Ojeda Leon --- Notes: v4: - Guarded the root bridge address retreiving and the handling of pre-populated resources in PciLib with PcdPciPreservePopulatedMappings token - Use the new way to fetch Root Bridge base addresses by calling GetProposedResources before allocation to then populate the resource apertures to take care of pre-populated BARs. MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h | 1 + .../Bus/Pci/PciBusDxe/PciEnumeratorSupport.c | 5 +- MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c | 49 ++++ .../Bus/Pci/PciBusDxe/PciResourceSupport.c | 277 +++++++++++++++++- .../Bus/Pci/PciBusDxe/PciResourceSupport.h | 20 ++ 5 files changed, 348 insertions(+), 4 deletions(-) diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h index ecb3d3cb5d..1200924e95 100644 --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h @@ -95,6 +95,7 @@ typedef enum { // struct _PCI_BAR { UINT64 BaseAddress; + UINT64 BaseAddressOffset; UINT64 Length; UINT64 Alignment; PCI_BAR_TYPE BarType; diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c index 9251388bc2..91caf162e0 100644 --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c @@ -1776,8 +1776,9 @@ PciParseBar ( return Offset + 4; } - PciIoDevice->PciBar[BarIndex].BarTypeFixed = FALSE; - PciIoDevice->PciBar[BarIndex].Offset = (UINT8)Offset; + PciIoDevice->PciBar[BarIndex].BarTypeFixed = FALSE; + PciIoDevice->PciBar[BarIndex].Offset = (UINT8)Offset; + PciIoDevice->PciBar[BarIndex].BaseAddressOffset = 0; if ((Value & 0x01) != 0) { // // Device I/Os diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c index 63d149b3b8..be6bbd01c7 100644 --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c @@ -595,6 +595,55 @@ PciHostBridgeResourceAllocator ( GetResourceFromDevice (RootBridgeDev, IoBridge, Mem32Bridge, PMem32Bridge, Mem64Bridge, PMem64Bridge); } + if (PcdGetBool (PcdPciPreservePopulatedMappings)) { + // + // If there are pre-populated resources (e.g. in virtualized platforms) + // the base addresses need to be retrieved to place them before the + // other resources. This also implies the firmware received the PCI + // apertures fixed and that these are consistent with the pre-populated + // values. At this point, the resources have status ResBaseDiscoverd and + // have only the Base address set (partially filled and not allocated) + // + AcpiConfig = NULL; + Status = PciResAlloc->GetProposedResources ( + PciResAlloc, + RootBridgeDev->Handle, + &AcpiConfig + ); + + if (EFI_ERROR (Status)) { + return Status; + } + + // + // Get the resource base by interpreting acpi resource node + // + // + GetResourceBase ( + AcpiConfig, + &IoBase, + &Mem32Base, + &PMem32Base, + &Mem64Base, + &PMem64Base + ); + + if (AcpiConfig != NULL) { + FreePool (AcpiConfig); + } + + // + // Handle pre-populated resources + // + PopulateResourceAperture ( + Mem32Base, + PMem32Base, + Mem64Base, + PMem64Base, + RootBridgeDev + ); + } + // // Create resourcemap by going through all the devices subject to this root bridge // diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c index 8ffd05f327..e2a550bf8e 100644 --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.c @@ -324,6 +324,266 @@ CalculateApertureIo16 ( Bridge->Length = MAX (Bridge->Length, PaddingAperture); } +/** + Analyze a single BAR and if it is pre-populated (its baseAddress is already + set), calculate the offset inside the root bridge. + + @param Bar PCI BAR to be placed if pre-populated + @param Base Base address of the root bridge + +**/ +STATIC +VOID +PrePopulateBAR ( + IN OUT PCI_BAR *Bar, + IN UINT64 Base + ) +{ + if (Base == gAllOne) { + return; + } + + // + // Pre-populated IO space BARs are not supported + // + if ((Bar->BarType == PciBarTypeIo16) || (Bar->BarType == PciBarTypeIo32)) { + return; + } + + if (Bar->BaseAddress < Base) { + return; + } + + if (Bar->BaseAddress != 0) { + Bar->BaseAddressOffset = Bar->BaseAddress - Base; + } +} + +/** + Calculate and set the address offset inside the root bridge of device + BARs that are pre-populated. + + @param Dev PCI device whose BARs are to be analyzed + @param Base Base address of the root bridge under which the device + is located. + + */ +STATIC +VOID +PopulateDeviceBars ( + IN UINT64 Mem32Base, + IN UINT64 PMem32Base, + IN UINT64 Mem64Base, + IN UINT64 PMem64Base, + IN PCI_IO_DEVICE *Dev + ) +{ + UINTN Index; + + for (Index = 0; Index < PCI_MAX_BAR; Index++) { + switch (Dev->PciBar[Index].BarType) { + case PciBarTypeMem32: + PrePopulateBAR (&Dev->PciBar[Index], Mem32Base); + break; + + case PciBarTypePMem32: + PrePopulateBAR (&Dev->PciBar[Index], PMem32Base); + break; + + case PciBarTypeMem64: + PrePopulateBAR (&Dev->PciBar[Index], Mem64Base); + break; + + case PciBarTypePMem64: + PrePopulateBAR (&Dev->PciBar[Index], PMem64Base); + break; + + default: + break; + } + } +} + +/** + Considering the Bridge attributes specifying the supported resources, + merge the base addresses to fake the non-suported types belonging to + the alternative type. + + @param Mem32Base 32-bit base address + @param PMem32Base 32-bit prefetchable base address + @param Mem64Base 64-bit base address + @param PMem64Base 64-bit prefetchable base address + @param Bridge PCI resource node for given bridge device. + + */ +STATIC +VOID +MergeBridgeResourceBases ( + IN OUT UINT64 *Mem32Base, + IN OUT UINT64 *PMem32Base, + IN OUT UINT64 *Mem64Base, + IN OUT UINT64 *PMem64Base, + IN PCI_IO_DEVICE *Bridge + ) +{ + // + // if root bridge supports combined Pmem Mem decoding + // merge these two type of resource + // + if (BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM_MEM_COMBINE_SUPPORTED)) { + *PMem32Base = *Mem32Base; + *PMem64Base = *Mem64Base; + } + + // + // If bridge doesn't support Pmem32 + // degrade it to mem32 + // + if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM32_DECODE_SUPPORTED)) { + *PMem32Base = *Mem32Base; + } + + // + // If firmware is in 32-bit mode, + // then degrade PMEM64/MEM64 requests + // + if (sizeof (UINTN) <= 4) { + *Mem64Base = *Mem32Base; + *PMem64Base = *PMem32Base; + } else { + // + // if the bridge does not support MEM64, degrade MEM64 to MEM32 + // + if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_MEM64_DECODE_SUPPORTED)) { + *Mem64Base = *Mem32Base; + } + + // + // if the bridge does not support PMEM64, degrade PMEM64 to PMEM32 + // + if (!BridgeSupportResourceDecode (Bridge, EFI_BRIDGE_PMEM64_DECODE_SUPPORTED)) { + *PMem64Base = *PMem32Base; + } + } +} + +/** + Populate a root bridge's resource aperture according to + initial conditions ruled by individual pre-populated + resources + + @param Bridge PCI resource node for given bridge device. + @param Base Resource aperture base address + +**/ +VOID +PopulateResourceAperture ( + IN UINT64 Mem32Base, + IN UINT64 PMem32Base, + IN UINT64 Mem64Base, + IN UINT64 PMem64Base, + IN OUT PCI_IO_DEVICE *Bridge + ) +{ + PCI_IO_DEVICE *Temp; + LIST_ENTRY *CurrentLink; + + // + // Verify if pre-populated BARs need to be respected, otherwise + // there is no need to pre-populate any resource + // + if (!PcdGetBool (PcdPciPreservePopulatedMappings)) { + return; + } + + CurrentLink = Bridge->ChildList.ForwardLink; + + // + // Merge base addresses of the different types depending on resource + // decoding supported by the bridge + // + MergeBridgeResourceBases ( + &Mem32Base, + &PMem32Base, + &Mem64Base, + &PMem64Base, + Bridge + ); + + while (CurrentLink != NULL && CurrentLink != &Bridge->ChildList) { + Temp = PCI_IO_DEVICE_FROM_LINK (CurrentLink); + + PopulateDeviceBars ( + Mem32Base, + PMem32Base, + Mem64Base, + PMem64Base, + Temp + ); + + // + // Recursive call to analyze the hierarchical tree under a bridge + // + if (IS_PCI_BRIDGE (&Temp->Pci)) { + PopulateResourceAperture ( + Mem32Base, + PMem32Base, + Mem64Base, + PMem64Base, + Temp + ); + } + + CurrentLink = CurrentLink->ForwardLink; + } +} + +/** + Calculate the current used resource of the bridge provided that some + of the resources under it might been pre-populated. + + @param Bridge PCI resource node for given bridge device. + +**/ +STATIC +UINT64 +GetBridgePopulatedAperture ( + IN PCI_RESOURCE_NODE *Bridge + ) +{ + UINT64 Aperture; + UINT64 EndAddress; + LIST_ENTRY *CurrentLink; + PCI_RESOURCE_NODE *Node; + PCI_BAR *Bar; + + Aperture = 0; + + // + // Analyze all resource nodes looking for the prepopulated with the + // highest address used. + // + for ( CurrentLink = GetFirstNode (&Bridge->ChildList) + ; !IsNull (&Bridge->ChildList, CurrentLink) + ; CurrentLink = GetNextNode (&Bridge->ChildList, CurrentLink) + ) + { + Node = RESOURCE_NODE_FROM_LINK (CurrentLink); + + Bar = &Node->PciDev->PciBar[Node->Bar]; + + if (Bar->BaseAddress != 0) { + EndAddress = Bar->BaseAddressOffset + Bar->Length; + + if (EndAddress > Aperture) { + Aperture = EndAddress; + } + } + } + + return Aperture; +} + /** This function is used to calculate the resource aperture for a given bridge device. @@ -337,6 +597,7 @@ CalculateResourceAperture ( ) { UINT64 Aperture[2]; + UINT64 InitialAperture; LIST_ENTRY *CurrentLink; PCI_RESOURCE_NODE *Node; @@ -349,8 +610,14 @@ CalculateResourceAperture ( return; } - Aperture[PciResUsageTypical] = 0; - Aperture[PciResUsagePadding] = 0; + // + // Initialize apertures at bridge's current resource usage + // which might be occupied by pre-populated resources. + // + InitialAperture = GetBridgePopulatedAperture (Bridge); + Aperture[PciResUsageTypical] = InitialAperture; + Aperture[PciResUsagePadding] = InitialAperture; + // // Assume the bridge is aligned // @@ -361,6 +628,12 @@ CalculateResourceAperture ( { Node = RESOURCE_NODE_FROM_LINK (CurrentLink); + PCI_BAR *Bar = &Node->PciDev->PciBar[Node->Bar]; + if (PcdGetBool (PcdPciPreservePopulatedMappings) && Bar->BaseAddress) { + Node->Offset = Bar->BaseAddressOffset; + continue; + } + // // It's possible for a bridge to contain multiple padding resource // nodes due to DegradeResource(). diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h index 1527d4eafa..c5ab5f3fd9 100644 --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciResourceSupport.h @@ -111,6 +111,26 @@ CalculateApertureIo16 ( IN PCI_RESOURCE_NODE *Bridge ); +/** + Populate a root bridge's resource aperture according to + initial conditions ruled by individual pre-populated + resources + + @note Pre-populated PIO BARs are not supported. + + @param Bridge PCI resource node for given bridge device. + @param Base Resource aperture base address + +**/ +VOID +PopulateResourceAperture ( + IN UINT64 Mem32Base, + IN UINT64 PMem32Base, + IN UINT64 Mem64Base, + IN UINT64 PMem64Base, + IN OUT PCI_IO_DEVICE *Bridge + ); + /** This function is used to calculate the resource aperture for a given bridge device. -- 2.17.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879