From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-fw-2101.amazon.com (smtp-fw-2101.amazon.com [72.21.196.25]) by mx.groups.io with SMTP id smtpd.web09.7675.1643121258074346533 for ; Tue, 25 Jan 2022 06:34:18 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=e12R2KnV; spf=pass (domain: amazon.de, ip: 72.21.196.25, mailfrom: prvs=0179bbfd8=ncoleon@amazon.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1643121258; x=1674657258; h=from:to:cc:subject:date:message-id:mime-version; bh=9hjBr85EePGAVKMsMzjTUoh7aM0sF9J2ldJ3no0PXxE=; b=e12R2KnViYBWi9fV3zgtyPFCTzftjGTx1q83gmS2M/v4FUsy6dzCjR0B fCKwSO5VsCUKsJFDifYPEbEonlOXTS4iWQPe0wtSi/vvo/Ph0Fah/7FyM vjse8RpTC8wI5nP09OGgdliOnBkvxQaqK1DIsDDpq0x7eK9Iti0KqxUtS w=; X-IronPort-AV: E=Sophos;i="5.88,315,1635206400"; d="scan'208";a="168388099" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1a-e823fbde.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-2101.iad2.amazon.com with ESMTP; 25 Jan 2022 14:34:06 +0000 Received: from EX13D49EUC003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-iad-1a-e823fbde.us-east-1.amazon.com (Postfix) with ESMTPS id 44DBCC092F; Tue, 25 Jan 2022 14:34:05 +0000 (UTC) Received: from ub4014a598e6c52.ant.amazon.com (10.43.160.207) by EX13D49EUC003.ant.amazon.com (10.43.164.91) with Microsoft SMTP Server (TLS) id 15.0.1497.28; Tue, 25 Jan 2022 14:34:02 +0000 From: "Ojeda Leon, Nicolas" To: CC: , Nicolas Ojeda Leon , Alexander Graf , Gerd Hoffmann Subject: [PATCH v3 0/8] Handling of multiple PCI host bridges specified Date: Tue, 25 Jan 2022 15:30:07 +0100 Message-ID: X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-Originating-IP: [10.43.160.207] X-ClientProxiedBy: EX13D46UWB004.ant.amazon.com (10.43.161.204) To EX13D49EUC003.ant.amazon.com (10.43.164.91) Content-Type: text/plain Increased control is provided in Ovmf platforms to define and configure the specifications of multiple PCI host bridges in the hypervisor. The host propagates this information to the guest, initially through fw-cfg interface. In some AWS EC2 platforms, we expose a PCI topology including several root bridges portraying information about physical distribution that enables the guest to optimize accesses. Current PCI driver for Ovmf enables the explicit definition of multiple root bridges and contains the logic to fix their resources based on a platform-specific PCD entries. However, we need a way to control, from the hypervisor, how many and which resources each PCI host bridge can use. For this reason, this patch series introduces a mechanism to provide PCI host bridges information like bus number range, attributes, allocation attributes, PIO aperture as well as 32 and 64- bit prefetchable and non-prefetchable MMIO ranges through a fw-cfg item created by the hypervisor and consumed by the guest firmware. In order to offer a generic and extensible way to disclose non-discoverable hardware information from the host to the guest, a new library called HardwareInfoLib is created in the OvmfPkg. In essence, this library offers the functionality to parse a generic BLOB into a list as well as the methods to iterate over such list, including filtering options. The library is conceived in a generic way so that further hardware elements can also be described using it. For such purpose the length of the BLOB is not restricted but instead regarded as a sequence of header-info elements that allow the parsing during runtime. Furthermore, specific functionality is provided wrapping QemuFwCfgReadBytes to extract hardware descriptions, in the aforementioned format, in a static way so that early in the Pei stage the library can be used to identify address space requirements. The core of the library offers enough flexibility to process as many elements, even from different hardware types (heterogenous), as needed in a single run. This library is extended for the particular use case already exposed, PCI host bridges, and this same code offers an example of how to tailor it for further hardware components. Furthermore, in this kind of high-performance platforms, we exploit PCIe features like Access Control Services to configure peer-to-peer channels between devices. This allows us to create direct communication channels that do not require packets to reach the Root Complex but instead can follow a direct path from source to target. To enable Guest Virtual Machines to profit from this performance improvement, we configure resources (BARs) of peer-to-peer intended devices with Host Physical Addresses. In this scenario, devices can be instructed, from the guest VM, to perform DMA operations targeting a peer address space, and the PCIe fabric can take care of directly routing them. Therefore, long and busy links towards the Root Complex are avoided. When we configure resources this way, the guest must respect the pre-populated BARs so that devices preserve the address ranges configured in the apertures of physical PCIe ports that enable routing at the hardware level. Similarly, revealing details about the underlying PCI hierarchy empowers the guest to perform topology-aware optimizations and benefit from an enhanced performance. Nicolas Ojeda Leon (8): OvmfPkg/Library: Create base HardwareInfoLib for PCI Host Bridges Ovmf/HardwareInfoLib: Create Pei lib to parse directly from fw-cfg Ovmf/HardwareInfoLib: Add Dxe lib to dynamically parse heterogenous data Ovmf/PlatformPei: Use host-provided GPA end if available OvmfPkg/PciHostBridgeUtilityLib: Initialize RootBridges apertures with spec MdeModulePkg, OvmfPkg: Add Pcd token for PCI pre-populated BARs MdeModulePkg/Pci MdePkg: Create service to retrieve PCI base addresses MdeModulePkg/PciBusDxe: Handling of pre-populated PCI BARs MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h | 1 + MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf | 1 + .../Bus/Pci/PciBusDxe/PciEnumeratorSupport.c | 5 +- MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c | 21 + .../Bus/Pci/PciBusDxe/PciResourceSupport.c | 277 +++++++++- .../Bus/Pci/PciBusDxe/PciResourceSupport.h | 20 + .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c | 81 +++ .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.h | 29 + MdeModulePkg/MdeModulePkg.dec | 6 + .../PciHostBridgeResourceAllocation.h | 33 ++ OvmfPkg/Include/Library/HardwareInfoLib.h | 165 ++++++ .../HardwareInfoLib/DxeHardwareInfoLib.inf | 43 ++ .../Library/HardwareInfoLib/HardwareInfoDxe.c | 255 +++++++++ .../HardwareInfoPciHostBridgeLib.c | 515 ++++++++++++++++++ .../HardwareInfoPciHostBridgeLib.h | 262 +++++++++ .../Library/HardwareInfoLib/HardwareInfoPei.c | 85 +++ .../HardwareInfoLib/HardwareInfoTypesLib.h | 64 +++ .../HardwareInfoLib/PeiHardwareInfoLib.inf | 43 ++ .../QemuFwCfgHardwareInfoLib.c | 89 +++ .../PciHostBridgeUtilityLib.c | 362 +++++++++--- .../PciHostBridgeUtilityLib.inf | 5 + OvmfPkg/OvmfPkgX64.dsc | 3 + OvmfPkg/PlatformPei/MemDetect.c | 146 ++++- OvmfPkg/PlatformPei/PlatformPei.inf | 2 + 24 files changed, 2425 insertions(+), 88 deletions(-) create mode 100644 OvmfPkg/Include/Library/HardwareInfoLib.h create mode 100644 OvmfPkg/Library/HardwareInfoLib/DxeHardwareInfoLib.inf create mode 100644 OvmfPkg/Library/HardwareInfoLib/HardwareInfoDxe.c create mode 100644 OvmfPkg/Library/HardwareInfoLib/HardwareInfoPciHostBridgeLib.c create mode 100644 OvmfPkg/Library/HardwareInfoLib/HardwareInfoPciHostBridgeLib.h create mode 100644 OvmfPkg/Library/HardwareInfoLib/HardwareInfoPei.c create mode 100644 OvmfPkg/Library/HardwareInfoLib/HardwareInfoTypesLib.h create mode 100644 OvmfPkg/Library/HardwareInfoLib/PeiHardwareInfoLib.inf create mode 100644 OvmfPkg/Library/HardwareInfoLib/QemuFwCfgHardwareInfoLib.c -- 2.17.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879