From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-fw-80006.amazon.com (smtp-fw-80006.amazon.com [99.78.197.217]) by mx.groups.io with SMTP id smtpd.web12.11865.1630766317242906002 for ; Sat, 04 Sep 2021 07:38:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=TJMQxz85; spf=pass (domain: amazon.de, ip: 99.78.197.217, mailfrom: prvs=874eef02e=ncoleon@amazon.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1630766317; x=1662302317; h=from:to:cc:subject:date:message-id:mime-version; bh=Ha6cD3CgPiwyuwWcSG8NRmhcStH9/qlHt+xv1c+36T8=; b=TJMQxz85R3R7HAi/QAAFNPCcK0U8i9T7x3cwReBgZjHrL05LYEaj7j5T 5O/PNrkT2WNYxpfSK6u/ysz2JQg8OL420PADe+kCrs3JjhAuehNELe/ou ooVqWlkY/Dm5KRjjm8xGK+syAPuFQ58b/hzBALIgxBkHoroB57RtawIgN c=; X-IronPort-AV: E=Sophos;i="5.85,268,1624320000"; d="scan'208";a="24534373" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-1e-c7c08562.us-east-1.amazon.com) ([10.25.36.214]) by smtp-border-fw-80006.pdx80.corp.amazon.com with ESMTP; 04 Sep 2021 14:38:30 +0000 Received: from EX13D49EUC003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1e-c7c08562.us-east-1.amazon.com (Postfix) with ESMTPS id 306B8247A7A for ; Sat, 4 Sep 2021 14:38:28 +0000 (UTC) Received: from ub4014a598e6c52.ant.amazon.com (10.43.162.52) by EX13D49EUC003.ant.amazon.com (10.43.164.91) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Sat, 4 Sep 2021 14:38:26 +0000 From: Nicolas Ojeda Leon To: CC: Nicolas Ojeda Leon Subject: [PATCH 0/5] Handling of multiple PCI host bridges with pre-populated resources in Ovmf. Date: Sat, 4 Sep 2021 16:37:51 +0200 Message-ID: X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-Originating-IP: [10.43.162.52] X-ClientProxiedBy: EX13D16UWC001.ant.amazon.com (10.43.162.117) To EX13D49EUC003.ant.amazon.com (10.43.164.91) Content-Type: text/plain Increased control is provided in Ovmf platforms to define and configure the specifications of multiple PCI host bridges in the hypervisor. The host propagates this information to the guest through fw-cfg interface. In some AWS EC2 platforms, we expose a PCI topology including several root bridges portraying information about physical distribution that enables the guest to optimize accesses. Current PCI driver for Ovmf platforms supports a single host bridges and EDK2 contains the logic to determine its MMIO resources. However, we need a way to define more than 1 host bridge and control, from the hypervisor, how many and which resources each of them can use. For this reason, this patch series introduces a mechanism to provide bus number, pxm number, bus number ranges as well as 32 and 64- bit prefetchable and non--prefetchable MMIO ranges through a fw-cfg item created by the hypervisor and consumed by the guest firmware. Furthermore, in these kind of high performance platforms, we exploit PCIe features like Access Control Services to configure peer-to-peer channels between devices. This allows us to create direct communication channels that do not require packets to reach the Root Complex but instead can follow a direct path from source to target. To enable Guest Virtual Machines to profit from this performance improvement, we configure resources (BARs) of peer-to-peer intended devices with Host Physical Addresses. In this scenario, devices can be instructed, from the guest VM, to perform DMA operations targeting a peer address space, and the PCIe fabric can take care of directly routing them. Therefore long and busy links towards the Root Complex are avoided. When we configure resources this way, the guest must respect the pre-populated BARs so that devices preserve the address ranges configured in the apertures of physical PCIe ports that enable routing at the hardware level. Nicolas Ojeda Leon (5): OvmfPkg/PlatformPei: Extend 64-bit PCI range for multiple host bridges OvmfPkg/PciHostBridgeUtilityLib: Initialize RootBridges apertures with spec MdeModulePkg OvmfPkg: Add Pcd token for PCI pre-populated BARs MdeModulePkg/Pci MdePkg: Create service to retrieve PCI base addresses MdeModulePkg/PciBusDxe: Handling of pre-populated PCI BARs MdeModulePkg/MdeModulePkg.dec | 6 + OvmfPkg/OvmfPkgX64.dsc | 1 + MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf | 1 + .../PciHostBridgeUtilityLib.inf | 4 + OvmfPkg/PlatformPei/PlatformPei.inf | 1 + MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h | 1 + .../Bus/Pci/PciBusDxe/PciResourceSupport.h | 20 ++ .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.h | 29 ++ .../PciHostBridgeResourceAllocation.h | 33 ++ .../Include/Library/PciHostBridgeInfoLib.h | 43 +++ .../Bus/Pci/PciBusDxe/PciEnumeratorSupport.c | 1 + MdeModulePkg/Bus/Pci/PciBusDxe/PciLib.c | 21 ++ .../Bus/Pci/PciBusDxe/PciResourceSupport.c | 281 +++++++++++++++++- .../Bus/Pci/PciHostBridgeDxe/PciHostBridge.c | 80 +++++ .../PciHostBridgeUtilityLib.c | 145 ++++++++- OvmfPkg/PlatformPei/MemDetect.c | 83 ++++++ 16 files changed, 746 insertions(+), 4 deletions(-) create mode 100644 OvmfPkg/Include/Library/PciHostBridgeInfoLib.h -- 2.17.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879