From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.81]) by mx.groups.io with SMTP id smtpd.web11.24775.1594152005411495541 for ; Tue, 07 Jul 2020 13:00:05 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EpS1YH1b; spf=pass (domain: redhat.com, ip: 207.211.31.81, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594152004; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bqz9u1qnp/5Hm7FMfX+JOfYm4argR7JITHiCVU03PnY=; b=EpS1YH1bchug4Hs4Nf+YGf2JQijrthfuPZwxOj7AGsvlA46zDz9+pJIK66ZyGnwH1hmfGR bCB9X5ZMbzXO/+EKhZU9mtUjf5pv6blvzFvLr50S7X88p/1pYvVLJ03SOV3CHm/h1EQyiq UmKd87XZzXMhzvh0nKJ9heo+Yi0w2bU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-49-T9qQsS76NvOdQxfM-ZBnkw-1; Tue, 07 Jul 2020 15:59:58 -0400 X-MC-Unique: T9qQsS76NvOdQxfM-ZBnkw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C82E0107ACCA; Tue, 7 Jul 2020 19:59:57 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-114-90.ams2.redhat.com [10.36.114.90]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0B311C0064; Tue, 7 Jul 2020 19:59:56 +0000 (UTC) Subject: Re: [edk2-devel] 2nd OVMF question about why InitRootBridge does not set ResourceAssigned? To: devel@edk2.groups.io, afish@apple.com References: <19FB2158-0F1A-4EDF-86FC-04095AE1066E@apple.com> From: "Laszlo Ersek" Message-ID: Date: Tue, 7 Jul 2020 21:59:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <19FB2158-0F1A-4EDF-86FC-04095AE1066E@apple.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Hi Andrew, On 07/03/20 19:22, Andrew Fish via groups.io wrote: > When tacking down the EFI_MEMORY_UC issue I noticed that InitRootBridg() [1] does not set RootBus-> ResourceAssigned = TRUE. If it was TRUE then the entire PCI aptitude would end up in the GCD map. Given it is not set (set by ZeroMem) the EFI_MEMORY_UC only ends up for the actual allocations as far as I can tell. I’m wondering why it was done this way? > > I did notice if I set RootBus-> ResourceAssigned to TRUE the serial console did not come up and I did not get a chance to debug that? Maybe there was a resource conflict with the ISA bus driver or some such? > > [1] https://github.com/tianocore/edk2/blob/master/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c#L112 The ResourceAssigned field was introduced in commit c5be19f37821 ("MdeModulePkg/PciHostBridgeLib: Add ResourceAssigned field", 2016-05-11). See also the subsequent commit in the history, commit 401f8cd110f7 ("MdeModulePkg/PciHostBridgeDxe: Honor ResourceAssigned", 2016-05-11). The idea is that setting "ResourceAssigned = TRUE" tells PciHostBridgeDxe that the ranges exposed in the PCI_ROOT_BRIDGE structure are not apertures for PciBusDxe to allocate resources from, but ranges *already* programmed -- by a previously running agent -- into the root bridge. In the ArmVirtQemu platform, and when OVMF runs on QEMU, we need to set ResourceAssigned to FALSE, as we really want PciBusDxe: - to carve out resources for PCI devices from the bridge apertures, - and to program the bridge ranges ultimately. When OVMF runs on Xen however, the PCI enumeration / resource assignment is performed by a component that runs earlier (IIRC it is performed by an agent that is somehow part of the hypervisor), and so the edk2 PCI infrastructure is supposed to perform a "light" enumeration only. That's why: - we set ResourceAssigned to TRUE in "OvmfPkg/Library/PciHostBridgeLib/XenSupport.c", - and also why we set "PcdPciDisableBusEnumeration" to TRUE in "PlatformPei/Xen.c". The "light enumeration" case -- concerning both "PcdPciDisableBusEnumeration" and "ResourceAssigned" -- applies to UefiPayloadPkg as well. There -- I think -- the pre-programming is done by Coreboot. The above means that, regardless of "ResourceAssigned", only those MMIO areas get marked as UC (or even get added to the GCP memory space map) that are allocated to specific PCI devices: - With ResourceAssigned=TRUE, the ranges are parsed out of the device registers. - With ResourceAssigned=FALSE, the ranges are assigned by PciBusDxe, and programmed into the registers. But in both cases, the GCD memory space map reflects the ranges that the devices actually decode. Setting ResourceAssigned=TRUE in OVMF (running on QEMU) would completely break PCI enumeration, as we expose large apertures, and really want PciBusDxe to grab actual BARs out of those. On physical machines, I assume root bridges come with pre-determined, distinct physical address ranges that they can decode, for MMIO accesses. So I guess in those cases, it could be useful to add the apertures in full (before the actual resource assignments) to GCD. On QEMU however, it's the other way around. First, the apertures are not distinct, instead all root bridges share the same aperture (per type). Second, the aperture is not pre-determined -- QEMU is fine with any MMIO BAR assigments, as long as the BARs do not overlap some *otherwise assigned* guest-physical address range (such as RAM, or platform MMIO). Therefore, the root bridge apertures exposed by OVMF's PciHostBridgeLib are synthetic. (For example, a recurrent topic on these lists is how OVMF's 64-bit MMIO aperture should be sized -- right now it defaults to 32GiB, but you can resize it on the QEMU command line, and some users would prefer smarter defaults. See for example the TianoCore#2796 ticket. But I digress.) In ArmVirtQemu, and in OVMF running on QEMU, once we expose the synthetic aperture(s), and PciBusDxe does carve out / assign actual resources, then QEMU *deduces* the root bridge apertures from the actual device programming performed by OVMF. Then QEMU generates the ACPI tables (the _CRS object) accordingly. And OVMF fetches the ACPI payload from QEMU, and installs it for the OS to consume. This is explained in more detail in commit 9116c9c5d885 ("OvmfPkg: introduce gRootBridgesConnectedEventGroupGuid", 2016-03-23). In short, in ArmVirtQemu and in OVMF running on QEMU, the resources assigned by PciBusDxe determine, ultimately, what QEMU will place in the _CRS, and what the OS will see in the _CRS. (Side note: where I say "synthetic", that does not mean "easy to change" at all. Coming up with good 32-bit and 64-bit MMIO apertures for the PciHostBridgeLib instance was an arduous task. Among other things, this originates from the very quirky placement of the 32-bit address ranges on i440fx and q35 that are usable for PCI MMIO. QEMU doesn't dictate the apertures for the root bridges, but if you allocate an MMIO BAR from an inappropriate address range, you're doomed.) So... I'm trying to finish with coherent answers to your questions... - ResourceAssigned is set to FALSE because we want PciBusDxe to perform the resource assignment / device programming. On QEMU, the BARs are not pre-programmed, the firmware has to do it. And, the _CRS in ACPI is calculated by QEMU as a *function* of PciBusDxe's actions. - Setting ResourceAssigned to TRUE would break everything on QEMU. That's because the address ranges we expose in PCI_ROOT_BRIDGE objects correspond to the "bounding boxes" in guest-physical address space where QEMU can accommodate MMIO BARs, on all the PCI root bridges taken together. No resource assignments occur apart from PciBusDxe, the fields in PCI_ROOT_BRIDGE do not stand for ranges previously programmed into devices (bridges or endpoints). Thanks Laszlo