2nd OVMF question about why InitRootBridge does not set ResourceAssigned?

2nd OVMF question about why InitRootBridge does not set ResourceAssigned?

[Andrew Fish]

When tacking down the EFI_MEMORY_UC issue I noticed that InitRootBridg() [1] does not set RootBus-> ResourceAssigned = TRUE. If it was TRUE then the entire PCI aptitude would end up in the GCD map. Given it is not set (set by ZeroMem) the EFI_MEMORY_UC only ends up for the actual allocations as far as I can tell. I’m wondering why it was done this way?

I did notice if I set RootBus-> ResourceAssigned to TRUE the serial console did not come up and I did not get a chance to debug that? Maybe there was a resource conflict with the ISA bus driver or some such? 

[1] https://github.com/tianocore/edk2/blob/master/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c#L112

Thanks,

Andrew Fish

Re: [edk2-devel] 2nd OVMF question about why InitRootBridge does not set ResourceAssigned?

[Laszlo Ersek]

Hi Andrew,

On 07/03/20 19:22, Andrew Fish via groups.io wrote:
> When tacking down the EFI_MEMORY_UC issue I noticed that InitRootBridg() [1] does not set RootBus-> ResourceAssigned = TRUE. If it was TRUE then the entire PCI aptitude would end up in the GCD map. Given it is not set (set by ZeroMem) the EFI_MEMORY_UC only ends up for the actual allocations as far as I can tell. I’m wondering why it was done this way?
> 
> I did notice if I set RootBus-> ResourceAssigned to TRUE the serial console did not come up and I did not get a chance to debug that? Maybe there was a resource conflict with the ISA bus driver or some such? 
> 
> [1] https://github.com/tianocore/edk2/blob/master/OvmfPkg/Library/PciHostBridgeLib/PciHostBridgeLib.c#L112

The ResourceAssigned field was introduced in commit c5be19f37821
("MdeModulePkg/PciHostBridgeLib: Add ResourceAssigned field", 2016-05-11).

See also the subsequent commit in the history, commit 401f8cd110f7
("MdeModulePkg/PciHostBridgeDxe: Honor ResourceAssigned", 2016-05-11).

The idea is that setting "ResourceAssigned = TRUE" tells
PciHostBridgeDxe that the ranges exposed in the PCI_ROOT_BRIDGE
structure are not apertures for PciBusDxe to allocate resources from,
but ranges *already* programmed -- by a previously running agent -- into
the root bridge.

In the ArmVirtQemu platform, and when OVMF runs on QEMU, we need to set
ResourceAssigned to FALSE, as we really want PciBusDxe:
- to carve out resources for PCI devices from the bridge apertures,
- and to program the bridge ranges ultimately.

When OVMF runs on Xen however, the PCI enumeration / resource assignment
is performed by a component that runs earlier (IIRC it is performed by
an agent that is somehow part of the hypervisor), and so the edk2 PCI
infrastructure is supposed to perform a "light" enumeration only. That's
why:
-  we set ResourceAssigned to TRUE in
"OvmfPkg/Library/PciHostBridgeLib/XenSupport.c",
- and also why we set "PcdPciDisableBusEnumeration" to TRUE in
"PlatformPei/Xen.c".

The "light enumeration" case -- concerning both
"PcdPciDisableBusEnumeration" and "ResourceAssigned" -- applies to
UefiPayloadPkg as well. There -- I think -- the pre-programming is done
by Coreboot.

The above means that, regardless of "ResourceAssigned", only those MMIO
areas get marked as UC (or even get added to the GCP memory space map)
that are allocated to specific PCI devices:

- With ResourceAssigned=TRUE, the ranges are parsed out of the device
registers.
- With ResourceAssigned=FALSE, the ranges are assigned by PciBusDxe, and
programmed into the registers.

But in both cases, the GCD memory space map reflects the ranges that the
devices actually decode.

Setting ResourceAssigned=TRUE in OVMF (running on QEMU) would completely
break PCI enumeration, as we expose large apertures, and really want
PciBusDxe to grab actual BARs out of those.


On physical machines, I assume root bridges come with pre-determined,
distinct physical address ranges that they can decode, for MMIO
accesses. So I guess in those cases, it could be useful to add the
apertures in full (before the actual resource assignments) to GCD.

On QEMU however, it's the other way around. First, the apertures are not
distinct, instead all root bridges share the same aperture (per type).
Second, the aperture is not pre-determined -- QEMU is fine with any MMIO
BAR assigments, as long as the BARs do not overlap some *otherwise
assigned* guest-physical address range (such as RAM, or platform MMIO).

Therefore, the root bridge apertures exposed by OVMF's PciHostBridgeLib
are synthetic. (For example, a recurrent topic on these lists is how
OVMF's 64-bit MMIO aperture should be sized -- right now it defaults to
32GiB, but you can resize it on the QEMU command line, and some users
would prefer smarter defaults. See for example the TianoCore#2796
ticket. But I digress.)

In ArmVirtQemu, and in OVMF running on QEMU, once we expose the
synthetic aperture(s), and PciBusDxe does carve out / assign actual
resources, then QEMU *deduces* the root bridge apertures from the actual
device programming performed by OVMF. Then QEMU generates the ACPI
tables (the _CRS object) accordingly. And OVMF fetches the ACPI payload
from QEMU, and installs it for the OS to consume.

This is explained in more detail in commit 9116c9c5d885 ("OvmfPkg:
introduce gRootBridgesConnectedEventGroupGuid", 2016-03-23).

In short, in ArmVirtQemu and in OVMF running on QEMU, the resources
assigned by PciBusDxe determine, ultimately, what QEMU will place in the
_CRS, and what the OS will see in the _CRS.

(Side note: where I say "synthetic", that does not mean "easy to change"
at all. Coming up with good 32-bit and 64-bit MMIO apertures for the
PciHostBridgeLib instance was an arduous task. Among other things, this
originates from the very quirky placement of the 32-bit address ranges
on i440fx and q35 that are usable for PCI MMIO. QEMU doesn't dictate the
apertures for the root bridges, but if you allocate an MMIO BAR from an
inappropriate address range, you're doomed.)

So... I'm trying to finish with coherent answers to your questions...

- ResourceAssigned is set to FALSE because we want PciBusDxe to perform
the resource assignment / device programming. On QEMU, the BARs are not
pre-programmed, the firmware has to do it. And, the _CRS in ACPI is
calculated by QEMU as a *function* of PciBusDxe's actions.

- Setting ResourceAssigned to TRUE would break everything on QEMU.
That's because the address ranges we expose in PCI_ROOT_BRIDGE objects
correspond to the "bounding boxes" in guest-physical address space
where QEMU can accommodate MMIO BARs, on all the PCI root bridges taken
together. No resource assignments occur apart from PciBusDxe, the fields
in PCI_ROOT_BRIDGE do not stand for ranges previously programmed into
devices (bridges or endpoints).

Thanks
Laszlo

Re: [edk2-devel] 2nd OVMF question about why InitRootBridge does not set ResourceAssigned?

[Laszlo Ersek]

On 07/07/20 21:59, Laszlo Ersek wrote:

> (Side note: where I say "synthetic", that does not mean "easy to change"
> at all. Coming up with good 32-bit and 64-bit MMIO apertures for the
> PciHostBridgeLib instance was an arduous task. Among other things, this
> originates from the very quirky placement of the 32-bit address ranges
> on i440fx and q35 that are usable for PCI MMIO. QEMU doesn't dictate the
> apertures for the root bridges, but if you allocate an MMIO BAR from an
> inappropriate address range, you're doomed.)

A key commit regarding the above is 49edde15230a ("OvmfPkg/PlatformPei:
set 32-bit UC area at PciBase / PciExBarBase (pc/q35)", 2019-06-03).

It was very difficult to find a 32-bit PCI MMIO aperture, facing any
random guest RAM size, such that we could mark the aperture in
PlatformPei as UC with no more than the 8 variable MTRRs that QEMU provides.

https://bugzilla.tianocore.org/show_bug.cgi?id=1814
https://bugzilla.tianocore.org/show_bug.cgi?id=1859

Thanks
Laszlo

Re: [edk2-devel] 2nd OVMF question about why InitRootBridge does not set ResourceAssigned?

[Andrew Fish]

Laszlo,

Thanks for the background. 

Turns out I was chasing the wrong thing :(. In our tree all MMIO is marked NX and I was talking a page fault. When I tracked it down I realized it was an issue with the EFER MSR not getting the NXE (No-Execute Enable) bit set, so any page table with the NX bit set faults. 

I had a bug in my copy of the DXE IPL that was not calling EnableExecuteDisableBit() so NXE is never set, and that is why I got the page fault. 

Thanks,

Andrew Fish

> On Jul 7, 2020, at 1:21 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> 
> On 07/07/20 21:59, Laszlo Ersek wrote:
> 
>> (Side note: where I say "synthetic", that does not mean "easy to change"
>> at all. Coming up with good 32-bit and 64-bit MMIO apertures for the
>> PciHostBridgeLib instance was an arduous task. Among other things, this
>> originates from the very quirky placement of the 32-bit address ranges
>> on i440fx and q35 that are usable for PCI MMIO. QEMU doesn't dictate the
>> apertures for the root bridges, but if you allocate an MMIO BAR from an
>> inappropriate address range, you're doomed.)
> 
> A key commit regarding the above is 49edde15230a ("OvmfPkg/PlatformPei:
> set 32-bit UC area at PciBase / PciExBarBase (pc/q35)", 2019-06-03).
> 
> It was very difficult to find a 32-bit PCI MMIO aperture, facing any
> random guest RAM size, such that we could mark the aperture in
> PlatformPei as UC with no more than the 8 variable MTRRs that QEMU provides.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1814
> https://bugzilla.tianocore.org/show_bug.cgi?id=1859
> 
> Thanks
> Laszlo
>