Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40

["Gerd Hoffmann via groups.io" <kraxel=redhat.com@groups.io>]

On Fri, Nov 22, 2024 at 02:32:14PM -0800,  via groups.io wrote:
> Thanks, Gerd.
> 
> This result is unfortunately more confusing. When I force physbits=40
> and virtual bits=48 via -cpu
> host,host-phys-bits=on,host-phys-bits-limit=40,la57=off, and set
> `pci=realloc pci=nocrs` (and I confirmed that I see "40 bits physical,
> 48 bits virtual" in the guest lscpu), I still see the BAR assignment
> errors:

> [   20.543572] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
> [   20.543572] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:06:00.0)

Well, 40 phys-bits equals 1TB address space, which is not enough for 900
GB of ram plus pci devices.  You need at least 41 (for 2TB address
space).

> These are the same widths I see when not using CPU host passthrough,
> but when I don't use host passthrough, I don't see these errors. (so I
> guess there is something else that impacts this somehow, beyond the
> address widths.)

Probably gigabyte pages (-cpu $name,pdpe1gb={on|off}).  Without gigabyte
pages OVMF will not use more than 40 phys-bits, to limit the amount of
memory needed for page tables, and also avoid slow boot (creating large
identity mappings with 2M pages takes time ...).

You can grep the firmware logs for '^Platform' to see some messages
about the address space layout OVMF is creating.

> I do see a bit of a speedup with phys-bits=42 and la57=off, and the
> GPUs do work with those settings, but I do still see some "can't claim
> BAR 0" / "no compatible bridge window" messages during boot, so maybe
> that speedup is just because some BARs are getting skipped.

/proc/iomem shows the memory layout:

$ sudo cat /proc/iomem
[ ... low RAM regions ... ]
80000000-dfffffff : PCI Bus 0000:00             <-- 32-bit MMIO window.
  [ ... pci buses and devices ... ]
e0000000-efffffff : PCI ECAM 0000 [bus 00-ff]   <-- PCI config space.
  e0000000-efffffff : Reserved
    e0000000-efffffff : pnp 00:04
[ ... some devices like ioapic ... ]
[ ... high RAM regions ... ]
7000000000-77ffffffff : PCI Bus 0000:00         <-- 64-bit MMIO window.
  [ ... pci buses and devices ... ]

(this is with phys-bits 39, with more phys-bits the 64-bit MMIO window
is larger and mapped higher).

> Each of those pci_write_config_word() calls takes about 2 seconds,

That is extremely slow.  How does /proc/iomem look like?  Anything
overlapping the ECAM maybe?

> I don't know if that is necessarily indicative of an issue with the
> guest kernel or not, but wanted to add it here as a datapoint in case
> it rings any bells.

Could be in the guest, could also be the host (which creates the EPT
page tables).

take care,
  Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120830): https://edk2.groups.io/g/devel/message/120830
Mute This Topic: https://groups.io/mt/109651206/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-