Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40

["Gerd Hoffmann via groups.io" <kraxel=redhat.com@groups.io>]

On Wed, Nov 20, 2024 at 07:20:39AM -0800,  via groups.io wrote:
> @Gerd
> 
> > Do you also see the slowdown without the GPU in a otherwise identical
> guest configuration?
> 
> No - without the GPUs, the entire boot process takes less than 30
> seconds (which is true before and after the dynamic mmio window size
> patch (
> https://github.com/tianocore/edk2/commit/ecb778d0ac62560aa172786ba19521f27bc3f650
> ) ).

OK.

> > Looks quite high to me.  What amount of guest memory we are talking
> about?
> 
> It is a pretty large memory allocation - over 900GB - so I'm not
> surprised that the initial allocation during `virsh start` takes a
> while when PCIe devices are passed through, since that allocation has
> to happen at init time. `virsh start` also takes the same amount of
> time with or without the dynamic mmio window size patch, but its time
> does scale with amount of memory allocated. (although I expect that,
> given that the time consuming part is just that memory allocation.)

Yes, with PCIe pass through devices all memory is allocated upfront
(and pinned).  So allocating almost 1TB taking some time looks
plausible indeed.

> > More details would be helpful indeed.  Is that a general overall
> slowdown?  Is it some specific part which takes alot of time?
> 
> The part of the kernel boot that I highlighted in
> https://edk2.groups.io/g/devel/attachment/120801/2/this-part-takes-2-3-minutes.txt
> (which I think is PCIe device initialization and BAR assignment) is
> the part that seems slower than it should be. Each section of that log
> starting with "acpiphp: Slot <slot> registered" takes probably 15
> seconds, so this whole section adds up to a few minutes. That part
> also does not scale with memory allocation, just with number of GPUs
> passed through. (in this log, I had 4 GPUs attached, IIRC).

Ah, so it's not the firmware assigning the PCI bars but the linux
kernel looking at the allocations.

Worth a try whenever
	'-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'
on the qemu command line changes something.  That turns off acpi-based
hotplug for the pcie slots.

> (and subsequently, the GPUs are not usable in the VMs (but the PCI
> devices are still present)). So it would make sense if the fast boot
> time in those versions is simply attributed to the kernel "giving up"
> on all of those right away, before the slow path starts. The only
> confusing part to me then is why I would not see this part (
> https://edk2.groups.io/g/devel/attachment/120801/2/this-part-takes-2-3-minutes.txt
> ) going so slowly when I use a version of OVMF with the dynamic mmio
> window size patch reverted but with my guest kernel having
> `pci=realloc pci=nocrs` set. Under those circumstances, I have a fast
> boot time and my passed-through GPUs work. (although I do still see
> some outputs like this during linux boot:

With the dynamic mmio window OVMF places the 64-bit PCI MMIO window
high in the available address space.  Might be this causes trouble
for some reason.

The address space size can be controlled via qemu cpu properties:
'-cpu $name,host-phys-bits=on,host-phys-bits-limit=$bits'.

The address space allocations can be checked in /proc/iomem

take care,
  Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120810): https://edk2.groups.io/g/devel/message/120810
Mute This Topic: https://groups.io/mt/109651206/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-