Thanks, Gerd.

 

This result is unfortunately more confusing. When I force physbits=40 and virtual bits=48 via -cpu host,host-phys-bits=on,host-phys-bits-limit=40,la57=off, and set `pci=realloc pci=nocrs` (and I confirmed that I see "40 bits physical, 48 bits virtual" in the guest lscpu), I still see the BAR assignment errors:

[   20.543572] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
[   20.543572] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:06:00.0)

These are the same widths I see when not using CPU host passthrough, but when I don't use host passthrough, I don't see these errors. (so I guess there is something else that impacts this somehow, beyond the address widths.)

This holds true even when I try forcing various different values for opt/ovmf/X-PciMmio64Mb (I tried 32768, 65536, 131072, 262144, and 524288). (in some cases, I would see more BARs get assigned successfully, but would still get the above error whenever trying to load the GPU driver.)

 

> So you can try increase host-phys-bits-limit.  Values between 40 and 46 should have an effect

 

I do see a bit of a speedup with phys-bits=42 and la57=off, and the GPUs do work with those settings, but I do still see some "can't claim BAR 0" / "no compatible bridge window" messages during boot, so maybe that speedup is just because some BARs are getting skipped.

 

 

I also did some profiling of the guest kernel today (with my original slow VM config), to see if that gave me any valuable information. One thing of note is that it seems like the most time is spent in this pci_write_config_word() call in __pci_read_base() of drivers/pci/probe.c. Each of those pci_write_config_word() calls takes about 2 seconds, but it adds up to a significant chunk of the initialization time since __pci_read_base() is executed somewhere between 20-40 times in my VM. 

I don't know if that is necessarily indicative of an issue with the guest kernel or not, but wanted to add it here as a datapoint in case it rings any bells.

 

Thanks,

Mitchell Augustin

_._,_._,_

_._,_._,_