From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 04E1A740032 for ; Fri, 22 Nov 2024 22:32:16 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=JMHHn48z3oFwvTf2KMEsSRZWCTJ1Cs12wZVwoHCz0Dg=; c=relaxed/simple; d=groups.io; h=Subject:To:From:User-Agent:MIME-Version:Date:References:In-Reply-To:Message-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type; s=20240830; t=1732314736; v=1; x=1732573935; b=SP2JVA9j1PenLOzkJIFcR4lKFexuKoaN+QcsnfWnbimE13BHL8VkUudpU7vMfhu/MQ/7We1Y /+pG3yt6jyczJTnHd9XttvGd0bRuvnDDTvL6HExbcA6CfS/Ar222rhHUYOidbNwahXr3U016SCd j2ralwIS2Oof8TDuJqtSY9NELYKj3bTkqU2ew0KB0L9Tt+CVeZDU8fzWwg+AmITCnYKHtxEIskc gM1hpVvFWJzwNuOBqOvQV1LUaB/NbdHm/UIbyTWsFIRJtxdqdTmYkYnhVmt0M9Ze0zJQ8BYmksw gDZnAXOZ9ReskWA5hsSeaICvQcXK+L4AjQGLKe6b50YXg== X-Received: by 127.0.0.2 with SMTP id 2aHUYY7687511x3xC6gt4tLL; Fri, 22 Nov 2024 14:32:15 -0800 Subject: Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40 To: "Gerd Hoffmann" , devel@edk2.groups.io From: "mitchell.augustin via groups.io" X-Originating-Location: Waterloo, Illinois, US (65.87.55.115) X-Originating-Platform: Linux Chrome 129 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Fri, 22 Nov 2024 14:32:14 -0800 References: In-Reply-To: Message-ID: <5971.1732314734914804329@groups.io> Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mitchell.augustin@canonical.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: zc7d2V9ne8HeuB28FYnuUxGWx7686176AA= Content-Type: multipart/alternative; boundary="WTtUHw4E5TbsFW9pUd5D" X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=SP2JVA9j; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io --WTtUHw4E5TbsFW9pUd5D Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Thanks, Gerd. This result is unfortunately more confusing. When I force physbits=3D40 and= virtual bits=3D48 via -cpu host,host-phys-bits=3Don,host-phys-bits-limit= =3D40,la57=3Doff, and set `pci=3Drealloc pci=3Dnocrs` (and I confirmed that= I see "40 bits physical, 48 bits virtual" in the guest lscpu), I still see= the BAR assignment errors: [ =C2=A0 20.543572] NVRM: This PCI I/O region assigned to your NVIDIA devic= e is invalid: [ =C2=A0 20.543572] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:06:00.0) These are the same widths I see when not using CPU host passthrough, but wh= en I don't use host passthrough, I don't see these errors. (so I guess ther= e is something else that impacts this somehow, beyond the address widths.) This holds true even when I try forcing various different values for opt/ov= mf/X-PciMmio64Mb (I tried 32768, 65536, 131072, 262144, and 524288). (in so= me cases, I would see more BARs get assigned successfully, but would still = get the above error whenever trying to load the GPU driver.) > So you can try increase host-phys-bits-limit. =C2=A0Values between 40 and= 46 should have an effect I do see a bit of a speedup with phys-bits=3D42 and la57=3Doff, and the GPU= s do work with those settings, but I do still see some "can't claim BAR 0" = / "no compatible bridge window" messages during boot, so maybe that speedup= is just because some BARs are getting skipped. I also did some profiling of the guest kernel today (with my original slow = VM config), to see if that gave me any valuable information. One thing of n= ote is that it seems like the most time is spent in this pci_write_config_w= ord() call in __pci_read_base() of drivers/pci/probe.c ( https://git.kernel= .org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/probe.c?h= =3Dv6.12#n251 ). Each of those pci_write_config_word() calls takes about 2 = seconds, but it adds up to a significant chunk of the initialization time s= ince __pci_read_base() is executed somewhere between 20-40 times in my VM. I don't know if that is necessarily indicative of an issue with the guest k= ernel or not, but wanted to add it here as a datapoint in case it rings any= bells. Thanks, Mitchell Augustin -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120821): https://edk2.groups.io/g/devel/message/120821 Mute This Topic: https://groups.io/mt/109651206/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- --WTtUHw4E5TbsFW9pUd5D Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Thanks, Gerd.
 
This result is unfortunately more confusing. When I force physbits=3D4= 0 and virtual bits=3D48 via -cpu host,host-phys-bits=3Don,host-phys-bits-li= mit=3D40,la57=3Doff, and set `pci=3Drealloc pci=3Dnocrs` (and I confirmed t= hat I see "40 bits physical, 48 bits virtual" in the guest lscpu), I still = see the BAR assignment errors:
[   20.543572] NVRM: This PCI I/O region assigned to your NVIDIA = device is invalid:
[   20.543572] NVRM: BAR0 is 0M @ 0x0 (PCI:000= 0:06:00.0)
These are the same widths I see when not using CPU host = passthrough, but when I don't use host passthrough, I don't see these error= s. (so I guess there is something else that impacts this somehow, beyond th= e address widths.)
This holds true even when I try forcing various different values for o= pt/ovmf/X-PciMmio64Mb (I tried 32768, 65536, 131072, 262144, and 524288). (= in some cases, I would see more BARs get assigned successfully, but would s= till get the above error whenever trying to load the GPU driver.)
 
> So you can try increase host-phys-bits-limit.  Values betwee= n 40 and 46 should have an effect
 
I do see a bit of a speedup with phys-bits=3D42 and la57=3Doff, and th= e GPUs do work with those settings, but I do still see some "can't claim BA= R 0" / "no compatible bridge window" messages during boot, so maybe that sp= eedup is just because some BARs are getting skipped.
 
 
I also did some profiling of the guest kernel today (with my original = slow VM config), to see if that gave me any valuable information. One thing= of note is that it seems like the most time is spent in this pci_write= _config_word() call in __pci_read_base() of drivers/pci/probe.c. Each o= f those pci_write_config_word() calls takes about 2 seconds, but it adds up= to a significant chunk of the initialization time since __pci_read_base() = is executed somewhere between 20-40 times in my VM. 
I don't know if that is necessarily indicative of an issue with the gu= est kernel or not, but wanted to add it here as a datapoint in case it ring= s any bells.
 
Thanks,
Mitchell Augustin
_._,_._,_
Groups.io Links:

=20 You receive all messages sent to this group. =20 =20

View/Reply Online (#120821) | =20 | Mute= This Topic | New Topic
Your Subscriptio= n | Contact Group Owner | Unsubscribe [rebecca@openfw.io]

_._,_._,_
--WTtUHw4E5TbsFW9pUd5D--