From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 9C3487803CD for ; Tue, 26 Nov 2024 22:27:30 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=F2dOdKocdtOSBIW4WGrEe4ixLCaoLTPzwQIDWRP/H48=; c=relaxed/simple; d=groups.io; h=Subject:To:From:User-Agent:MIME-Version:Date:References:In-Reply-To:Message-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type; s=20240830; t=1732660050; v=1; x=1732919249; b=Qyzy5E2/46+hfRfiMl42ak1p72tJjb4IM0Q9hoKDhM26sXtdF3v+eGvKQAtwANVbgTUboNiI QYgkvsK+dDyM87CnDxzFOLTQ2pRp2HvzV77bNMbALK4JKaLS7GaXqSfdddbdlLLqOnpnDFg9eM4 kvqCi9XPJMZuEMDhcwJSNmzVdrBYM+S5CA1wyxJ0uaTdEavEE+IOqRu2UFCyZ8bjaPVd7n5W7cw 2DUUix3YeQIVZ8+ZVuDSvxGtmzadfV5cJbdJ0SQbQEOoqmfD6wXFy3YKu8CJqZqJiQYgJc3jhBw xCEAt9b+ese8QzQLYO7bMvzzeLeBzhIoDYPNTrjNcsztw== X-Received: by 127.0.0.2 with SMTP id M6HqYY7687511x6pWW6Z6gt1; Tue, 26 Nov 2024 14:27:29 -0800 Subject: Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40 To: "Gerd Hoffmann" , devel@edk2.groups.io From: "mitchell.augustin via groups.io" X-Originating-Location: Waterloo, Illinois, US (65.87.55.115) X-Originating-Platform: Linux Chrome 129 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Tue, 26 Nov 2024 14:27:28 -0800 References: In-Reply-To: Message-ID: <2202.1732660048387996911@groups.io> Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mitchell.augustin@canonical.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: YlC5li3NhkeZwVxnZA2HTxLhx7686176AA= Content-Type: multipart/alternative; boundary="xsUgw1jbzlJwSqW7P4MN" X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b="Qyzy5E2/"; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io --xsUgw1jbzlJwSqW7P4MN Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Thanks Gerd, > When hotplugging devices you might need the 'pref64-reserve=3D' pro= perty for '-device pcie-root-port' to make the bridge window larger. Double thanks for this - it was exactly what I needed, and I am now able to= get hotplug working on a VM where the GPUs weren't attached at boot. (and = I can confirm that there is no PCI initialization slowdown when I do it tha= t way, as opposed to attaching at boot time.) > I think it would be better to just give the PciMmio64Mb config hints high= er priority, i.e. if they are present (and don't conflict with installed me= mory) simply use them as-is Seems reasonable. Since I was able to get the hotplug approach working as I= 'd hoped, this may not end up being needed for me (if hotplug is appropriat= e for my colleague's use case), but I'll keep this in mind. > But I'd also like to figure what the exact problem is. =C2=A0Ideally OVMF= should just work without requiring manual configuration. =C2=A0One of the = reasons to add the dynamic mmio window was to allow pci devices with large = bars to work fine without manual tuning ... Agreed - I'm hoping to get closer to that answer in this thread ( https://l= ore.kernel.org/all/CAHTA-uYp07FgM6T1OZQKqAdSA5JrZo0ReNEyZgQZub4mDRrV5w@mail= .gmail.com/ ) that I started in the kvm/pci lists. As far as I can tell, Pl= atformDynamicMMIOWindow isn't doing anything wrong here - it gives me a val= id sized window - it just so happens that we were benefiting from the old i= mproperly sized window in a roundabout way as long as our `pci=3Drealloc` w= orkaround was in place in older OVMF versions. > Can you try to change the manual configuration to be more like the dynami= c mmio window configuration, one change at a time (first size, next base, p= ossibly multiple base addresses) and see where exactly it breaks? Does it m= ake a difference if you add a iommu to the virtual machine? > Hmm. =C2=A0Looks like the device has more resources on the host. Maybe *t= hat* is the problem. > What does 'sudo lspci -v' print for the NPU on the host and in the guest? I'll be out for a few days on holiday, so I'll take a look at these sometim= e next week. Thanks again for all the help so far, Mitchell Augustin -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120846): https://edk2.groups.io/g/devel/message/120846 Mute This Topic: https://groups.io/mt/109651206/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- --xsUgw1jbzlJwSqW7P4MN Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Thanks Gerd,
 
> When hotplugging devices you might need the 'pref64-reserve=3D<= ;size>' property for '-device pcie-root-port' to make the bridge window = larger.
 
Double thanks for this - it was exactly what I needed, and I am now ab= le to get hotplug working on a VM where the GPUs weren't attached at boot. = (and I can confirm that there is no PCI initialization slowdown when I do i= t that way, as opposed to attaching at boot time.)
 
> I think it would be better to just give the PciMmio64Mb config hi= nts higher priority, i.e. if they are present (and don't conflict with inst= alled memory) simply use them as-is
 
Seems reasonable. Since I was able to get the hotplug approach working= as I'd hoped, this may not end up being needed for me (if hotplug= is appropriate for my colleague's use case), but I'll keep this in mind.
 
> But I'd also like to figure what the exact problem is.  Idea= lly OVMF should just work without requiring manual configuration.  One= of the reasons to add the dynamic mmio window was to allow pci devices wit= h large bars to work fine without manual tuning ...
 
Agreed - I'm hoping to get closer to that answer in this thread that I st= arted in the kvm/pci lists. As far as I can tell, PlatformDynamicMMIOWindow= isn't doing anything wrong here - it gives me a valid sized window - it ju= st so happens that we were benefiting from the old improperly sized window = in a roundabout way as long as our `pci=3Drealloc` workaround was in place = in older OVMF versions.
 
> C= an you try to change the manual configuration to be more like the dynamic mmio = window configuration, one change at a time (first size, next base, possibly mu= ltiple base addresses) and see where exactly it breaks? Does it make a difference if you add a= iommu to the virtual machine? 
> Hmm.  Looks like the device has more resources on the host. = Maybe *that* is the problem.
> What does 'sudo lspci -v' print for the NPU on the host and in th= e guest?
 
I'll be out for a few days on holiday, so I'll take a look at these so= metime next week.
 
Thanks again for all the help so far,
 
Mitchell Augustin
 
_._,_._,_
Groups.io Links:

=20 You receive all messages sent to this group. =20 =20

View/Reply Online (#120846) | =20 | Mute= This Topic | New Topic
Your Subscriptio= n | Contact Group Owner | Unsubscribe [rebecca@openfw.io]

_._,_._,_
--xsUgw1jbzlJwSqW7P4MN--