From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 5D8C29415B2 for ; Mon, 25 Nov 2024 11:18:32 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=LWvffvOcwjCsABWd/PKn7psI+q0FC8Qm/8xCMOr2AnU=; c=relaxed/simple; d=groups.io; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:In-Reply-To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type:Content-Disposition:Content-Transfer-Encoding; s=20240830; t=1732533512; v=1; x=1732792710; b=kGmm3dC2kZqR3drHvcBHPBqZUljH3uSVhi+1ntmgULkPa4tqCEaNmVHm0WWLrK0MIxBybA94 4hpEKuo/B04BJAL06znraQL/L2B36OjXItZ3g41JMku7tEId8pRU/IIfvY/gf39JtVry8r0KL7M 1sBn3UgiTig7RezGZuTj0dEUjMYxIlXGO+ys9ieEOH4I852tofL5H4gQJE59FnmBweHueXk81jB mHfIe9y448C0Y5onCtfKoqF4x53Tgu3lLj+7KoT2TkOJcVIfw1ophjJ6a5zIb4g3tRrKpwLZvtJ yiy+qycgrWrF990Z450MpqoWvEwfnT7YfSQwAhNTiT87A== X-Received: by 127.0.0.2 with SMTP id muDhYY7687511xPrMzQnt66s; Mon, 25 Nov 2024 03:18:30 -0800 X-Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web10.16546.1732533509096486112 for ; Mon, 25 Nov 2024 03:18:29 -0800 X-Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-326-X4F5natkNhe1Cs_WvEtllQ-1; Mon, 25 Nov 2024 06:18:23 -0500 X-MC-Unique: X4F5natkNhe1Cs_WvEtllQ-1 X-Mimecast-MFC-AGG-ID: X4F5natkNhe1Cs_WvEtllQ X-Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C090E19560B8; Mon, 25 Nov 2024 11:18:19 +0000 (UTC) X-Received: from sirius.home.kraxel.org (unknown [10.39.192.111]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 64FDE1956052; Mon, 25 Nov 2024 11:18:19 +0000 (UTC) X-Received: by sirius.home.kraxel.org (Postfix, from userid 1000) id 122D21800392; Mon, 25 Nov 2024 12:18:17 +0100 (CET) Date: Mon, 25 Nov 2024 12:18:17 +0100 From: "Gerd Hoffmann via groups.io" To: mitchell.augustin@canonical.com Cc: devel@edk2.groups.io Subject: Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40 Message-ID: References: <5971.1732314734914804329@groups.io> MIME-Version: 1.0 In-Reply-To: <5971.1732314734914804329@groups.io> X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: NHYAKfnjwRXoNd5uZD8gj-b4DymwcUU3F4POz_1tHOg_1732533500 X-Mimecast-Originator: redhat.com Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Mon, 25 Nov 2024 03:18:30 -0800 Resent-From: kraxel@redhat.com Reply-To: devel@edk2.groups.io,kraxel@redhat.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: Yfe1iLboOyViILr5xlSDUuenx7686176AA= Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=kGmm3dC2; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io On Fri, Nov 22, 2024 at 02:32:14PM -0800, via groups.io wrote: > Thanks, Gerd. > > This result is unfortunately more confusing. When I force physbits=40 > and virtual bits=48 via -cpu > host,host-phys-bits=on,host-phys-bits-limit=40,la57=off, and set > `pci=realloc pci=nocrs` (and I confirmed that I see "40 bits physical, > 48 bits virtual" in the guest lscpu), I still see the BAR assignment > errors: > [   20.543572] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: > [   20.543572] NVRM: BAR0 is 0M @ 0x0 (PCI:0000:06:00.0) Well, 40 phys-bits equals 1TB address space, which is not enough for 900 GB of ram plus pci devices. You need at least 41 (for 2TB address space). > These are the same widths I see when not using CPU host passthrough, > but when I don't use host passthrough, I don't see these errors. (so I > guess there is something else that impacts this somehow, beyond the > address widths.) Probably gigabyte pages (-cpu $name,pdpe1gb={on|off}). Without gigabyte pages OVMF will not use more than 40 phys-bits, to limit the amount of memory needed for page tables, and also avoid slow boot (creating large identity mappings with 2M pages takes time ...). You can grep the firmware logs for '^Platform' to see some messages about the address space layout OVMF is creating. > I do see a bit of a speedup with phys-bits=42 and la57=off, and the > GPUs do work with those settings, but I do still see some "can't claim > BAR 0" / "no compatible bridge window" messages during boot, so maybe > that speedup is just because some BARs are getting skipped. /proc/iomem shows the memory layout: $ sudo cat /proc/iomem [ ... low RAM regions ... ] 80000000-dfffffff : PCI Bus 0000:00 <-- 32-bit MMIO window. [ ... pci buses and devices ... ] e0000000-efffffff : PCI ECAM 0000 [bus 00-ff] <-- PCI config space. e0000000-efffffff : Reserved e0000000-efffffff : pnp 00:04 [ ... some devices like ioapic ... ] [ ... high RAM regions ... ] 7000000000-77ffffffff : PCI Bus 0000:00 <-- 64-bit MMIO window. [ ... pci buses and devices ... ] (this is with phys-bits 39, with more phys-bits the 64-bit MMIO window is larger and mapped higher). > Each of those pci_write_config_word() calls takes about 2 seconds, That is extremely slow. How does /proc/iomem look like? Anything overlapping the ECAM maybe? > I don't know if that is necessarily indicative of an issue with the > guest kernel or not, but wanted to add it here as a datapoint in case > it rings any bells. Could be in the guest, could also be the host (which creates the EPT page tables). take care, Gerd -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120830): https://edk2.groups.io/g/devel/message/120830 Mute This Topic: https://groups.io/mt/109651206/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-