From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 5BDB794169B for ; Thu, 21 Nov 2024 12:32:41 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=6AB3RHh8Wudia9caLfJAEutleSLC1GFK7wuwSIEpNGY=; c=relaxed/simple; d=groups.io; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:In-Reply-To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type:Content-Disposition:Content-Transfer-Encoding; s=20240830; t=1732192361; v=1; x=1732451559; b=LDO8yMdEw83iQzEFKPODEKZAjnNZ8Bxc0ouVgVuCv7c3RMdZ8q71gAW4buKl/wh15tFf4mkL rmX35Of2QTSDo7ZMmfRdBe2BKatOc/XcuwbN30nOVBAM5WNPkNxIg/1cKqlD+LubdlG43XzVcRf 8Y+veYOvaAXKOE2raqG+dYfqyGOfgXK9ZhZsenLEg22FHDRt1wNi3SYSlIm4J6gPjos9wWPF1qF xIXNKTVfoWEOIyTIUWoLeK6fxpvRFAHm0Fdhq7bbaVW/SjoCWPOeYMIrBye4AzvX2wAls7vJ/Qy SFP9ixk4oSDbjhEX8IB6lw6QqrJQSAUka6IWTLT3Sanmg== X-Received: by 127.0.0.2 with SMTP id E59DYY7687511xnCwfI8geDU; Thu, 21 Nov 2024 04:32:39 -0800 X-Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.groups.io with SMTP id smtpd.web10.9621.1732192359121646287 for ; Thu, 21 Nov 2024 04:32:39 -0800 X-Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-475-4cpBAQaaMaCgEWPfwLzOJQ-1; Thu, 21 Nov 2024 07:32:37 -0500 X-MC-Unique: 4cpBAQaaMaCgEWPfwLzOJQ-1 X-Mimecast-MFC-AGG-ID: 4cpBAQaaMaCgEWPfwLzOJQ X-Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 070DB19560B8; Thu, 21 Nov 2024 12:32:36 +0000 (UTC) X-Received: from sirius.home.kraxel.org (unknown [10.39.194.164]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AA75A30000DF; Thu, 21 Nov 2024 12:32:35 +0000 (UTC) X-Received: by sirius.home.kraxel.org (Postfix, from userid 1000) id 3DB8618010BF; Thu, 21 Nov 2024 13:32:33 +0100 (CET) Date: Thu, 21 Nov 2024 13:32:33 +0100 From: "Gerd Hoffmann via groups.io" To: mitchell.augustin@canonical.com Cc: devel@edk2.groups.io Subject: Re: [edk2-devel] [BUG] Extremely slow boot times with CPU and GPU passthrough and host phys-bits > 40 Message-ID: References: <18748.1732116039858528046@groups.io> MIME-Version: 1.0 In-Reply-To: <18748.1732116039858528046@groups.io> X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: us8Wx_fC8CZyIp_9CImnii3S5H336ZNbgtMinxAmxKA_1732192356 X-Mimecast-Originator: redhat.com Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Thu, 21 Nov 2024 04:32:39 -0800 Resent-From: kraxel@redhat.com Reply-To: devel@edk2.groups.io,kraxel@redhat.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: ETgbHrUenXxIdHBXZSEUpUdRx7686176AA= Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=LDO8yMdE; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io On Wed, Nov 20, 2024 at 07:20:39AM -0800, via groups.io wrote: > @Gerd > > > Do you also see the slowdown without the GPU in a otherwise identical > guest configuration? > > No - without the GPUs, the entire boot process takes less than 30 > seconds (which is true before and after the dynamic mmio window size > patch ( > https://github.com/tianocore/edk2/commit/ecb778d0ac62560aa172786ba19521f27bc3f650 > ) ). OK. > > Looks quite high to me.  What amount of guest memory we are talking > about? > > It is a pretty large memory allocation - over 900GB - so I'm not > surprised that the initial allocation during `virsh start` takes a > while when PCIe devices are passed through, since that allocation has > to happen at init time. `virsh start` also takes the same amount of > time with or without the dynamic mmio window size patch, but its time > does scale with amount of memory allocated. (although I expect that, > given that the time consuming part is just that memory allocation.) Yes, with PCIe pass through devices all memory is allocated upfront (and pinned). So allocating almost 1TB taking some time looks plausible indeed. > > More details would be helpful indeed.  Is that a general overall > slowdown?  Is it some specific part which takes alot of time? > > The part of the kernel boot that I highlighted in > https://edk2.groups.io/g/devel/attachment/120801/2/this-part-takes-2-3-minutes.txt > (which I think is PCIe device initialization and BAR assignment) is > the part that seems slower than it should be. Each section of that log > starting with "acpiphp: Slot registered" takes probably 15 > seconds, so this whole section adds up to a few minutes. That part > also does not scale with memory allocation, just with number of GPUs > passed through. (in this log, I had 4 GPUs attached, IIRC). Ah, so it's not the firmware assigning the PCI bars but the linux kernel looking at the allocations. Worth a try whenever '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off' on the qemu command line changes something. That turns off acpi-based hotplug for the pcie slots. > (and subsequently, the GPUs are not usable in the VMs (but the PCI > devices are still present)). So it would make sense if the fast boot > time in those versions is simply attributed to the kernel "giving up" > on all of those right away, before the slow path starts. The only > confusing part to me then is why I would not see this part ( > https://edk2.groups.io/g/devel/attachment/120801/2/this-part-takes-2-3-minutes.txt > ) going so slowly when I use a version of OVMF with the dynamic mmio > window size patch reverted but with my guest kernel having > `pci=realloc pci=nocrs` set. Under those circumstances, I have a fast > boot time and my passed-through GPUs work. (although I do still see > some outputs like this during linux boot: With the dynamic mmio window OVMF places the 64-bit PCI MMIO window high in the available address space. Might be this causes trouble for some reason. The address space size can be controlled via qemu cpu properties: '-cpu $name,host-phys-bits=on,host-phys-bits-limit=$bits'. The address space allocations can be checked in /proc/iomem take care, Gerd -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120810): https://edk2.groups.io/g/devel/message/120810 Mute This Topic: https://groups.io/mt/109651206/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-