From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail05.groups.io (mail05.groups.io [45.79.224.7]) by spool.mail.gandi.net (Postfix) with ESMTPS id 62EEAAC1014 for ; Wed, 4 Dec 2024 10:22:56 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=OUggsQPWn3gEXNxB8g0JZfQfQAYkR0X3LrH8s/uyfSM=; c=relaxed/simple; d=groups.io; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Resent-Date:Resent-From:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Type:Content-Transfer-Encoding; s=20240830; t=1733307776; v=1; x=1733566974; b=LiXIhR2M9P63Qsn9pZ18yPAxyIOECbsnSkGvwAprMjqvZxIzo17HhcxFk2CuqUPue9kh2AnW UzpN6Jkg9ylcx3/hYERxxWJrSLe0fgtdgdcaPvkos7cfG2bR9rMbiVrGhIvipa6aGNgKS0KghMr bMRxyjl+6Dgjl8pwVZPELMQqIYo3wBBuC7CvvTIhSRLI2t0cy/fTaHU6ShWV/PW7qri14GYEF8J Z+8yGPnq1oBpYBV45yYSkqaslnF3f+FKGOVFNZVMS8Um17HcUu7T/qM54tKkozm0Icq7DpNsPMH y9TkkdKdbjpGr6QIidC6FGo7MTWa5yBAk/mA8y3b8KQug== X-Received: by 127.0.0.2 with SMTP id 7qmlYY7687511xr8FiOulTY5; Wed, 04 Dec 2024 02:22:54 -0800 X-Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mx.groups.io with SMTP id smtpd.web11.10714.1733307773980239121 for ; Wed, 04 Dec 2024 02:22:54 -0800 X-Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 6908B5C6B40 for ; Wed, 4 Dec 2024 10:22:10 +0000 (UTC) X-Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0CB46C4CED6 for ; Wed, 4 Dec 2024 10:22:53 +0000 (UTC) X-Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-53de84e4005so7673864e87.0 for ; Wed, 04 Dec 2024 02:22:52 -0800 (PST) X-Gm-Message-State: SrThgaZ6Yci8Hw7rlLvpPjIdx7686176AA= X-Google-Smtp-Source: AGHT+IFsFZZSUx0HmSwb2jknaAWf2UY0xVyJ8gK7FPFQe1mp8TDwm2WpQy0ZPz9+yDYiHOw7nv68mZLl3qbrN7wfDbU= X-Received: by 2002:a05:6512:b8f:b0:53d:dc3a:ed92 with SMTP id 2adb3069b0e04-53e12a22f87mr3546614e87.37.1733307771393; Wed, 04 Dec 2024 02:22:51 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: "Ard Biesheuvel via groups.io" Date: Wed, 4 Dec 2024 11:22:40 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [edk2-devel] FDF parser performance degrades rapidly on non-trivially sized inputs To: devel@edk2.groups.io, davidr@ghs.com, Rebecca Cran Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Resent-Date: Wed, 04 Dec 2024 02:22:54 -0800 Resent-From: ardb@kernel.org Reply-To: devel@edk2.groups.io,ardb@kernel.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20240830 header.b=LiXIhR2M; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 45.79.224.7 as permitted sender) smtp.mailfrom=bounce@groups.io On Wed, 4 Dec 2024 at 05:20, davidr via groups.io wrote: > > Hi, > > I was testing out dumping a raw OVMF_VARS.fd into an FDF data section and= noticed that my rebuilds of OVMF with no code changes went from 15 seconds= to over 1 minute. The only change was the data in the NV_VARIABLE_STORE da= ta section in OvmfPkg/Include/Fdf/VarStore.fdf.inc which changed the size o= f the file from about 4 KiB to about 256 KiB. I was rather curious as to wh= y my build times changed so much and profiled the build process with cProfi= le. Specifically https://github.com/tianocore/edk2/blob/master/BaseTools/So= urce/Python/GenFds/FdfParser.py#L279 takes the vast majority of the time. > > You can reproduce this by adding 256 KiB of "#" characters to the end of = OvmfPkg/Include/Fdf/VarStore.fdf.inc and building OVMF, which produced this= result for me: > Build total time: 00:01:31 > 34883442 function calls (34368868 primitive calls) in 91.191 sec= onds > Ordered by: internal time > ncalls tottime percall cumtime percall filename:lineno(function) > 18769 73.501 0.004 75.942 0.004 FdfParser.py:276(_SkipWhite= Space) > 728 5.738 0.008 5.738 0.008 {method 'acquire' of '_thre= ad.lock' objects} > 12 1.569 0.131 1.569 0.131 {method 'poll' of 'select.p= oll' objects} > 2849255 1.248 0.000 1.439 0.000 FdfParser.py:354(_GetOneCha= r) > 2305959 0.873 0.000 1.131 0.000 FdfParser.py:293(_EndOfFile= ) > 5374641 0.831 0.000 0.831 0.000 FdfParser.py:368(_CurrentCh= ar) > 2 0.526 0.263 1.278 0.639 FdfParser.py:497(Preprocess= File) > > Changing _SkippedChars from a string to StringIO reduced my build time to= 19 seconds: > Build total time: 00:00:19 > 36552029 function calls (36037563 primitive calls) in 18.618 sec= onds > Ordered by: internal time > ncalls tottime percall cumtime percall filename:lineno(function) > 728 5.391 0.007 5.391 0.007 {method 'acquire' of '_thre= ad.lock' objects} > 18769 1.593 0.000 3.452 0.000 FdfParser.py:277(_SkipWhite= Space) > 12 1.551 0.129 1.551 0.129 {method 'poll' of 'select.p= oll' objects} > 2849255 0.850 0.000 0.979 0.000 FdfParser.py:355(_GetOneCha= r) > 5374641 0.742 0.000 0.742 0.000 FdfParser.py:369(_CurrentCh= ar) > 2305959 0.741 0.000 0.951 0.000 FdfParser.py:294(_EndOfFile= ) > 2 0.511 0.256 1.237 0.618 FdfParser.py:498(Preprocess= File) > > All of these tests were run using the python3 binary provided in the dock= er container created by https://github.com/tianocore/containers/tree/main/U= buntu-22/Dockerfile. > > This seems like an easy change to make builds a tiny bit faster. > Thanks for the report. Mind sending a patch / PR ? -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#120866): https://edk2.groups.io/g/devel/message/120866 Mute This Topic: https://groups.io/mt/109914552/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-