[edk2-devel] FDF parser performance degrades rapidly on non-trivially sized inputs

["davidr via groups.io" <davidr=ghs.com@groups.io>]

[-- Attachment #1: Type: text/plain, Size: 3150 bytes --]

Hi,

I was testing out dumping a raw OVMF_VARS.fd into an FDF data section and noticed that my rebuilds of OVMF with no code changes went from 15 seconds to over 1 minute. The only change was the data in the NV_VARIABLE_STORE data section in OvmfPkg/Include/Fdf/VarStore.fdf.inc which changed the size of the file from about 4 KiB to about 256 KiB. I was rather curious as to why my build times changed so much and profiled the build process with cProfile. Specifically https://github.com/tianocore/edk2/blob/master/BaseTools/Source/Python/GenFds/FdfParser.py#L279 takes the vast majority of the time.

You can reproduce this by adding 256 KiB of "#" characters to the end of OvmfPkg/Include/Fdf/VarStore.fdf.inc and building OVMF, which produced this result for me:
Build total time: 00:01:31
34883442 function calls (34368868 primitive calls) in 91.191 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
18769   73.501    0.004   75.942    0.004 FdfParser.py:276(_SkipWhiteSpace)
728    5.738    0.008    5.738    0.008 {method 'acquire' of '_thread.lock' objects}
12    1.569    0.131    1.569    0.131 {method 'poll' of 'select.poll' objects}
2849255    1.248    0.000    1.439    0.000 FdfParser.py:354(_GetOneChar)
2305959    0.873    0.000    1.131    0.000 FdfParser.py:293(_EndOfFile)
5374641    0.831    0.000    0.831    0.000 FdfParser.py:368(_CurrentChar)
2    0.526    0.263    1.278    0.639 FdfParser.py:497(PreprocessFile)

Changing _SkippedChars from a string to StringIO reduced my build time to 19 seconds:
Build total time: 00:00:19
36552029 function calls (36037563 primitive calls) in 18.618 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
728    5.391    0.007    5.391    0.007 {method 'acquire' of '_thread.lock' objects}
18769    1.593    0.000    3.452    0.000 FdfParser.py:277(_SkipWhiteSpace)
12    1.551    0.129    1.551    0.129 {method 'poll' of 'select.poll' objects}
2849255    0.850    0.000    0.979    0.000 FdfParser.py:355(_GetOneChar)
5374641    0.742    0.000    0.742    0.000 FdfParser.py:369(_CurrentChar)
2305959    0.741    0.000    0.951    0.000 FdfParser.py:294(_EndOfFile)
2    0.511    0.256    1.237    0.618 FdfParser.py:498(PreprocessFile)

All of these tests were run using the python3 binary provided in the docker container created by https://github.com/tianocore/containers/tree/main/Ubuntu-22/Dockerfile ( https://github.com/tianocore/containers/blob/main/Ubuntu-22/Dockerfile ).

This seems like an easy change to make builds a tiny bit faster.

Thanks,
David


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120865): https://edk2.groups.io/g/devel/message/120865
Mute This Topic: https://groups.io/mt/109914552/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-



[-- Attachment #2: Type: text/html, Size: 4883 bytes --]