Wasn’t there another push (somewhere in the last 8 months, my brain is foggy) to adopt LZMA2? Or was it a different algorithm? - Bret From: Daniel Schaefer via groups.io Sent: Thursday, December 3, 2020 4:12 AM To: Laszlo Ersek; devel@edk2.groups.io Cc: Lin, Derek (HPS SW) Subject: [EXTERNAL] Re: [edk2-devel] Multithreaded compression with LZMA2 From: Laszlo Ersek Sent: Thursday, December 3, 2020 18:24 To: devel@edk2.groups.io ; Schaefer, Daniel Cc: Lin, Derek (HPS SW) Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 On 12/02/20 03:59, Daniel Schaefer wrote: > Hi everyone, > > I'm looking into how to speed up the build process and noticed that our > build > uses LZMA to encrypt the main firmware volume. Since it's quite big it > takes a > while but only uses one CPU thread. > > LZMA2 is a version of LZMA which can be multi-threaded and achieve much > faster > compression times. I did a quick benchmark using the `xz` command-line > tool, > which uses a modified version of the LZMA SDK that EDK2 uses. The > results are: > > Uncompressed size: 64M > > | Algo | Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA | 19.67s | 0.9s | 9.1M | 1 | > | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | > | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | > > Using those commands: > > time xz --format=lzma testfile > time unlzma testfile.lzma > > time xz --lzma2 testfile > time unxz testfile.xz > > time xz -T4 --lzma2 testfile > time unxz testfile.xz > > This is quite a significant improvement of build time, while > decompression time > and size only slightly increase. If that's a concern, then LZMA2 could > be used > for development only. > > I haven't investigated the details of how to support this in the code > but it > appears to be a simple change, since the LZMA SDK that we use already > supports > LZMA2. > > What do you think? "xz -T" works by splitting the input into blocks, and it generates a multi-block compressed output. Yes, that's correct. > I'm unsure if the current LZMA decompressor that runs inside the firmware (= guided section extractor) copes with multi-block input. I think you're right that it doesn't. But we can make the guided section extractor use that same algorithm(LZMA2) and assign it a different GUID, right?