Re: [edk2-devel] Multithreaded compression with LZMA2

["Daniel Schaefer" <daniel.schaefer@hpe.com>]

On 12/3/20 11:57 PM, Bret Barkelew via groups.io wrote:
> Wasn’t there another push (somewhere in the last 8 months, my brain is foggy) to adopt LZMA2? Or was it a different algorithm?

I couldn't find anything: https://edk2.groups.io/g/devel/search?q=LZMA2

> 
> - Bret
> 
> *From: *Daniel Schaefer via groups.io <mailto:daniel.schaefer=hpe.com@groups.io>
> *Sent: *Thursday, December 3, 2020 4:12 AM
> *To: *Laszlo Ersek <mailto:lersek@redhat.com>; devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> *Cc: *Lin, Derek (HPS SW) <mailto:derek.lin2@hpe.com>
> *Subject: *[EXTERNAL] Re: [edk2-devel] Multithreaded compression with LZMA2
> 
> *From:* Laszlo Ersek <lersek@redhat.com>
> 
> *Sent:*Thursday, December 3, 2020 18:24
> *To:* devel@edk2.groups.io <devel@edk2.groups.io>; Schaefer, Daniel <daniel.schaefer@hpe.com>
> *Cc:* Lin, Derek (HPS SW) <derek.lin2@hpe.com>
> *Subject:* Re: [edk2-devel] Multithreaded compression with LZMA2
> 
> On 12/02/20 03:59, Daniel Schaefer wrote:
>  > Hi everyone,
>  >
>  > I'm looking into how to speed up the build process and noticed that our
>  > build
>  > uses LZMA to encrypt the main firmware volume. Since it's quite big it
>  > takes a
>  > while but only uses one CPU thread.
>  >
>  > LZMA2 is a version of LZMA which can be multi-threaded and achieve much
>  > faster
>  > compression times. I did a quick benchmark using the `xz` command-line
>  > tool,
>  > which uses a modified version of the LZMA SDK that EDK2 uses. The
>  > results are:
>  >
>  > Uncompressed size: 64M
>  >
>  > | Algo  | Comp Time | Decomp Time | Size | Threads |
>  > | ----- | --------- | ----------- | ---- | ------- |
>  > | LZMA  |    19.67s |        0.9s | 9.1M |       1 |
>  > | LZMA2 |    20.11s |        1.2s | 9.2M |       1 |
>  > | LZMA2 |     8.31s |        1.0s | 9.4M |       4 |
>  >
>  > Using those commands:
>  >
>  > time xz --format=lzma testfile
>  > time unlzma testfile.lzma
>  >
>  > time xz --lzma2 testfile
>  > time unxz testfile.xz
>  >
>  > time xz -T4 --lzma2 testfile
>  > time unxz testfile.xz
>  >
>  > This is quite a significant improvement of build time, while
>  > decompression time
>  > and size only slightly increase. If that's a concern, then LZMA2 could
>  > be used
>  > for development only.
>  >
>  > I haven't investigated the details of how to support this in the code
>  > but it
>  > appears to be a simple change, since the LZMA SDK that we use already
>  > supports
>  > LZMA2.
>  >
>  > What do you think?
> 
> "xz -T" works by splitting the input into blocks, and it generates a
> multi-block compressed output.
> 
> Yes, that's correct.
> 
>  > I'm unsure if the current LZMA
> decompressor that runs inside the firmware (= guided section extractor)
> copes with multi-block input.
> 
> I think you're right that it doesn't. But we can make the guided section extractor use that same algorithm(LZMA2) and assign it a different GUID, right?
> 
>