Re: 回复: [edk2-devel] Multithreaded compression with LZMA2

["Daniel Schaefer" <daniel.schaefer@hpe.com>]

On 12/4/20 10:28 AM, gaoliming wrote:
> Daniel：
>    Yes. New guided section extractor matches new compression algorithm.

Good. I see that we use version 18.05 of the LZMA SDK, while there's already 19.00:
https://www.7-zip.org/sdk.html

Should we use this opportunity to update?

> For the compression algorithm, its compression ratio, compression performance,
> the decompression performance, the decompression taken memory are all
> required to be considered.

For compression ratio and performance, see my earlier emails. Summary:
Compression ratio is basically the same.
Performance is also the same, except when using 4 threads it compresses our
main image in just 40% of the time. Some images don't compress as well, they
take the same time to compress.

For decompression memory usage I used the xz commands on Linux again:

# LZMA1
$ /usr/bin/time -v unxz testfile.lzma
Maximum resident set size (kbytes): 10228, 10492, 10460, 10200, 10244 => 10324.8

# LZMA2
$ /usr/bin/time -v unxz testfile.xz
Maximum resident set size (kbytes): 10456, 10460, 10224, 10212, 10548 => 10380.0

Result: Basically the same.

I don't know how I would measure this in EDK2. Any ideas?


 From the manpage of xz and LZMA authors:

 > LZMA2  is an updated version of LZMA1 to fix some practical issues of LZMA1.
 > Compression speed and  ratios  of  LZMA1 and LZMA2 are practically the same.
 > LZMA2 is better than LZMA, if you compress already compressed data.

Here's a benchmark which compares both of them, among others:
https://stephane.lesimple.fr/blog/lzop-vs-compress-vs-gzip-vs-bzip2-vs-lzma-vs-lzma2xz-benchmark-reloaded/
Same result: They're basically the same.


> Thanks
> Liming
>> -----邮件原件-----
>> 发件人: bounce+27952+68292+4905953+8761045@groups.io
>> <bounce+27952+68292+4905953+8761045@groups.io> 代表 Laszlo Ersek
>> 发送时间: 2020年12月4日 7:35
>> 收件人: Schaefer, Daniel <daniel.schaefer@hpe.com>; devel@edk2.groups.io
>> 抄送: Lin, Derek (HPS SW) <derek.lin2@hpe.com>
>> 主题: Re: [edk2-devel] Multithreaded compression with LZMA2
>>
>> On 12/03/20 13:11, Schaefer, Daniel wrote:
>>>
>>> From: Laszlo Ersek <lersek@redhat.com>
>>> Sent: Thursday, December 3, 2020 18:24
>>> To: devel@edk2.groups.io <devel@edk2.groups.io>; Schaefer, Daniel
>> <daniel.schaefer@hpe.com>
>>> Cc: Lin, Derek (HPS SW) <derek.lin2@hpe.com>
>>> Subject: Re: [edk2-devel] Multithreaded compression with LZMA2
>>>
>>
>>> "xz -T" works by splitting the input into blocks, and it generates a
>>> multi-block compressed output.
>>>
>>> Yes, that's correct.
>>>
>>>> I'm unsure if the current LZMA
>>> decompressor that runs inside the firmware (= guided section extractor)
>>> copes with multi-block input.
>>>
>>> I think you're right that it doesn't. But we can make the guided section
>> extractor use that same algorithm(LZMA2) and assign it a different GUID,
>> right?
>>
>> I guess so...
>>
>> Thanks
>> Laszlo