From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.byosoft.com.cn (mail.byosoft.com.cn [58.240.74.242]) by mx.groups.io with SMTP id smtpd.web09.619.1606886519921990665 for ; Tue, 01 Dec 2020 21:22:01 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=none, err=permanent DNS error (domain: byosoft.com.cn, ip: 58.240.74.242, mailfrom: gaoliming@byosoft.com.cn) Received: from DESKTOPS6D0PVI ([58.246.60.130]) (envelope-sender ) by 192.168.6.13 with ESMTP for ; Wed, 02 Dec 2020 13:21:44 +0800 X-WM-Sender: gaoliming@byosoft.com.cn X-WM-AuthFlag: YES X-WM-AuthUser: gaoliming@byosoft.com.cn From: "gaoliming" To: , , Cc: References: <097F435E-840E-4A20-AF20-8D2A4CC125F2@apple.com> In-Reply-To: <097F435E-840E-4A20-AF20-8D2A4CC125F2@apple.com> Subject: =?UTF-8?B?5Zue5aSNOiBbZWRrMi1kZXZlbF0gTXVsdGl0aHJlYWRlZCBjb21wcmVzc2lvbiB3aXRoIExaTUEy?= Date: Wed, 2 Dec 2020 13:21:49 +0800 Message-ID: <013801d6c86b$0941a040$1bc4e0c0$@byosoft.com.cn> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQHflGFSZn2WPqT5OCaPh7KljJ49mwK94PI8qbu+DPA= Content-Type: multipart/alternative; boundary="----=_NextPart_000_0139_01D6C8AE.176618C0" Content-Language: zh-cn ------=_NextPart_000_0139_01D6C8AE.176618C0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable Daniel: Can you provide the compressed image size? And, what image is used to be compressed? Is it the generated FV image? =20 Thanks Liming =B7=A2=BC=FE=C8=CB: bounce+27952+68159+4905953+8761045@groups.io =B4=FA=B1=ED Andrew Fish vi= a groups.io =B7=A2=CB=CD=CA=B1=BC=E4: 2020=C4=EA12=D4=C22=C8=D5 11:37 =CA=D5=BC=FE=C8=CB: devel@edk2.groups.io; daniel.schaefer@hpe.com =B3=AD=CB=CD: derek.lin2@hpe.com =D6=F7=CC=E2: Re: [edk2-devel] Multithreaded compression with LZMA2 =20 =20 On Dec 1, 2020, at 6:59 PM, Daniel Schaefer > wrote: =20 Hi everyone, I'm looking into how to speed up the build process and noticed that our build uses LZMA to encrypt the main firmware volume. Since it's quite big it tak= es a while but only uses one CPU thread. LZMA2 is a version of LZMA which can be multi-threaded and achieve much faster compression times. I did a quick benchmark using the `xz` command-line too= l, which uses a modified version of the LZMA SDK that EDK2 uses. The results are: Uncompressed size: 64M | Algo | Comp Time | Decomp Time | Size | Threads | | ----- | --------- | ----------- | ---- | ------- | | LZMA | 19.67s | 0.9s | 9.1M | 1 | | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | Using those commands: time xz --format=3Dlzma testfile time unlzma testfile.lzma time xz --lzma2 testfile time unxz testfile.xz time xz -T4 --lzma2 testfile time unxz testfile.xz This is quite a significant improvement of build time, while decompression time and size only slightly increase. If that's a concern, then LZMA2 could be used for development only. I haven't investigated the details of how to support this in the code but = it appears to be a simple change, since the LZMA SDK that we use already supports LZMA2. What do you think? =20 Interesting idea. What OS did you use? I tried this on macOS on some large= r FVs and I did not see much difference? I tried a 17.5 MiB FV and it was around 3 seconds both ways.=20 =20 Maybe it would be worth while seeing how it works on various systems? I guess it might be data set related?=20 =20 Thanks, =20 Andrew Fish Thanks, Daniel =20 ------=_NextPart_000_0139_01D6C8AE.176618C0 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

Daniel:

Can y= ou provide the compressed image size? And, what image is used to be compres= sed? Is it the generated FV image?

&= nbsp;

Thanks

Liming

=B7=A2=BC=FE=C8=CB:<= /span> bounce+27952+6= 8159+4905953+8761045@groups.io <bounce+27952+68159+4905953+8761045@group= s.io> =B4=FA=B1=ED Andrew Fish via grou= ps.io
=B7=A2=CB=CD=CA=B1= =BC=E4: 2020=C4=EA= 12=D4=C22=C8=D5 11:37
=CA=D5=BC=FE=C8=CB: devel@edk2.groups.io; daniel.schaefer@hpe.com
<= /span>=B3=AD=CB=CD: der= ek.lin2@hpe.com
=D6=F7=CC=E2: Re: [edk2-devel] Multithreaded compression with LZMA2=

&= nbsp;

=

On Dec 1, 2020, at 6:59 = PM, Daniel Schaefer <daniel.s= chaefer@hpe.com> wrote:

Hi everyone,

I'm looking into how to speed up the build proce= ss and noticed that our build
uses LZMA to encrypt the main firmware vol= ume. Since it's quite big it takes a
while but only uses one CPU thread.=

LZMA2 is a version of LZMA which can be multi-threaded and achieve = much faster
compression times. I did a quick benchmark using the `xz` co= mmand-line tool,
which uses a modified version of the LZMA SDK that EDK2= uses. The results are:

Uncompressed size: 64M

| Algo |= Comp Time | Decomp Time | Size | Threads |
| ----- | --------- | ------= ----- | ---- | ------- |
| LZMA |    19.67s | = ;      0.9s | 9.1M |     =   1 |
| LZMA2 |    20.11s |    &= nbsp;   1.2s | 9.2M |       1 = |
| LZMA2 |     8.31s |     &nbs= p;  1.0s | 9.4M |       4 |

= Using those commands:

time xz --format=3Dlzma testfile
time unlzm= a testfile.lzma

time xz --lzma2 testfile
time unxz testfile.xz
time xz -T4 --lzma2 testfile
time unxz testfile.xz

This is q= uite a significant improvement of build time, while decompression time
a= nd size only slightly increase. If that's a concern, then LZMA2 could be us= ed
for development only.

I haven't investigated the details of ho= w to support this in the code but it
appears to be a simple change, sinc= e the LZMA SDK that we use already supports
LZMA2.

What do you th= ink?

Interesting idea. What OS did you use? I t= ried this on macOS on some larger FVs and I did not see much difference? I = tried a 17.5 MiB FV and it was around 3 seconds both ways. =

Maybe it = would be worth while seeing how it works on various systems? I guess it mig= ht be data set related?

Thanks,

Andrew Fish

=

Thanks,
Daniel

&nb= sp;

= ------=_NextPart_000_0139_01D6C8AE.176618C0--