From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) by mx.groups.io with SMTP id smtpd.web08.2248.1606880210693564846 for ; Tue, 01 Dec 2020 19:36:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=d0i8Caq9; spf=pass (domain: apple.com, ip: 17.171.2.72, mailfrom: afish@apple.com) Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 0B23RPk3056412; Tue, 1 Dec 2020 19:36:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=tl89ulPeKs2HerdqVwHQv/nizJuML1A9ZD/NUNCR+V4=; b=d0i8Caq9J2GNXEbkq6nN2gQ+C0vMm1LRadD8KSRpE9Wy5AdVzNRtkASlyEyrBu/tASIS MwV7PdVdnao2niFHGvSsCTGLS10Ub6d2h2Fy2W5rP+ZAzzGMDhfNn/622ULoYhHouBLI XQgpi+ULhJWWq22SN7y+U345VNV0OZDs3+YuLUsnFuFOiYHiu7t1C0AHkAQ4KSyrF6T9 6spsH/S8dQahWTuP1gNGVgAr7Lq0gDc9WRvZ+E8D5LnhAC7WfPqHvFtSIn5yhjoQqONk gLNL/jNtGKDGOKSp1qREqva8MAckYBMEcxc1zWsygYjiP/zv60ZNJ6m5oISksnGvTt7w ug== Received: from rn-mailsvcp-mta-lapp03.rno.apple.com (rn-mailsvcp-mta-lapp03.rno.apple.com [10.225.203.151]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 353pe0dp23-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 01 Dec 2020 19:36:49 -0800 Received: from rn-mailsvcp-mmp-lapp01.rno.apple.com (rn-mailsvcp-mmp-lapp01.rno.apple.com [17.179.253.14]) by rn-mailsvcp-mta-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPS id <0QKP0102D0PCDGE0@rn-mailsvcp-mta-lapp03.rno.apple.com>; Tue, 01 Dec 2020 19:36:48 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp01.rno.apple.com by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) id <0QKP00I0007P5P00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Tue, 01 Dec 2020 19:36:48 -0800 (PST) X-Va-A: X-Va-T-CD: c64572409350aadfe9dd0c1fae934561 X-Va-E-CD: b95c40ed1c3291306cea340e88f6f8a0 X-Va-R-CD: 02061f8b227aca91b0c960ea434c49ac X-Va-CD: 0 X-Va-ID: 9f1e130b-8f9e-4151-a0aa-1248d73eb6e6 X-V-A: X-V-T-CD: c64572409350aadfe9dd0c1fae934561 X-V-E-CD: b95c40ed1c3291306cea340e88f6f8a0 X-V-R-CD: 02061f8b227aca91b0c960ea434c49ac X-V-CD: 0 X-V-ID: 35301130-fff3-415f-9533-606add6a34dc X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-01_12:2020-11-30,2020-12-01 signatures=0 Received: from [17.235.42.194] (unknown [17.235.42.194]) by rn-mailsvcp-mmp-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPSA id <0QKP00PG00PB5L00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Tue, 01 Dec 2020 19:36:48 -0800 (PST) From: "Andrew Fish" Message-id: <097F435E-840E-4A20-AF20-8D2A4CC125F2@apple.com> MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.1\)) Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 Date: Tue, 01 Dec 2020 19:36:46 -0800 In-reply-to: Cc: derek.lin2@hpe.com To: devel@edk2.groups.io, daniel.schaefer@hpe.com References: X-Mailer: Apple Mail (2.3654.20.0.2.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-01_12:2020-11-30,2020-12-01 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_6A6CF049-AA3E-40EB-BFFA-51D1F4AF8FCE" --Apple-Mail=_6A6CF049-AA3E-40EB-BFFA-51D1F4AF8FCE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Dec 1, 2020, at 6:59 PM, Daniel Schaefer wr= ote: >=20 > Hi everyone, >=20 > I'm looking into how to speed up the build process and noticed that our = build > uses LZMA to encrypt the main firmware volume. Since it's quite big it t= akes a > while but only uses one CPU thread. >=20 > LZMA2 is a version of LZMA which can be multi-threaded and achieve much = faster > compression times. I did a quick benchmark using the `xz` command-line t= ool, > which uses a modified version of the LZMA SDK that EDK2 uses. The result= s are: >=20 > Uncompressed size: 64M >=20 > | Algo | Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA | 19.67s | 0.9s | 9.1M | 1 | > | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | > | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | >=20 > Using those commands: >=20 > time xz --format=3Dlzma testfile > time unlzma testfile.lzma >=20 > time xz --lzma2 testfile > time unxz testfile.xz >=20 > time xz -T4 --lzma2 testfile > time unxz testfile.xz >=20 > This is quite a significant improvement of build time, while decompressi= on time > and size only slightly increase. If that's a concern, then LZMA2 could b= e used > for development only. >=20 > I haven't investigated the details of how to support this in the code bu= t it > appears to be a simple change, since the LZMA SDK that we use already su= pports > LZMA2. >=20 > What do you think? >=20 Interesting idea. What OS did you use? I tried this on macOS on some large= r FVs and I did not see much difference? I tried a 17.5 MiB FV and it was a= round 3 seconds both ways.=20 Maybe it would be worth while seeing how it works on various systems? I gu= ess it might be data set related?=20 Thanks, Andrew Fish > Thanks, > Daniel >=20 >=20 >=20 --Apple-Mail=_6A6CF049-AA3E-40EB-BFFA-51D1F4AF8FCE Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On Dec 1,= 2020, at 6:59 PM, Daniel Schaefer <daniel.schaefer@hpe.com> wrote:

Hi everyone,

I'm looking into how to speed up the= build process and noticed that our build
uses LZMA to encrypt the main firmware volume. Since it'= s quite big it takes a
= while but only uses one CPU thread.

LZMA2 is a version of = LZMA which can be multi-threaded and achieve much faster
compression times. I did a quick benchmar= k using the `xz` command-line tool,
which uses a modified version of the LZMA SDK that EDK2 uses.= The results are:

Uncompressed size: 64M

| = Algo  | Comp Time | Decomp Time | Size | Threads |
| ----- | --------- | ----------- | ---- |= ------- |
| LZMA  = ;|    19.67s |        0.9= s | 9.1M |       1 |
| LZMA2 |    20.11s |  &nbs= p;     1.2s | 9.2M |      = ; 1 |
| LZMA2 | &n= bsp;   8.31s |        1.0= s | 9.4M |       4 |

Using th= ose commands:

time xz --format=3Dlzma testfile
time unlzma testfile.lzma
= time xz --lzma2 testfile
time unxz testfile.xz

time xz -T4 --lzma2 testfile=
time unxz testfile.xz<= /span>

This is quite a significant improvement of build time, whil= e decompression time
an= d size only slightly increase. If that's a concern, then LZMA2 could be use= d
for development only.=

I haven't investigated the details of how to support this = in the code but it
app= ears to be a simple change, since the LZMA SDK that we use already supports=
LZMA2.

What do you think?


Interesting idea. What OS did you use? I tried this on macOS on some = larger FVs and I did not see much difference? I tried a 17.5 MiB FV and it = was around 3 seconds both ways. 

M= aybe it would be worth while seeing how it works on various systems? I gues= s it might be data set related? 

T= hanks,

Andrew Fish

=
Thanks,<= br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 1= 2px; font-style: normal; font-variant-caps: normal; font-weight: normal; le= tter-spacing: normal; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0p= x; text-decoration: none;" class=3D"">Daniel



--Apple-Mail=_6A6CF049-AA3E-40EB-BFFA-51D1F4AF8FCE--