From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (NAM12-BN8-obe.outbound.protection.outlook.com [40.107.237.109]) by mx.groups.io with SMTP id smtpd.web08.12269.1607011063910435282 for ; Thu, 03 Dec 2020 07:57:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=h983HvK9; spf=pass (domain: microsoft.com, ip: 40.107.237.109, mailfrom: bret.barkelew@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EzsVkSdjr4IGyrN9p2L6adzjxsRSuhDKkH1Yl6/+HBZxgKRKBVjurRnlqLZnFcBJPNI+leg2Fl5VEpOMCGILYJ8kIcXLDTmv/IENsEeZKRZJdcEeFug9SToCIxWRFEjalkyGMN74YwJUQTpm62r8cpbmH2uB8Kg6kuCPROgvOrE3VzgTu05Wq5WheNGAedKzLXZESiqog1uFHY1+DfBmfOF7+mJbGgEmQhjMfUTFU1JNCrzxBPLDoW8c504ibgbm+8YDd8fdmSS2x4dxX2pgrOTRLlIdllrOaAlJMZuqEcCLbtbYeXFHAUHR0rgHqHJHYrc5e5KmPBXg0Nx2N5G6ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2RS6Yjzn9Do78h6Uf2FzEgjEYrubC8Xa3pGRmzQP5k=; b=QX7FoiPVh8bciBYIYbq2Lz+Hg6VNSXF9PCg9QsAUair/gC6/U9hL2sKOX5DE0LGrnLP5V6UKL8YPahMcAQ8Z1+VwYX+lw8zEb2xqTSkQPiDW5jFvu84qjB8L/nb0ZQZS7N/QTr2m+j5zcpuk15u5xohkZ9VT1dVGJBWaIR0TlxUnbC385k3D9tGHyFZC9+GuB4S5BjrUZhrFeNVWMQhRRHB0GP2CwEFzdQy/aZMYS2xgiWny70cBZPhAOj0OHliuzgMr63spMy3QtqZwoglUHpTZ84pEtzmdtHbA4ei9daQTXEEUy59ioEmAOwg0pRkALrkWSSNwjLNQc3DxyROTVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2RS6Yjzn9Do78h6Uf2FzEgjEYrubC8Xa3pGRmzQP5k=; b=h983HvK9pBf7z7WDvH2ttAPl7PF1frEA13yEqKOkS8mNNOYHGTd3muQ2aZtQ4EVGi8bTu95tw72H/M+d2aPcZc5CQSPDb9aXUMDVv7dlfqYOA0B/JStBEtJFIInU66j1acygJl/8K0ynvTDGazAh6Qn04nL75LFP6prWAzo7Hac= Received: from (2603:10b6:302:10::22) by MW2PR2101MB0892.namprd21.prod.outlook.com (2603:10b6:302:10::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.2; Thu, 3 Dec 2020 15:57:41 +0000 Received: from MW2PR2101MB0890.namprd21.prod.outlook.com ([fe80::a807:8b5b:a667:d7c2]) by MW2PR2101MB0890.namprd21.prod.outlook.com ([fe80::a807:8b5b:a667:d7c2%4]) with mapi id 15.20.3654.005; Thu, 3 Dec 2020 15:57:41 +0000 From: "Bret Barkelew" To: "devel@edk2.groups.io" , "daniel.schaefer@hpe.com" , Laszlo Ersek CC: "Lin, Derek (HPS SW)" Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 Thread-Topic: [edk2-devel] Multithreaded compression with LZMA2 Thread-Index: AQHWyW2L2W9H7K3UpkmtT6uQSnmngKnlhwcx Date: Thu, 3 Dec 2020 15:57:41 +0000 Message-ID: References: ,<9ebc277a-4acd-f567-cc08-fa076e4a5766@redhat.com>, In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-12-03T15:57:03.3583050Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Privileged authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=microsoft.com; x-originating-ip: [71.212.128.184] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: a7d6a81e-e5b9-42e9-3605-08d897a42aa6 x-ms-traffictypediagnostic: MW2PR2101MB0892: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5516; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UzwxnGWRhJwEyzDbYom2Op29CnAkg2KXY1Qa0452xezLGDWVU4e19jdovZ3ZqD2z3KWL/RB1GDt9HmDnEH7KfLuSKfBie63wOiEWulY7oDQi6xyJCdNO1vxoYk9ZmZPralHZrqQGtNfuseFj/XzUkha0USLoumtYyN9xoPECk1V2xIaCG1LWRnALSjtaxI9imP6s/XR1WDzgePeleK0dnxS3bAbXxX3X5P6zBqEQLC7Y4tBxszTl99gwxeS0uMnSiR+Au5MrOcykcIBFwoif0+uVffbLn4LMrZ5rVlWdT76cQfQqPd8F0eKfZmD3lmZ2KC4mayTpfQInRk3gLFGoBM1drM2gY/kXkyHY2k5Zvk3293S1Avld0WfCKyLA6B3q+U3+/T57XGOKfnAB4hZiVg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR2101MB0890.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(346002)(396003)(376002)(39860400002)(366004)(2906002)(83380400001)(7696005)(10290500003)(8936002)(4326008)(8990500004)(186003)(53546011)(33656002)(166002)(76116006)(26005)(66446008)(6506007)(66616009)(99936003)(66946007)(64756008)(66556008)(316002)(9686003)(82960400001)(5660300002)(478600001)(66476007)(82950400001)(76236003)(8676002)(52536014)(71200400001)(110136005)(86362001)(55016002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?aNgiswdr/AFeDD5pRa0xDPESexNPJymCw0gcvQnuAH2wOgUsUXSBvybx?= =?Windows-1252?Q?ZeYZWxtvGVnwxsB/aW1j0qoPnKGdLi7nZzqv9DO21jQt/Mb1IyPmjdpj?= =?Windows-1252?Q?rHzodao384kGvn3ZaWICZKOtNCHzWM35J01GklM60X9yG5K13oBRl8GH?= =?Windows-1252?Q?uEM3s5eCkX5DjJLVi+4nga3OYxJucErSzGF/hSNBLvQsZ5Rjf7yaC3Nv?= =?Windows-1252?Q?pM88EU068Babmol9y3xTyEJQuf+M5lYLCHC6bBcqXI8lQm0DGgk9NNLb?= =?Windows-1252?Q?MqMxxAXMO7jWFkcX07vPCSmAfg8e+fNQ1YkQgSTYsJYFduFQ6HeAxVHD?= =?Windows-1252?Q?m+vMhkcxN82vHH8IOAVzE1K/ZnsGGkEYe6KD1h00hOUs92iwHNhq+whp?= =?Windows-1252?Q?rnQxUFmZMWYVOlC2V+jXVTwlm6TnZBq607zIRxzPI8jpBvHO8bsII6zw?= =?Windows-1252?Q?qxcIX1Z1yPIG2waMi6fX4HOrg+csJUuo5gOkrKHnGuiGV8WfquUSVDXL?= =?Windows-1252?Q?QFxXap5SL/W5wrQkkOTDBw1zDHCRWKgd3TzuXwX1nOFdW8m4/grKgks+?= =?Windows-1252?Q?W+6RwbE4QGp9/z1J+4cS+PkWNKkENpHPpDaDY1caOjNwL2UvS3BNcHa2?= =?Windows-1252?Q?/9/LqFN3KH6XiB1o7ddVF2pUtngaOIVlCWlnq6QJmrVMdkerEP4vlmX+?= =?Windows-1252?Q?2EoWXaa3b/96frGSzGRH3ZZq4lPIDS/OoWxwLwjNLUD7MjuE5jTmpSTW?= =?Windows-1252?Q?KxQ2URnU3gtkJ1ZqNnk+MCK1QEtnpLqlN7RG76vt7YI05IDoxm8KBI9H?= =?Windows-1252?Q?tdyGQjdYoit2gABiPj8YgUuhZrk2HFyb2QG6gyFD01KaM7FTiXMWRhyx?= =?Windows-1252?Q?muCFecI6QPCvbTw6QFnYW+u+JMZDIWLaN9IiFdAfA9XUlQ+lQjZMkwKH?= =?Windows-1252?Q?MsMdU8KKsy5+maHwCxqUw6qv9LkKiOV5zS3or2nwxjbO5d1k0ybOIQEC?= =?Windows-1252?Q?l4/D0+QZbo7oSbxiOofZIUFLJ4mCeBlr2HQ4HA3Uj71ZmXb6VUfIYaXj?= =?Windows-1252?Q?6bX5UshQBTtvN8G1SWrKOoXx3Rsrv0IpjIpl6w=3D=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW2PR2101MB0890.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a7d6a81e-e5b9-42e9-3605-08d897a42aa6 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 15:57:41.7342 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: QRVQEPxLQWtw1en7fDsK1hMSousK8iIT5fclEQBFw8a2HwBL3GLCDZW7MTOOlFtbdqdqVRSW3iZlSsLAEeXB/A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR2101MB0892 X-Groupsio-MsgNum: 68254 Content-Language: en-US Content-Type: multipart/related; boundary="_004_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_"; type="multipart/alternative" --_004_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_ Content-Type: multipart/alternative; boundary="_000_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_" --_000_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Wasn=92t there another push (somewhere in the last 8 months, my brain is fo= ggy) to adopt LZMA2? Or was it a different algorithm? - Bret From: Daniel Schaefer via groups.io Sent: Thursday, December 3, 2020 4:12 AM To: Laszlo Ersek; devel@edk2.groups.io Cc: Lin, Derek (HPS SW) Subject: [EXTERNAL] Re: [edk2-devel] Multithreaded compression with LZMA2 From: Laszlo Ersek Sent: Thursday, December 3, 2020 18:24 To: devel@edk2.groups.io ; Schaefer, Daniel Cc: Lin, Derek (HPS SW) Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 On 12/02/20 03:59, Daniel Schaefer wrote: > Hi everyone, > > I'm looking into how to speed up the build process and noticed that our > build > uses LZMA to encrypt the main firmware volume. Since it's quite big it > takes a > while but only uses one CPU thread. > > LZMA2 is a version of LZMA which can be multi-threaded and achieve much > faster > compression times. I did a quick benchmark using the `xz` command-line > tool, > which uses a modified version of the LZMA SDK that EDK2 uses. The > results are: > > Uncompressed size: 64M > > | Algo | Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA | 19.67s | 0.9s | 9.1M | 1 | > | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | > | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | > > Using those commands: > > time xz --format=3Dlzma testfile > time unlzma testfile.lzma > > time xz --lzma2 testfile > time unxz testfile.xz > > time xz -T4 --lzma2 testfile > time unxz testfile.xz > > This is quite a significant improvement of build time, while > decompression time > and size only slightly increase. If that's a concern, then LZMA2 could > be used > for development only. > > I haven't investigated the details of how to support this in the code > but it > appears to be a simple change, since the LZMA SDK that we use already > supports > LZMA2. > > What do you think? "xz -T" works by splitting the input into blocks, and it generates a multi-block compressed output. Yes, that's correct. > I'm unsure if the current LZMA decompressor that runs inside the firmware (=3D guided section extractor) copes with multi-block input. I think you're right that it doesn't. But we can make the guided section e= xtractor use that same algorithm(LZMA2) and assign it a different GUID, rig= ht? --_000_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Wasn=92t there another push (somewhere in the last = 8 months, my brain is foggy) to adopt LZMA2? Or was it a different algorith= m?

 

- Bret

 

From: Daniel Schaefer via grou= ps.io
Sent: Thursday, December 3, 2020 4:12 AM
To: Laszlo Ersek; devel@edk2.groups.io
Cc: Lin, Derek (HPS SW) Subject: [EXTERNAL] Re: [edk2-devel] Multithreaded compression with= LZMA2

 

 

From: Laszlo Ersek <lersek@redhat.com>=

Sent: Thursday, December 3, 2020 18:24
To: devel@edk2.groups.io <devel@edk2.groups.io>; Schaefer, Da= niel <daniel.schaefer@hpe.com>
Cc: Lin, Derek (HPS SW) <derek.lin2@hpe.com>
Subject: Re: [edk2-devel] Multithreaded compression with LZMA2

 

On 12/02/20 03:59, Daniel Schaefer wrote:
> Hi everyone,
>
> I'm looking into how to speed up the build process and noticed that o= ur
> build
> uses LZMA to encrypt the main firmware volume. Since it's quite big i= t
> takes a
> while but only uses one CPU thread.
>
> LZMA2 is a version of LZMA which can be multi-threaded and achieve mu= ch
> faster
> compression times. I did a quick benchmark using the `xz` command-lin= e
> tool,
> which uses a modified version of the LZMA SDK that EDK2 uses. The
> results are:
>
> Uncompressed size: 64M
>
> | Algo  | Comp Time | Decomp Time | Size | Threads |
> | ----- | --------- | ----------- | ---- | ------- |
> | LZMA  |    19.67s |    &nbs= p;   0.9s | 9.1M |       1 |
> | LZMA2 |    20.11s |     &nb= sp;  1.2s | 9.2M |       1 |
> | LZMA2 |     8.31s |    &nbs= p;   1.0s | 9.4M |       4 |
>
> Using those commands:
>
> time xz --format=3Dlzma testfile
> time unlzma testfile.lzma
>
> time xz --lzma2 testfile
> time unxz testfile.xz
>
> time xz -T4 --lzma2 testfile
> time unxz testfile.xz
>
> This is quite a significant improvement of build time, while
> decompression time
> and size only slightly increase. If that's a concern, then LZMA2 coul= d
> be used
> for development only.
>
> I haven't investigated the details of how to support this in the code=
> but it
> appears to be a simple change, since the LZMA SDK that we use already=
> supports
> LZMA2.
>
> What do you think?

"xz -T" works by splitting the input into blocks, and it generat= es a
multi-block compressed output.

 

Yes, that's correct.

 

> I'm unsure if t= he current LZMA
decompressor that runs inside the firmware (=3D guided section extractor)<= br> copes with multi-block input.

I think you're right that it doesn't. But we can make the guided section e= xtractor use that same algorithm(LZMA2) and assign it a different GUID, rig= ht?

 

--_000_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_-- --_004_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_ Content-Type: image/png; name="69F4FCEBBB92465AA3A93438CD55E1D3.png" Content-Description: 69F4FCEBBB92465AA3A93438CD55E1D3.png Content-Disposition: inline; filename="69F4FCEBBB92465AA3A93438CD55E1D3.png"; size=140; creation-date="Thu, 03 Dec 2020 15:57:41 GMT"; modification-date="Thu, 03 Dec 2020 15:57:41 GMT" Content-ID: Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAsQAAAABCAYAAADZ77itAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAhSURBVEhL7cMBDQAACAMg+5cygQkeRoMIG9WT VVXVn7MHYi5moJeByLMAAAAASUVORK5CYII= --_004_MW2PR2101MB0890493E7FB314D605B341BDEFF21MW2PR2101MB0890_--