From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by mx.groups.io with SMTP id smtpd.web08.9150.1606997532748249868 for ; Thu, 03 Dec 2020 04:12:12 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=YV3CuARG; spf=pass (domain: hpe.com, ip: 148.163.147.86, mailfrom: prvs=0606a97907=daniel.schaefer@hpe.com) Received: from pps.filterd (m0134420.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B3C3wMZ020222; Thu, 3 Dec 2020 12:12:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=pps0720; bh=UQSzdC4ujvIfC9GJiCnoYLKkbIfBcaari60NwTQT9MM=; b=YV3CuARGvPLzkkEhmamZ77GwAa/9Kd6j5BkpyE02OfgDp08b8iK1pgu3GSBAoDzGN7vR ygMu9dnzlNcRoX/fCi02pX7VBp4iwJeQzp9d/x7DmbrLFTmWhTGkF/2UQVV0im50yp7q AEJxN7TNNFAhdt69usJYhHNu9K1U5DVQYq6Id+N5scbJssZEniIHTQEPdkIsGLMn/ofH DlL+/OpDJLiNi6UzbqW691vfosMoxdIggjw8ZaSsF4IYds5WCgRUJerlJv7l4g//dSfF dyBpr5zDX9tcTJkRVwvZQkq8uyo0HPfW3YSaLLfZKHHfnDGQA2/XQT7yJ6vUXLvAFQGb oA== Received: from g4t3426.houston.hpe.com (g4t3426.houston.hpe.com [15.241.140.75]) by mx0b-002e3701.pphosted.com with ESMTP id 355suqqu7j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 03 Dec 2020 12:12:11 +0000 Received: from G4W9121.americas.hpqcorp.net (exchangepmrr1.us.hpecorp.net [16.210.21.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g4t3426.houston.hpe.com (Postfix) with ESMTPS id E5AFD69; Thu, 3 Dec 2020 12:12:10 +0000 (UTC) Received: from G4W9119.americas.hpqcorp.net (2002:10d2:14d6::10d2:14d6) by G4W9121.americas.hpqcorp.net (2002:10d2:1510::10d2:1510) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 3 Dec 2020 12:11:20 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (15.241.52.10) by G4W9119.americas.hpqcorp.net (16.210.20.214) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Thu, 3 Dec 2020 12:11:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jdb85TBTxpmmfH5i9s89PfrTDTcXzb20xqdNiU+8/uajfi4N3KWNNFzGuW3bh4pZlRWnFCgKyuso9gnbRMQdKVKhkul+n6fqtt6JTYdJxOvUGvORHVK/mHFhVxlshcyyeu/WcKxzuWnSR5Q+UuuKosYU1e1TjaT8MH1Ztb6iH2kHCrro73I5vEAa8EmEFE+/hdr9iabCZ9rmi5h0Tzsrtt4rrR/9blywLnfN+fV/+pcXUysjWQwGxC78y6qSREBwacnLkSrdY8a3lb6+pST7ObLLWSaaG4I68Yn5VOv9cdVy8fqZ2MT1g+9WWd5hAxT0IiSRJwgQYH68Dtix9I+2Rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UQSzdC4ujvIfC9GJiCnoYLKkbIfBcaari60NwTQT9MM=; b=VARw2FxOIGNRzN+BhHRsJ8Ef/Vy+LdZl6TIi8oWPBusE0yEUUxlapveIyPFPxr4lMQnD2g1K5nRkK9cmUMxoEwb9cvlAro0rJbcIZeu1weNgyGdWdovtOYjT/KhCTq6SaN6CaYwBlpEYnMORcQCsluv373Z0NuNd5yk0UhJw8YbjpkSJkDnpWvdmcfdBZOheAtkvg1N7Fup9wwz8vNUS6IlWXHXg8N32tL9O55/+0JEQSuRSg+Gyo2jPYDwAGAoif0c/0aZGMuYdyJPb4EeG6ZDUP2x0ZID8KIzLuDU42WcQ22eAlorfw9+5Z/dopuJOT9BrnNwoctAjy76BV7kXig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hpe.com; dmarc=pass action=none header.from=hpe.com; dkim=pass header.d=hpe.com; arc=none Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) by DF4PR8401MB0460.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:7607::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.18; Thu, 3 Dec 2020 12:11:19 +0000 Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f]) by DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f%8]) with mapi id 15.20.3632.019; Thu, 3 Dec 2020 12:11:19 +0000 From: "Daniel Schaefer" To: Laszlo Ersek , "devel@edk2.groups.io" CC: "Lin, Derek (HPS SW)" Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 Thread-Topic: [edk2-devel] Multithreaded compression with LZMA2 Thread-Index: AQHWyFc4Cw2uAxBBUUOr13251yGZ56nlLGsAgAAcJtw= Date: Thu, 3 Dec 2020 12:11:19 +0000 Message-ID: References: ,<9ebc277a-4acd-f567-cc08-fa076e4a5766@redhat.com> In-Reply-To: <9ebc277a-4acd-f567-cc08-fa076e4a5766@redhat.com> Accept-Language: en-US, de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=hpe.com; x-originating-ip: [1.200.166.254] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 4d6058e9-7a61-4c13-14ee-08d897848ad0 x-ms-traffictypediagnostic: DF4PR8401MB0460: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7691; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lw9F+aF5pDVszkbafYSnycP10KAzMK56Uy485CUTvI2+mMuIv06xmMm3qKacx9jgnr+CoLsxYYXJI7Ihbb84pY3dVb/td5gFqxLsp14+saGT9lijIfyA4rHblzYJJKnAbgNwJnneFBZvTJMOan7FkoDVcU22sNX+UdmJ/CzZ9Wc0wlgTevYcD3oo/d9OiUetLU7JFHtNYrADMZO+tZAdEXSYkwXkVvdlYGoPQSWpEAurd3Mi0ji5m4Ks/5IPdweGQfi1dGuoL4M5Gf30lJN8SgtoLcJOOJMpormKi5mmFt8jEG06vtTipRpLLPV7iOouuVhEAq5k+XWdaMd4DjlLxA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(396003)(136003)(346002)(376002)(366004)(39860400002)(86362001)(33656002)(71200400001)(5660300002)(66556008)(91956017)(66446008)(66946007)(64756008)(66476007)(76116006)(26005)(6506007)(8676002)(186003)(53546011)(8936002)(110136005)(4326008)(316002)(7696005)(2906002)(52536014)(9686003)(55016002)(83380400001)(478600001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?H973O2lgI7+R4EVBSDdeB8u+sPk/aRBcgzzslgLKg5tXyd6UUU8OYZQHSk?= =?iso-8859-1?Q?eNvB7EVxpPdKZA5+DP1RoYFS/r02gVNeAgBuBldRgdaXwKhrdaunS5VyF8?= =?iso-8859-1?Q?7N1jPBzkUSlMnyllkaKsV2pv7yfyZEwO+FUIfQUNznfSXmeUS/qwSQFE5Z?= =?iso-8859-1?Q?KvXDu4nBnhmqsmw4hK/JPZ8exZ1gVqt0BCnP/lu6ZN61NiJ02U8+sBRf4E?= =?iso-8859-1?Q?uAuhywgUqCteKgEC7KJLGhOSIlYazJS5V9r3qpbPoWMOnZOTI7iHrSIf24?= =?iso-8859-1?Q?3uaEoLxWJdztM2CDNZxv6HOA7r0urnatR+PkXQfapSPcVngpcDHO3OM42G?= =?iso-8859-1?Q?UWFTKDWbT1Yjeo02vxy5ps/Op/SGTg3bCnUCqlQ8zPVLCWzdSgfRNiO0Ax?= =?iso-8859-1?Q?6eKrf8237TfOaAIFQi8/NOz81z25m8ckwmEMizvdngownEftL6exGFlY3l?= =?iso-8859-1?Q?tShE1UvPtotlh+H/kkoaQ4i5LWlABAutjIh9lMtCJ9IQRI/JR9zkYCSFvg?= =?iso-8859-1?Q?O0F7J/xB0BDEFdizVaim0xNVPQJyUGbzkvW8Ud7hbLz7sGqMQdkskg1cUO?= =?iso-8859-1?Q?xbxgqIKh2c8vbJvKTQr/jnxKHDMzimYWg1byrmZrzcwjxML8LhpgerJkJy?= =?iso-8859-1?Q?Q/C61BCYyFK10zwCzGZp+HoAhvNm1eOsBVO5thcJW6uXK2R3VJJ4MFyLDv?= =?iso-8859-1?Q?SgMz3FFdvXE9uXyMpMDqnhKzwRQB6J9oyhIpLx5ZL1Bn3UwA1l8T34rQsD?= =?iso-8859-1?Q?2UNiyR+bvNr464yQlboDY5Aaeio0EL64XOs4uFi+BFDzg9A4fHJ4EVtA2n?= =?iso-8859-1?Q?zx63upDtjyW8G4k/1HEt/ZNLiJU6aK3ynXG91V4O+Re7IGpuQDeG6GhKOt?= =?iso-8859-1?Q?gG3YqyllZZy53bH+KKFH+QeXKeHPueDr5teGuLUtuSP3DzGNWq7mwI6L9C?= =?iso-8859-1?Q?f2rUGr30L6nlwxqBQp1t2MycvnF9g4GoeZkRBSE0pkvUCr8eZ2tZrSwFe1?= =?iso-8859-1?Q?JqEfhEGzj6SBqliv4=3D?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 4d6058e9-7a61-4c13-14ee-08d897848ad0 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 12:11:19.2395 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 71NjEegWp+VAK1SM+IWKI96ZEs9YRc061GUZr4lf6jsyfQs8thzwHW7C33r4SLUJl0xGfvhCEZ+W9NQi7F24JA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR8401MB0460 X-OriginatorOrg: hpe.com X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-03_06:2020-12-03,2020-12-03 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 malwarescore=0 priorityscore=1501 lowpriorityscore=0 impostorscore=0 spamscore=0 adultscore=0 clxscore=1015 mlxscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012030075 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_DF4PR8401MB04449394D5540D7E922BED40E0F20DF4PR8401MB0444_" --_000_DF4PR8401MB04449394D5540D7E922BED40E0F20DF4PR8401MB0444_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable From: Laszlo Ersek Sent: Thursday, December 3, 2020 18:24 To: devel@edk2.groups.io ; Schaefer, Daniel Cc: Lin, Derek (HPS SW) Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 On 12/02/20 03:59, Daniel Schaefer wrote: > Hi everyone, > > I'm looking into how to speed up the build process and noticed that our > build > uses LZMA to encrypt the main firmware volume. Since it's quite big it > takes a > while but only uses one CPU thread. > > LZMA2 is a version of LZMA which can be multi-threaded and achieve much > faster > compression times. I did a quick benchmark using the `xz` command-line > tool, > which uses a modified version of the LZMA SDK that EDK2 uses. The > results are: > > Uncompressed size: 64M > > | Algo | Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA | 19.67s | 0.9s | 9.1M | 1 | > | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | > | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | > > Using those commands: > > time xz --format=3Dlzma testfile > time unlzma testfile.lzma > > time xz --lzma2 testfile > time unxz testfile.xz > > time xz -T4 --lzma2 testfile > time unxz testfile.xz > > This is quite a significant improvement of build time, while > decompression time > and size only slightly increase. If that's a concern, then LZMA2 could > be used > for development only. > > I haven't investigated the details of how to support this in the code > but it > appears to be a simple change, since the LZMA SDK that we use already > supports > LZMA2. > > What do you think? "xz -T" works by splitting the input into blocks, and it generates a multi-block compressed output. Yes, that's correct. > I'm unsure if the current LZMA decompressor that runs inside the firmware (=3D guided section extractor) copes with multi-block input. I think you're right that it doesn't. But we can make the guided section ex= tractor use that same algorithm(LZMA2) and assign it a different GUID, righ= t? --_000_DF4PR8401MB04449394D5540D7E922BED40E0F20DF4PR8401MB0444_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

From:Sent: Thursday, December 3,= 2020 18:24
To: devel@edk2.groups.io <devel@edk2.groups.io>; Schaefer, Dan= iel <daniel.schaefer@hpe.com>
Cc: Lin, Derek (HPS SW) <derek.lin2@hpe.com>
Subject: Re: [edk2-devel] Multithreaded compression with LZMA2
 
On 12/02/20 03:59, Daniel Schaefer wrote:
> Hi everyone,
>
> I'm looking into how to speed up the build process and noticed that ou= r
> build
> uses LZMA to encrypt the main firmware volume. Since it's quite big it=
> takes a
> while but only uses one CPU thread.
>
> LZMA2 is a version of LZMA which can be multi-threaded and achieve muc= h
> faster
> compression times. I did a quick benchmark using the `xz` command-line=
> tool,
> which uses a modified version of the LZMA SDK that EDK2 uses. The
> results are:
>
> Uncompressed size: 64M
>
> | Algo  | Comp Time | Decomp Time | Size | Threads |
> | ----- | --------- | ----------- | ---- | ------- |
> | LZMA  |    19.67s |     = ;   0.9s | 9.1M |       1 |
> | LZMA2 |    20.11s |     &nbs= p;  1.2s | 9.2M |       1 |
> | LZMA2 |     8.31s |     = ;   1.0s | 9.4M |       4 |
>
> Using those commands:
>
> time xz --format=3Dlzma testfile
> time unlzma testfile.lzma
>
> time xz --lzma2 testfile
> time unxz testfile.xz
>
> time xz -T4 --lzma2 testfile
> time unxz testfile.xz
>
> This is quite a significant improvement of build time, while
> decompression time
> and size only slightly increase. If that's a concern, then LZMA2 could=
> be used
> for development only.
>
> I haven't investigated the details of how to support this in the code<= br> > but it
> appears to be a simple change, since the LZMA SDK that we use already<= br> > supports
> LZMA2.
>
> What do you think?

"xz -T" works by splitting the input into blocks, and it generate= s a
multi-block compressed output.

Yes, that's correct.

> I'm unsure if the current LZMA
decompressor that runs inside the firmware (=3D guided section extractor) copes with multi-block input.

I think you're right that it doesn't. But we can make the guided section ex= tractor use that same algorithm(LZMA2) and assign it a different GUID, righ= t?

--_000_DF4PR8401MB04449394D5540D7E922BED40E0F20DF4PR8401MB0444_--