From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by mx.groups.io with SMTP id smtpd.web08.11520.1607070040086453910 for ; Fri, 04 Dec 2020 00:20:40 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@hpe.com header.s=pps0720 header.b=YE7OwFdb; spf=pass (domain: hpe.com, ip: 148.163.143.35, mailfrom: prvs=0607174a9d=daniel.schaefer@hpe.com) Received: from pps.filterd (m0150244.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B48Iig6003890; Fri, 4 Dec 2020 08:20:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=subject : to : cc : references : from : message-id : date : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pps0720; bh=8rGUI6/+YtGOEa5owgfOg+Ol0BuGy/XvT5SzF2JG6ZM=; b=YE7OwFdb7bWBwnCmEdxI/yfdWPAKghRy6niw3rRHACXwXECeGMhAEu5G1TB0KBJtbVWB YwJJQXzUPO9sS9kA0rARrVEin8D2mOneacMmdF4WfDyR3fdsuid0jIcEbjh6iPSB0Nej 0rS8SHzRzvOUTWHj9rif7ba5Oyhfb7R0nmzcwyfyL6vEMRTZ3UFVQvnZZNmz0oMFFn5n dWuxh3WkbKcPNijN9ZOJnp32firilMlyrKE2oMkVU/U5Y92rXdzkkMy+mSwLyGhvk4QL 9S3lHpUTKkxD1ix0cCsvuIZ00GfZIc9mDxVdI68+ygUi5/kps1DF/4SlTaFCArR30iX3 HA== Received: from g2t2352.austin.hpe.com (g2t2352.austin.hpe.com [15.233.44.25]) by mx0b-002e3701.pphosted.com with ESMTP id 3579vr2s87-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 04 Dec 2020 08:20:38 +0000 Received: from G4W9120.americas.hpqcorp.net (exchangepmrr1.us.hpecorp.net [16.210.21.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g2t2352.austin.hpe.com (Postfix) with ESMTPS id 15395A1; Fri, 4 Dec 2020 08:20:38 +0000 (UTC) Received: from G4W9119.americas.hpqcorp.net (2002:10d2:14d6::10d2:14d6) by G4W9120.americas.hpqcorp.net (2002:10d2:150f::10d2:150f) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 4 Dec 2020 08:19:57 +0000 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (15.241.52.10) by G4W9119.americas.hpqcorp.net (16.210.20.214) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 4 Dec 2020 08:19:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fV1xgVzGxljWYSVs4hes5RycGqKfUyL2Ma5gO6gQhgTSzMf1DhLqqi8g1j30fquLiiWvox2CAR3xw++9W3pZngwsUfflKMpVOxtJGdUPpZwzsFGUOYmQf8LJl35RCjwKdu2Wmx4sAPwE9kI/A8bK73rbMDAhoLjHle0JS+/HE8MyZ2Sc1EXq2nPjCl+Zb+6Hsq6vS8OsyBnQadSfAbACaFiUeEm7zrAX9Z7hL88G1PIQUb40pLSR/aX9JiHuD86lwj7n2yIL/hoaZbA6N9oI5vCyH86SWgJ5yMvqjHCAgjd7hG61virCDMm/LURO6fqzs9uviXrlgsTmEG8+iyvu/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ja1Xv8Sy2rk451P5QEgcp7Y1g+O8gqUl3vkAce630r8=; b=bajNT40xADJIFHErccslDnFpkqe2ZcuH5ADSxp4zrhERks1eF29EalYttGeRmvzEX/rjhX9AScgGvI9XG74r7aXFVtE0BNY75OK8KEFu3zuwdCQepX6G7FCr89OLQzsQBiTPDI73pvJ6hzf7QOiiCTgt+ivbjsWZnlMJPkG5cdBzZMizA+sJWjnj2AdTwUgPsoKttu7qqpI5d3+JFZcpGUmVE8YA2y8p8+TKsCy00G3wldNBqPA1MbvcCkpn+s8oQxrg9zjcoljswfo/26t9prWfoGUzALw/8ptKQVChTd6SdgsRTv+oqm1SwNwB0ChOIyUVSkQ8rcFdR4FYo8KIsA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hpe.com; dmarc=pass action=none header.from=hpe.com; dkim=pass header.d=hpe.com; arc=none Authentication-Results: hpe.com; dkim=none (message not signed) header.d=none;hpe.com; dmarc=none action=none header.from=hpe.com; Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) by DF4PR8401MB1242.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:7610::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.25; Fri, 4 Dec 2020 08:19:56 +0000 Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f]) by DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f%8]) with mapi id 15.20.3632.021; Fri, 4 Dec 2020 08:19:56 +0000 Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 To: , , Laszlo Ersek CC: "Lin, Derek (HPS SW)" References: <9ebc277a-4acd-f567-cc08-fa076e4a5766@redhat.com> From: "Daniel Schaefer" Message-ID: <1d4b21e7-859e-45cc-4ab7-ad5ca3351a8e@hpe.com> Date: Fri, 4 Dec 2020 16:19:41 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 In-Reply-To: X-Originating-IP: [15.211.146.34] X-ClientProxiedBy: SG2PR04CA0142.apcprd04.prod.outlook.com (2603:1096:3:16::26) To DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [10.10.6.166] (15.211.146.34) by SG2PR04CA0142.apcprd04.prod.outlook.com (2603:1096:3:16::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.17 via Frontend Transport; Fri, 4 Dec 2020 08:19:54 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: ac8cfd8f-570d-4248-b52d-08d8982d622d X-MS-TrafficTypeDiagnostic: DF4PR8401MB1242: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:317; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BOcffq12UY++ZDUZVoA1SX3vsDrs34u22VDdIrPpuIhgCrAKDldaCS5mMMLuK3T8TETtJVRVBqWFS4c8xpFtydhD292kkCCyjmsZYz884zSDx1+/9EmzV44BGNoeV5KpaQFwnCYajI/UKYckfHYE15Rt+zyFupl2c2rxC4J0H/jB8sEDegwOkxzeKmaUdDOK9hm72yzj88Nvgc8Za752FOseeSJWnaIN9paXlaiB+ruVtP+F1MSCGkmu0SdzCNjNk2AfKlQgezNo6khDn0s8iNeCz3JIVDY2RgBojtU3MIsjNj0CcMe4J13SP8w0rwP+OIjEryKl5g6wGBqXj/nRL3SbxSDFiz1XpFkhh8V6h4lHKBTTO0PhmvDlXBM5zUKT03uS4br8/03g0pRnfRplNJ78ujbv6y4Kp4/CGX67PRQY8Rw+X6vu/rmdnBQtf7VeRfbtIB8G2qx4kv3E6h64rj/VTGqZmHMhIlHdpwUmE3E= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(136003)(396003)(366004)(39860400002)(376002)(346002)(16526019)(316002)(44832011)(31686004)(6916009)(2906002)(2616005)(8676002)(52116002)(5660300002)(4326008)(6666004)(86362001)(8936002)(66556008)(966005)(31696002)(53546011)(956004)(36756003)(478600001)(186003)(16576012)(83380400001)(26005)(6486002)(66476007)(66946007)(43740500002)(45980500001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?Windows-1252?Q?GhuOUhFsY9Qpl86CxQ3URCE9H9EezEjQ8MwbIo+MjP+DLzmS1DNj7egS?= =?Windows-1252?Q?dwsRjQiVvJVlTrKJAipxJa3Dj/PyLmot2PoN5hfYQMm5fT0C7AMVx+GS?= =?Windows-1252?Q?gxbD5s4UNtpDiV5fcp7V7kGH6ey587IeYVkVHWjCGFgcB+XoO5NJ9J1F?= =?Windows-1252?Q?4NeSmyAsw9q1XfAv0ELG1IQ0lg5CHg2pFyG68c6dc5GyBkrpOBViw7ur?= =?Windows-1252?Q?X7Pfb8DiZ3sWS7pLMXZgyd3v1IPVT4v9xPNFHoofiysE2hROcTkvhDoG?= =?Windows-1252?Q?w/QcrkoUJmaBWXT+2UUMI0hZZqsv9QqwP6Vo8OMvkBdJ6MUv5Z/79ky5?= =?Windows-1252?Q?3Cx+3cxF4dwZeQs9b7tmJbCTnANO9OniRRbpbfu4RqIMdGKsQhToBJ26?= =?Windows-1252?Q?eRpYT9UdlgGwGEjQu3usCD0lEmksjY974RORJNTccLF3ApuhB7PSuh8A?= =?Windows-1252?Q?ma9UusE7tUWz+w82nmQZOJviv1lR6EKP0g7l86I6bzU5PoR1YIsfK/sJ?= =?Windows-1252?Q?zfLnKPWuCPjguQ5OZhP3JM6XI6n91SJN4DHUCgNIWNoua59mFMSS1caI?= =?Windows-1252?Q?FLLDfJHQpssLyyKjDZjHU1PGl+8QM/Bg7eytVZowUxyh+QsEERvXif/1?= =?Windows-1252?Q?/HBqizyRvX4DzcWdScig1v9leqG5ibyaSLNMXt1k50S6j4uAscYlEt6Z?= =?Windows-1252?Q?XPqmCvEwtiOod29v/akjmUJkoTil//G5zXuk4aaGWlOW8nm3fcWtFDV3?= =?Windows-1252?Q?uMBYH0fIBeXGpuYgVGS0E1sJZiPLLWG8SYGGRnpAdzt209x1QFOdvkL4?= =?Windows-1252?Q?TkVVZoPrtseXW8FINenUcuECmwBqUzNRPhtQ/tnmEv0JEqOXHo5Imv0g?= =?Windows-1252?Q?ZfZew+QpZ8ye3R4yJavunnNu+5r7bOGEQYDig9K81fNDTzzt/G2/1f3G?= =?Windows-1252?Q?R6HKVKLfQyMUSsFYRbiW5gnMNG9wVIv2iNyRBZBcbEgwiGA0RaqKPU8E?= =?Windows-1252?Q?2zHOfPzPhmGPCn96GRCAJGlNu7nZqZG1iIfadUh1geiDGqwZqgMgNa2L?= =?Windows-1252?Q?bT1XU9dd/3j3l7qw?= X-MS-Exchange-CrossTenant-Network-Message-Id: ac8cfd8f-570d-4248-b52d-08d8982d622d X-MS-Exchange-CrossTenant-AuthSource: DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2020 08:19:56.2767 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I0PdKMU9HRbT3bL5UHMxcGuE+xQbS7Q2KC/1mDoKQ1R09WnK7mtigwgH2ll+F1sTy1SMZL0MZq0uz6x5Gm2KGg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR8401MB1242 X-OriginatorOrg: hpe.com X-Proofpoint-UnRewURL: 5 URL's were un-rewritten MIME-Version: 1.0 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-04_02:2020-12-04,2020-12-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 bulkscore=0 clxscore=1011 mlxlogscore=999 lowpriorityscore=0 adultscore=0 priorityscore=1501 impostorscore=0 suspectscore=1 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012040048 X-MIME-Autoconverted: from 8bit to quoted-printable by mx0b-002e3701.pphosted.com id 0B48Iig6003890 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 12/3/20 11:57 PM, Bret Barkelew via groups.io wrote: > Wasn=92t there another push (somewhere in the last 8 months, my brain is= foggy) to adopt LZMA2? Or was it a different algorithm? I couldn't find anything: https://edk2.groups.io/g/devel/search?q=3DLZMA2 >=20 > - Bret >=20 > *From: *Daniel Schaefer via groups.io > *Sent: *Thursday, December 3, 2020 4:12 AM > *To: *Laszlo Ersek ; devel@edk2.groups.io > *Cc: *Lin, Derek (HPS SW) > *Subject: *[EXTERNAL] Re: [edk2-devel] Multithreaded compression with LZ= MA2 >=20 > *From:* Laszlo Ersek >=20 > *Sent:*Thursday, December 3, 2020 18:24 > *To:* devel@edk2.groups.io ; Schaefer, Daniel > *Cc:* Lin, Derek (HPS SW) > *Subject:* Re: [edk2-devel] Multithreaded compression with LZMA2 >=20 > On 12/02/20 03:59, Daniel Schaefer wrote: > > Hi everyone, > > > > I'm looking into how to speed up the build process and noticed that o= ur > > build > > uses LZMA to encrypt the main firmware volume. Since it's quite big i= t > > takes a > > while but only uses one CPU thread. > > > > LZMA2 is a version of LZMA which can be multi-threaded and achieve mu= ch > > faster > > compression times. I did a quick benchmark using the `xz` command-lin= e > > tool, > > which uses a modified version of the LZMA SDK that EDK2 uses. The > > results are: > > > > Uncompressed size: 64M > > > > | Algo=A0 | Comp Time | Decomp Time | Size | Threads | > > | ----- | --------- | ----------- | ---- | ------- | > > | LZMA=A0 |=A0=A0=A0 19.67s |=A0=A0=A0=A0=A0=A0=A0 0.9s | 9.1M |=A0= =A0=A0=A0=A0=A0 1 | > > | LZMA2 |=A0=A0=A0 20.11s |=A0=A0=A0=A0=A0=A0=A0 1.2s | 9.2M |=A0=A0= =A0=A0=A0=A0 1 | > > | LZMA2 |=A0=A0=A0=A0 8.31s |=A0=A0=A0=A0=A0=A0=A0 1.0s | 9.4M |=A0= =A0=A0=A0=A0=A0 4 | > > > > Using those commands: > > > > time xz --format=3Dlzma testfile > > time unlzma testfile.lzma > > > > time xz --lzma2 testfile > > time unxz testfile.xz > > > > time xz -T4 --lzma2 testfile > > time unxz testfile.xz > > > > This is quite a significant improvement of build time, while > > decompression time > > and size only slightly increase. If that's a concern, then LZMA2 coul= d > > be used > > for development only. > > > > I haven't investigated the details of how to support this in the code > > but it > > appears to be a simple change, since the LZMA SDK that we use already > > supports > > LZMA2. > > > > What do you think? >=20 > "xz -T" works by splitting the input into blocks, and it generates a > multi-block compressed output. >=20 > Yes, that's correct. >=20 > > I'm unsure if the current LZMA > decompressor that runs inside the firmware (=3D guided section extractor= ) > copes with multi-block input. >=20 > I think you're right that it doesn't. But we can make the guided section= extractor use that same algorithm(LZMA2) and assign it a different GUID, r= ight? >=20 >=20