From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by mx.groups.io with SMTP id smtpd.web08.1872.1606878051928318197 for ; Tue, 01 Dec 2020 19:00:52 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=Jkqpkbmx; spf=pass (domain: hpe.com, ip: 148.163.143.35, mailfrom: prvs=060597e447=daniel.schaefer@hpe.com) Received: from pps.filterd (m0134425.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B22tKbx016648 for ; Wed, 2 Dec 2020 03:00:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=to : from : subject : cc : message-id : date : content-type : content-transfer-encoding : mime-version; s=pps0720; bh=6PpOSIbOHqFG/rGBTBM48/k5zDT4SO96OTsQ6fnpe1w=; b=JkqpkbmxyuTrrBBaxZ1LWFxMoVjZBhuMsgOQCjprno5WLQGyVbYyApPxvbYq0iFQ0lF5 ON6wQsV6vmWTMapoM49NDrBdUkto7MTnhZSyiIukEQQL70xOqCHZ6aE3k3P3Z7tXo0/+ T+PiZDU12MZlK/ts6zVxfqb4Kt1IN/WHhhykbQhIrJCBSXasa4u1uaouWxVzcuN75+bB S4uNdR9TY2fTccZtR/JT2QqgedN2ZmiB95h0MKjt4c5jZLUZ6gXuCm2RhabUijFcCjku U7C16nU0MERYpA46ORin5mYf+9IbzZg2BOXbQFrBs3ifsKbFZnFLl+jRuwBdjyGb/3eo 6A== Received: from g9t5009.houston.hpe.com (g9t5009.houston.hpe.com [15.241.48.73]) by mx0b-002e3701.pphosted.com with ESMTP id 355r47d3he-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 02 Dec 2020 03:00:48 +0000 Received: from G1W8106.americas.hpqcorp.net (g1w8106.austin.hp.com [16.193.72.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g9t5009.houston.hpe.com (Postfix) with ESMTPS id 1093D6D for ; Wed, 2 Dec 2020 03:00:48 +0000 (UTC) Received: from G4W9330.americas.hpqcorp.net (16.208.32.116) by G1W8106.americas.hpqcorp.net (16.193.72.61) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 2 Dec 2020 03:00:02 +0000 Received: from G9W9209.americas.hpqcorp.net (2002:10dc:429c::10dc:429c) by G4W9330.americas.hpqcorp.net (2002:10d0:2074::10d0:2074) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 2 Dec 2020 03:00:01 +0000 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (15.241.52.11) by G9W9209.americas.hpqcorp.net (16.220.66.156) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Wed, 2 Dec 2020 03:00:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IFsVP2Lck/QN6masdpARdYl4ESuxLfNUspMt3M9NK4Dg3bdw6BsjUPDuzcaoRnWG0CjSvodzLu1bpFeU+cEg2LXn/tR7srqh7AynKEgP7SKH+53QvR87IpJvwtLnywhiEMpNM5LFjVwxfooySy4bhz524scA06+4WKsdzkL4SrgbtToMl3c1E18YsqvM1pmlZ/yuNi3PfN7Ef/K9v24kA/GFqI9EAxgbmwmQWAEQSZGXrUpxwl3gPxA8C34PA7m1YtYtDfNaUgK+eyXiz2BEljyPX4LWa/HOMjxscKqECG4HYCEN+lRbA++vJ56UfWNtCEyjAB1OCiIn7goAQEg/kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6PpOSIbOHqFG/rGBTBM48/k5zDT4SO96OTsQ6fnpe1w=; b=eBiuMhIFfuqNqrAs5jhbu7PKknQuJuvScE/jrLtvn9Oq2ObreFRZ9GF/bZOguhVIUXzqAksNFX0lERjKagCX194td0owLo9hEACy1ctnTQgaWliVXq6eUIwY+eDwawycbUCheH1rzPKQB8Ged8FRUZSzcEBMasVF0ptBq04T0BKSAAo/oQ+mG3RuWX6jbu9Ji4fpmmWjcg842hVi8/vQl3DJevS0nKPyO+/Aebd1K7BQ3OAFZJJRqdQKcDEWgj5anatMca0VRR+Tc/VSmhoEFnn9TeRK7Y9/ajdFxzKsvx3jvkiDhJpIKTIi1C9xScgTXFDVbWcqRFKpttMiiozm7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hpe.com; dmarc=pass action=none header.from=hpe.com; dkim=pass header.d=hpe.com; arc=none Authentication-Results: hpe.com; dkim=none (message not signed) header.d=none;hpe.com; dmarc=none action=none header.from=hpe.com; Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) by DF4PR8401MB1098.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:7612::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.17; Wed, 2 Dec 2020 03:00:00 +0000 Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f]) by DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f%8]) with mapi id 15.20.3611.025; Wed, 2 Dec 2020 02:59:59 +0000 To: "devel@edk2.groups.io" From: "Daniel Schaefer" Subject: Multithreaded compression with LZMA2 CC: Message-ID: Date: Wed, 2 Dec 2020 10:59:45 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 X-Originating-IP: [15.211.146.34] X-ClientProxiedBy: SG2PR02CA0043.apcprd02.prod.outlook.com (2603:1096:3:18::31) To DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [10.10.6.166] (15.211.146.34) by SG2PR02CA0043.apcprd02.prod.outlook.com (2603:1096:3:18::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.17 via Frontend Transport; Wed, 2 Dec 2020 02:59:58 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 82d6299f-5d76-4af7-1269-08d8966e5b49 X-MS-TrafficTypeDiagnostic: DF4PR8401MB1098: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uID1DaR5ffYVK105Uwq20bRmW3STrYNDlhQNq7QT3F88fwpB7tIRrfWp7L9fF3O+Gwt1bjozWpVKHlwAW8vEgeR9Ltt8HeAPA1lBQYpOvE6gZ0cyS8KwYDKARngFrN51Z9lyoRiPk8FA1/vARFsYke+eMTaLYfm9GR3Aw5dQ4VPdRi5CldRVoSW2ZalV9e70AlUZk7BzB1plWpI0aKc/LB3VOBXTEJtNn0Re3fVce3/HQshdChcP9CKcj91u9pmZq383f51CyOCX/mHRf9FwIgLkKyVshaJB724pfvHQOtax3HmXjZIFWUdlhMAG1T7ndt50Po17jPdjnNOhy/gbq3bOYTapeqv9zbmSZH71wkDsjccnkpj3iuTXhB/oIC6XDh4rvHKC8X2MAwCmF+PwA7Se4oOo9KL/774wsGYGSJc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(346002)(376002)(136003)(39860400002)(396003)(366004)(4326008)(186003)(86362001)(2906002)(8936002)(6916009)(6666004)(6486002)(16526019)(52116002)(36756003)(956004)(2616005)(31696002)(44832011)(26005)(8676002)(31686004)(83380400001)(66556008)(66476007)(66946007)(316002)(5660300002)(478600001)(16576012)(43740500002)(45980500001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?LzFWdVBCbXhERlc2ZjRkdDNvaDhyWERzb1NyWEQ2SDlSb291NEk2QmdiYzRP?= =?utf-8?B?SmxpU3M3NCtjZi9hN2xvekRFZk5vTmJJNS8xdTB2ajFoMFNpR2VucTAraU82?= =?utf-8?B?RXI1ZysvZU9nYytpbWhPNGlWdDVERE1vQUJOZXRRempwb3kxdGRYWWc0U0kz?= =?utf-8?B?elQ2YlJyUmtHNXV5ekxDL1VsK1B5dTFQMklwaENueGRSekZncFp5NWdzQUc2?= =?utf-8?B?R0xtSERENjNlVXYvRUhIODFYMjVWUktlc201NlR0SldZVEZMYVQ0VWxVd0J5?= =?utf-8?B?ZHVtSXhuRzN5MWdMemlEOUVSNVV2L2JOVnJHUjh3QVI2QTBWSkd0MDhTUytl?= =?utf-8?B?dHEwMGw3Z1EyMFRXZTJxMTEyL3BBVFpZRjlNaXU4TWR4TEpvQlk2UEZPNTJo?= =?utf-8?B?SGYrMjd3WEZSMC80NnRNWlFFVFlqNjVXYWl4bXdzaWJablRld2JWRFJSL2Qr?= =?utf-8?B?aEhtL0Z2TDhtTFQ0aGNGblRGMDNlRnFZem5sMXB6TFRyQzMvLzdlZEVPRk45?= =?utf-8?B?QWlDUHV3M3hSd0FkQWE4MWJCaDQwZTM5WlpqeFVyNFdaQ09taFFSZzdDaW5S?= =?utf-8?B?Y3ZkY0w3SDFYNHNnSEJWTk8yeW8ybnZsZm9vaFBuVTA1ZGFRcGhZU1Z4Rzlq?= =?utf-8?B?aXdjQk1lcko0Yk5seHNKK3QwUEs5V05DZnRjbm5yQ3E0RXlkRzJMb21tSTdB?= =?utf-8?B?MXFTaVhyRWp6TFowWml1WVZhc1NYWkhQVUdBR0hUbXVJK3dsc0VYMy93c2tF?= =?utf-8?B?eC9jb3kzUTIxWmxGQWFiRjhOc3U1R3cxUkhYTkVCT2RlWE0wK09jemw5Z1hH?= =?utf-8?B?bWY0SFBCaUUzcXNwSEpGdDg1TFErNVIycTRBeHJiZkRXQS93cjNYUEdwK2po?= =?utf-8?B?cks0NGhlQWI0UTA1VUZzamYvbW9WWHpMdFBJeHY0UVVIcXpMMUc1SmMvVDRn?= =?utf-8?B?ZFFIMUhzbHpMam91ZVMyMGtJdVRaOGRlS0JqVk12aHp1UDJJSDcxNFZ1eGhh?= =?utf-8?B?RSthRThvQXhnVEpxaXJuVWU5RzU2RlNhSkd3L0NHc2tRMTliSTRYdjI1SE5C?= =?utf-8?B?RzJNUE1ma3phd0JybGdiekNoZGV4WkZDcWhHSEdPaDM5UWthc3hQLy80Nk1T?= =?utf-8?B?MUsyT3lKYXd6a21lVGNnVVNaZWFFMmlCY0kyazV0TkNiSkN2STBlcWZBM3JW?= =?utf-8?B?TzVBQlR5Q08vd3JyaUs5cU1RMll2S1I4dDV5QURKUFhkZ3FRZHgwNWJtblp1?= =?utf-8?B?YkVsdU9KSm5SVUM3NnB2UzhBQVc3dU1FQ2hyYTFrQXM5NWU1R2pjR0xQZHFi?= =?utf-8?Q?X8bhEyu0A1jqJQARN/8N9iw0dUEvkq5bPF?= X-MS-Exchange-CrossTenant-Network-Message-Id: 82d6299f-5d76-4af7-1269-08d8966e5b49 X-MS-Exchange-CrossTenant-AuthSource: DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2020 02:59:59.6648 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Sg3/qNVprDoVUis6RGFtlFol8jAWOsvZ/erFWIG1uLoP45omBlJQGQvfGcd/yFPIV4Rf0Mvv2Mi/9qkxBwT1jQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR8401MB1098 X-OriginatorOrg: hpe.com X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-01_12:2020-11-30,2020-12-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 adultscore=0 spamscore=0 phishscore=0 mlxscore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 suspectscore=2 impostorscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012020017 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Hi everyone, I'm looking into how to speed up the build process and noticed that our build uses LZMA to encrypt the main firmware volume. Since it's quite big it takes a while but only uses one CPU thread. LZMA2 is a version of LZMA which can be multi-threaded and achieve much faster compression times. I did a quick benchmark using the `xz` command-line tool, which uses a modified version of the LZMA SDK that EDK2 uses. The results are: Uncompressed size: 64M | Algo | Comp Time | Decomp Time | Size | Threads | | ----- | --------- | ----------- | ---- | ------- | | LZMA | 19.67s | 0.9s | 9.1M | 1 | | LZMA2 | 20.11s | 1.2s | 9.2M | 1 | | LZMA2 | 8.31s | 1.0s | 9.4M | 4 | Using those commands: time xz --format=lzma testfile time unlzma testfile.lzma time xz --lzma2 testfile time unxz testfile.xz time xz -T4 --lzma2 testfile time unxz testfile.xz This is quite a significant improvement of build time, while decompression time and size only slightly increase. If that's a concern, then LZMA2 could be used for development only. I haven't investigated the details of how to support this in the code but it appears to be a simple change, since the LZMA SDK that we use already supports LZMA2. What do you think? Thanks, Daniel