From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-002e3701.pphosted.com (mx0b-002e3701.pphosted.com [148.163.143.35]) by mx.groups.io with SMTP id smtpd.web11.2026.1606897532647828715 for ; Wed, 02 Dec 2020 00:25:32 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@hpe.com header.s=pps0720 header.b=b8fDYt/X; spf=pass (domain: hpe.com, ip: 148.163.143.35, mailfrom: prvs=060597e447=daniel.schaefer@hpe.com) Received: from pps.filterd (m0150244.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B28IWOl030645; Wed, 2 Dec 2020 08:25:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=subject : to : cc : references : from : message-id : date : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pps0720; bh=7QTbEmcLa918+TNhpTVVOuwypH+3LtmVr9+sXkUkRC0=; b=b8fDYt/XrhfEmeXgC42ARZ2jS1UkhVyn9dLCgcZQN+dPBcautGAJhCdvTUi9qKHYXO4m V+VlIoa2fTGBGxk4tOQ2XamGS4FYDrbueXUJga2jS0wbtUhj6WYcxs6uVJMie0CYSz9/ 5Cu3JVNU8NJmj8nCjiwoZyo7woFWwdfS+AXlKExYI32CLwmBNSU4ktCwjsCu+I7iF+QS Tzm8TTTNg6pY6q1sZWSG0s4mz3k2edxKPfsPgITeoY8LVqOnyY0L8O7G/FedBVCKKgwQ 7N5A8+eS6ZrLMfiAabxGrDSCHioiE5O2pY7pWTUQqkCk6yHsxvHIkYrMs6lAR+ob1Vt4 nQ== Received: from g2t2354.austin.hpe.com (g2t2354.austin.hpe.com [15.233.44.27]) by mx0b-002e3701.pphosted.com with ESMTP id 355gnq2b6u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 02 Dec 2020 08:25:13 +0000 Received: from G2W6311.americas.hpqcorp.net (g2w6311.austin.hp.com [16.197.64.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g2t2354.austin.hpe.com (Postfix) with ESMTPS id 9F05BC8; Wed, 2 Dec 2020 08:25:11 +0000 (UTC) Received: from G9W8669.americas.hpqcorp.net (16.220.49.28) by G2W6311.americas.hpqcorp.net (16.197.64.53) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 2 Dec 2020 08:25:04 +0000 Received: from G4W10204.americas.hpqcorp.net (2002:10cf:5210::10cf:5210) by G9W8669.americas.hpqcorp.net (2002:10dc:311c::10dc:311c) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 2 Dec 2020 08:25:03 +0000 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (15.241.52.13) by G4W10204.americas.hpqcorp.net (16.207.82.16) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Wed, 2 Dec 2020 08:25:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SHX4/kE5xHv5haLhe1TWWFrdulhAOMMRS1kpigcqPx4IGggl59dJw07En+BXwgbrq91AZRl0uZDlGdzLitnpR1ao1KeT07seeaT3GCKKVyaORdI5X//Bkkv0XKSvHUx+aYlWkUOW0Yp6fiZKkxWlj6wlyj3QgfqRgCMXF0qCFJKAUtey0BF0uBSqT5j2eSgH4QF2cwU4ElMw44O90qtClXnjfmDcoFQB6D8tFYsyx6KSAa/yLDJVsWgHSLO2t26qRnoiSLiyIhDPSHo+7GaRVOw0LrVBPBh/W70lA7NJz9iCghgBeV2qAntEsnRW3OjegMsXc27nb4E2Z3IV4rncZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7QTbEmcLa918+TNhpTVVOuwypH+3LtmVr9+sXkUkRC0=; b=bzHb7V2BPsE2vZ2dogE+xUKn84zDN8peGu3DiNxkk632XgVqTW7xbe5Tcp+lvoZ60FaCE5sxMQU//eg+iKVFhnFE9GgzdJoFeyTirIx/vjH2eu+n6ckg/ylAdqeMC3lYk0KMiCYsa4xxdlhyL2dapNGuJhIPTN7TOT1gfObSwk90BVeRY6IY1k3hzCV6ebo8LRifjGF8hW1XDedK/S5kb8q3epd5F5GO0PvLf4gdA/VRg43uj+MOStkPsfbHfPaMeod3SaKxowSAhHjFEHEYoPMLikdsjH2i1u/hGfu4YqYVjsgiBNy5ourPT+h2BUpHP3gUYRgWkd7zGHyfuys0Gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hpe.com; dmarc=pass action=none header.from=hpe.com; dkim=pass header.d=hpe.com; arc=none Authentication-Results: hpe.com; dkim=none (message not signed) header.d=none;hpe.com; dmarc=none action=none header.from=hpe.com; Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) by DF4PR8401MB0665.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:7609::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20; Wed, 2 Dec 2020 08:25:02 +0000 Received: from DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f]) by DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM ([fe80::691c:7962:6e56:c91f%8]) with mapi id 15.20.3611.025; Wed, 2 Dec 2020 08:25:02 +0000 Subject: =?UTF-8?B?UmU6IOWbnuWkjTogW2VkazItZGV2ZWxdIE11bHRpdGhyZWFkZWQgY29tcHJlc3Npb24gd2l0aCBMWk1BMg==?= To: , , CC: References: <097F435E-840E-4A20-AF20-8D2A4CC125F2@apple.com> <013801d6c86b$0941a040$1bc4e0c0$@byosoft.com.cn> From: "Daniel Schaefer" Message-ID: <0b9f1f35-a958-fddf-d2aa-247f41a4b5b1@hpe.com> Date: Wed, 2 Dec 2020 16:24:47 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 In-Reply-To: <013801d6c86b$0941a040$1bc4e0c0$@byosoft.com.cn> X-Originating-IP: [15.211.146.34] X-ClientProxiedBy: SG2PR04CA0190.apcprd04.prod.outlook.com (2603:1096:4:14::28) To DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM (2a01:111:e400:760d::23) X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [10.10.6.166] (15.211.146.34) by SG2PR04CA0190.apcprd04.prod.outlook.com (2603:1096:4:14::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.17 via Frontend Transport; Wed, 2 Dec 2020 08:25:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 2d17b94a-4309-4cd8-2e2d-08d8969bc3b3 X-MS-TrafficTypeDiagnostic: DF4PR8401MB0665: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KZaShidnLm0AbJY7xHdE7C02cMJ2OeBZ+fZS30M9jsreuY4/0Gdz7v2QWZsCr6hEx52n6OeFwjwx6hvm5OeTJfUjYIkQpwA3B7U1JUWr43KkrOgI0+tLYKJJVy4WZekTlsm9p7XWRi2FSsoPZ7Eokw6IH6dr0BeqaBw5Z0d6qdjUs0dHn5bhHL4WTIptH7khlMwUFIkVzyQnxAuY0L7msXmXd+LEZyevxZVDpBF5pZHew77+W++Rv8OpAVLnHrq0RytGjCvXUwiKARe5yf5QsE8u2q9rKKxAA6J6yGf6+bo7RlkdNf3pC5/pT2m7WzB2OpTWNfAKP/6+iuYRA9wgr7/i1LfgxXNk19VM/zmTb1+8EhPXTSl8HEy3M5nMvcHHQGXhFb+6Ohjv1zoBvS4F1YWtFIEbXaxiMvKG5aogv3o= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(39860400002)(136003)(366004)(396003)(346002)(376002)(956004)(8936002)(6486002)(31686004)(2906002)(16576012)(5660300002)(16526019)(224303003)(66946007)(186003)(53546011)(36756003)(52116002)(6666004)(26005)(83380400001)(86362001)(66556008)(44832011)(316002)(31696002)(66476007)(2616005)(478600001)(4326008)(43740500002)(45980500001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?UEJTMGRqaGZwUGFxSWovNDJKWmt5aml5VFkyMjRHZy9qT2hidTYvNXo5TVA3?= =?utf-8?B?Sm9meVJqVWsyc28vWS9tbDh6TTNJV0Z4eVhldkJNTm0zQ3dQdFlWZi8xYzZ0?= =?utf-8?B?RHhwc0FSY0NTSTNLbmt6aGRaVGFCUkt2QnVXdzVVeEJzWktjdDY3NU9GM0lU?= =?utf-8?B?bkRWZHpGUVp0dGEvMnRJNEh0LzF3TmpwSzBFdmpGdkF2RUpMdWNyWVIwODc2?= =?utf-8?B?RkxSckRZNytZOUR2VjhLVzRaR0FudG1oTzYxbWc3VUlUVzR3eW1GdExUZTl3?= =?utf-8?B?SUpvQ1g5eDdYVGwwaGtxMVRGdFBJMVNlVzR5ZXNIR1JuVkMrdEMzWVlsR0dr?= =?utf-8?B?V0h2VHExM2JTN2R6QmhsUFl0QUFKN3JGZDVLR1BLa3Y3VjJ4T2svWW9TWW5i?= =?utf-8?B?cXRwNlJwdk5Cc1pkU29ESnExcGVaRHlWRkw0VG9GQWs1eEo3YnJtbzdMVDBj?= =?utf-8?B?c0tab0VJNmlhMmFjR1VtUytRazNDOWJVcWtGVUtSTk1id0w0U0V1Ui9mQW5Z?= =?utf-8?B?OG1iRlo3R1BiMi9CcmVadEJKdTVaR0hvTUxSTVViRVZNS3E3MWpmRzFLK2lo?= =?utf-8?B?M3J0RndtWlJaRDkwMHdHY2FESkJrRm1IMUhLcTVZRVZOUzlkVWJTTjI2SVhU?= =?utf-8?B?MmthMks0NThXbUY2UTlmSUdBRXBDajlRZnZJdnQvMHk4ZE9oalBzTWV0YnVw?= =?utf-8?B?cHUyZ3RKQXp4Z0tLTTJpTEYxL0NqKytNR2tqUEc0QmwzZDNzSXBjaGh2MWVR?= =?utf-8?B?eERxdDNidWlNUzZRZHE5MENTLzRPc0cxTHk2eHNIUGV6MmpqbkFKb1FoNU03?= =?utf-8?B?SVJwa1h4b3NjcEZpUWo4dmZ0R2p5ZXRsV0NMM3k2MVI3c0JDeEtMbTB0RXpH?= =?utf-8?B?bXVKNy8rekV0VExrUys1NXMzVDZPYlFJc0dMNHZKbmpJUzFURmM3LzQzWkNT?= =?utf-8?B?bnA3ZGd0VkhBdmUxWUhWWmxRTExkS3ZtY2dINjlweTZscElCNmZaZVd3WXhv?= =?utf-8?B?WWpWMHlQQllVVVVzUUpvZjJRNEQxc2R1S2FWSzlwbVh1MjFybURwS3FuZXQv?= =?utf-8?B?UVN1VWR6SFVVSlpiMTYyTHpJVnZQd3oveDQyL3NPOVc1RzdkZDRSMFVQUURT?= =?utf-8?B?a1Z2RnRvSy93ZkpZYzVDczFrWW16UHo4Vi9MZ0pKMzMzRk53aVk0Sjg1cjV3?= =?utf-8?B?MjhQTFB0dFNpaGptdGY0cUEybTR5SVdHMnUzTWlJS3JsWjYvZlMzQVBLdFFk?= =?utf-8?B?WVowVHE1TnFvVEF6bjhCRER0U0w0OEVaWlhXdTRwcWpqbG9RRm1Td0ZPSEdH?= =?utf-8?Q?H6Ag8ObM+OIXCSCa/M0J4neiAPYA0u85Uq?= X-MS-Exchange-CrossTenant-Network-Message-Id: 2d17b94a-4309-4cd8-2e2d-08d8969bc3b3 X-MS-Exchange-CrossTenant-AuthSource: DF4PR8401MB0444.NAMPRD84.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2020 08:25:02.2691 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x91Vn9xlQ8FEbYh31+lHdvJ5dJTh/c7y+Whr764C5tRDT/9FsMbwjMVPSjarho5r4WpA3oRo2fTOHw3VHFXkYw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR8401MB0665 X-OriginatorOrg: hpe.com X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312,18.0.737 definitions=2020-12-02_01:2020-11-30,2020-12-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 spamscore=0 mlxlogscore=999 priorityscore=1501 clxscore=1011 bulkscore=0 lowpriorityscore=0 impostorscore=0 malwarescore=0 suspectscore=6 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012020050 X-MIME-Autoconverted: from 8bit to quoted-printable by mx0b-002e3701.pphosted.com id 0B28IWOl030645 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 12/2/20 1:21 PM, gaoliming wrote: > Daniel: >=20 > =C2=A0Can you provide the compressed image size? And, what image is us= ed to be compressed? Is it the generated FV image? >=20 > Thanks >=20 > Liming >=20 > *=E5=8F=91=E4=BB=B6=E4=BA=BA:*bounce+27952+68159+4905953+8761045@groups= .io *=E4=BB=A3=E8=A1=A8 *A= ndrew Fish via groups.io > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:*2020=E5=B9=B412=E6=9C=882=E6=97=A5= 11:37 > *=E6=94=B6=E4=BB=B6=E4=BA=BA:*devel@edk2.groups.io; daniel.schaefer@hpe= .com > *=E6=8A=84=E9=80=81:*derek.lin2@hpe.com > *=E4=B8=BB=E9=A2=98:*Re: [edk2-devel] Multithreaded compression with LZ= MA2 >=20 >=20 >=20 > On Dec 1, 2020, at 6:59 PM, Daniel Schaefer > wrote: >=20 > Hi everyone, >=20 > I'm looking into how to speed up the build process and noticed that= our build > uses LZMA to encrypt the main firmware volume. Since it's quite big= it takes a > while but only uses one CPU thread. >=20 > LZMA2 is a version of LZMA which can be multi-threaded and achieve = much faster > compression times. I did a quick benchmark using the `xz` command-l= ine tool, > which uses a modified version of the LZMA SDK that EDK2 uses. The r= esults are: >=20 > Uncompressed size: 64M >=20 > | Algo =C2=A0| Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA =C2=A0| =C2=A0=C2=A0=C2=A019.67s | =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A00.9s | 9.1M | =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01 | > | LZMA2 | =C2=A0=C2=A0=C2=A020.11s | =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A01.2s | 9.2M | =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01 | > | LZMA2 | =C2=A0=C2=A0=C2=A0=C2=A08.31s | =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A01.0s | 9.4M | =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A04 | >=20 > Using those commands: >=20 > time xz --format=3Dlzma testfile > time unlzma testfile.lzma >=20 > time xz --lzma2 testfile > time unxz testfile.xz >=20 > time xz -T4 --lzma2 testfile > time unxz testfile.xz >=20 > This is quite a significant improvement of build time, while decomp= ression time > and size only slightly increase. If that's a concern, then LZMA2 co= uld be used > for development only. >=20 > I haven't investigated the details of how to support this in the co= de but it > appears to be a simple change, since the LZMA SDK that we use alrea= dy supports > LZMA2. >=20 > What do you think? >=20 > Interesting idea. What OS did you use? I tried this on macOS on some la= rger FVs and I did not see much difference? I tried a 17.5 MiB FV and it = was around 3 seconds both ways. >=20 > Maybe it would be worth while seeing how it works on various systems? I= guess it might be data set related? Hi Andrew and Liming, the FV file is our main FV with the majority of DXEs. It's 64MB uncompres= sed and 9MB compressed, as mentioned before. Unfortunately I cannot share th= at particular file with you but I am also suprised that it compresses so wel= l to just 14% of its original size. I'm running my tests on X64 Linux. I ran the same tests again on a more powerful machine with the hyperfine command. It runs the testcase 3 times= to warm up (e.g. caches) and then runs it 10 times that count towards the av= erage. The result is the same as before. The compression takes just 40% with 4 t= hreads. I don't observe any further speedup by using all 16 thread of the CPU. # Simple LZMA $ hyperfine --warmup 3 'xz -k --format=3Dlzma testfile && rm testfile.lzm= a' Benchmark #1: xz -k --format=3Dlzma testfile && rm testfile.lzma Time (mean =C2=B1 =CF=83): 12.755 s =C2=B1 0.151 s [User: 12.6= 91 s, System: 0.064 s] Range (min =E2=80=A6 max): 12.568 s =E2=80=A6 12.991 s 10 runs # LZMA2 with single thread $ hyperfine --warmup 3 'xz -k -T1 --lzma2 testfile && rm testfile.xz' Benchmark #1: xz -k -T1 --lzma2 testfile && rm testfile.xz Time (mean =C2=B1 =CF=83): 12.838 s =C2=B1 0.149 s [User: 12.7= 83 s, System: 0.055 s] Range (min =E2=80=A6 max): 12.546 s =E2=80=A6 13.053 s 10 runs # LZMA2 with 4 threads $ hyperfine --warmup 3 'xz -k -T4 --lzma2 testfile && rm testfile.xz' Benchmark #1: xz -k -T4 --lzma2 testfile && rm testfile.xz Time (mean =C2=B1 =CF=83): 5.241 s =C2=B1 0.025 s [User: 13.5= 37 s, System: 0.177 s] Range (min =E2=80=A6 max): 5.227 s =E2=80=A6 5.302 s 10 runs Using xz from mingw64 on Windows 10 X64 the 4-threaded compression takes = 47% of single-threaded. --- I wanted to try it on a bigger file and ran the same benchmarks on a 16MB= file to discover that compression is no faster with multithreading. The file s= hrinks from 16MB to 9MB. However, after my tests I discovered that this is the combined FV, which includes a few other compressed FVs. Here the multithreaded command is even able to compress the FV 0.2MB more. $ hyperfine --warmup 3 'xz -k -T1 --lzma2 testfile && rm testfile.xz' Benchmark #1: xz -k -T1 --lzma2 testfile && rm testfile.xz Time (mean =C2=B1 =CF=83): 2.874 s =C2=B1 0.134 s [User: 2.82= 5 s, System: 0.049 s] Range (min =E2=80=A6 max): 2.751 s =E2=80=A6 3.088 s 10 runs $ ls -lh -rw-r--r-- 1 zoid users 16M Dec 2 15:29 testfile -rw-r--r-- 1 zoid users 9.4M Dec 2 15:29 testfile.lzma $ hyperfine --warmup 3 'xz -k -T4 --lzma2 testfile && rm testfile.xz' Benchmark #1: xz -k -T4 --lzma2 testfile && rm testfile.xz Time (mean =C2=B1 =CF=83): 2.874 s =C2=B1 0.108 s [User: 2.81= 8 s, System: 0.070 s] Range (min =E2=80=A6 max): 2.775 s =E2=80=A6 3.081 s 10 runs $ ls -lh -rw-r--r-- 1 zoid users 16M Dec 2 15:29 testfile -rw-r--r-- 1 zoid users 9.2M Dec 2 15:29 testfile.xz So even with a file that doesn't compress as well, the performance doesn'= t get any worse. Thanks, Daniel