From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.groups.io with SMTP id smtpd.web09.8019.1606991104345754617 for ; Thu, 03 Dec 2020 02:25:04 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bTPp3BbX; spf=pass (domain: redhat.com, ip: 63.128.21.124, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1606991103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zsfh6q7jeL4AZeOEKzVnT1zf6slNOiwdI0kfAEP0bhE=; b=bTPp3BbXqnOVig75hFmleUtmniJOO8yrVf1QoBQ365ZaSqCHsQBiiDMjhlfT50E+E1ss6P XvfPiTMkXoZLiNr7OBpj6Y/BNrbb4Se22DXmdltoNU/JBZhs7VyAZvv2L+mJsQfngBr60a DyDk4DmztB0uWcnedCgYQpTkqz8p+2Y= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-468-Akuv1fhYNJK0og9X8QKNBg-1; Thu, 03 Dec 2020 05:25:01 -0500 X-MC-Unique: Akuv1fhYNJK0og9X8QKNBg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B03988558EF; Thu, 3 Dec 2020 10:25:00 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-113-182.ams2.redhat.com [10.36.113.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id C2F95189C4; Thu, 3 Dec 2020 10:24:59 +0000 (UTC) Subject: Re: [edk2-devel] Multithreaded compression with LZMA2 To: devel@edk2.groups.io, daniel.schaefer@hpe.com Cc: derek.lin2@hpe.com References: From: "Laszlo Ersek" Message-ID: <9ebc277a-4acd-f567-cc08-fa076e4a5766@redhat.com> Date: Thu, 3 Dec 2020 11:24:58 +0100 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=lersek@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit On 12/02/20 03:59, Daniel Schaefer wrote: > Hi everyone, > > I'm looking into how to speed up the build process and noticed that our > build > uses LZMA to encrypt the main firmware volume. Since it's quite big it > takes a > while but only uses one CPU thread. > > LZMA2 is a version of LZMA which can be multi-threaded and achieve much > faster > compression times. I did a quick benchmark using the `xz` command-line > tool, > which uses a modified version of the LZMA SDK that EDK2 uses. The > results are: > > Uncompressed size: 64M > > | Algo  | Comp Time | Decomp Time | Size | Threads | > | ----- | --------- | ----------- | ---- | ------- | > | LZMA  |    19.67s |        0.9s | 9.1M |       1 | > | LZMA2 |    20.11s |        1.2s | 9.2M |       1 | > | LZMA2 |     8.31s |        1.0s | 9.4M |       4 | > > Using those commands: > > time xz --format=lzma testfile > time unlzma testfile.lzma > > time xz --lzma2 testfile > time unxz testfile.xz > > time xz -T4 --lzma2 testfile > time unxz testfile.xz > > This is quite a significant improvement of build time, while > decompression time > and size only slightly increase. If that's a concern, then LZMA2 could > be used > for development only. > > I haven't investigated the details of how to support this in the code > but it > appears to be a simple change, since the LZMA SDK that we use already > supports > LZMA2. > > What do you think? "xz -T" works by splitting the input into blocks, and it generates a multi-block compressed output. I'm unsure if the current LZMA decompressor that runs inside the firmware (= guided section extractor) copes with multi-block input. Thanks Laszlo