From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.132.183.28; helo=mx1.redhat.com; envelope-from=lersek@redhat.com; receiver=edk2-devel@lists.01.org Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 912BD21F2AF8B for ; Thu, 28 Sep 2017 04:12:00 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 05D24108FA6; Thu, 28 Sep 2017 11:15:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 05D24108FA6 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=lersek@redhat.com Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-73.rdu2.redhat.com [10.10.120.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id 869938FBF2; Thu, 28 Sep 2017 11:15:13 +0000 (UTC) To: "Gao, Liming" , "Zhu, Yonghong" , "edk2-devel@lists.01.org" Cc: "Kinney, Michael D" , "Shaw, Kevin W" References: <1505803680-15860-1-git-send-email-yonghong.zhu@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E15EE06@SHSMSX152.ccr.corp.intel.com> From: Laszlo Ersek Message-ID: <524b81df-bb15-5fed-d23b-836aaa20be38@redhat.com> Date: Thu, 28 Sep 2017 13:15:12 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <4A89E2EF3DFEDB4C8BFDE51014F606A14E15EE06@SHSMSX152.ccr.corp.intel.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 28 Sep 2017 11:15:15 +0000 (UTC) Subject: Re: [Patch V2] Build spec: add description for build with binary cache X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Sep 2017 11:12:01 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 09/28/17 11:02, Gao, Liming wrote: > Laszlo: > Hash way may improve the incremental build performance. If hash value is not changed, module AutoGen and Make will be skipped. > Time stamp way bases on Makefile. This way will run AutoGen every time. If no change, Makefile will not be updated. I did some tests, with ArmVirtQemu (AARCH64) and OVMF (Ia32 / Ia32X64 / X64). First I built all these platforms twice in a row, *without* --hash: - build ArmVirtQemu (AARCH64) - build OVMF (Ia32) - build OVMF (Ia32X64) - build OVMF (X64) - repeat all four builds in the same order, and now capture times in the report Then I built them twice in a row, *with* --hash: - build ArmVirtQemu (AARCH64) - build OVMF (Ia32) - build OVMF (Ia32X64) - build OVMF (X64) - repeat all four builds in the same order, and now capture times in the report The idea was that in the "repeat" builds, there was nothing to actually rebuild. Then we can compare how much time "--hash" saved, between the "repeat" builds: * without --hash: build.aa64virt.report:Build Duration: 00:00:15 build.aa64virt.report:AutoGen Duration: 00:00:04 build.aa64virt.report:Make Duration: 00:00:04 build.aa64virt.report:GenFds Duration: 00:00:06 build.ovmf.32.report:Build Duration: 00:00:16 build.ovmf.32.report:AutoGen Duration: 00:00:04 build.ovmf.32.report:Make Duration: 00:00:06 build.ovmf.32.report:GenFds Duration: 00:00:06 build.ovmf.3264.report:Build Duration: 00:00:20 build.ovmf.3264.report:AutoGen Duration: 00:00:06 build.ovmf.3264.report:Make Duration: 00:00:07 build.ovmf.3264.report:GenFds Duration: 00:00:07 build.ovmf.report:Build Duration: 00:00:18 build.ovmf.report:AutoGen Duration: 00:00:04 build.ovmf.report:Make Duration: 00:00:06 build.ovmf.report:GenFds Duration: 00:00:06 * With --hash: build.aa64virt.report:Build Duration: 00:00:11 build.aa64virt.report:AutoGen Duration: 00:00:04 build.aa64virt.report:GenFds Duration: 00:00:06 build.ovmf.32.report:Build Duration: 00:00:11 build.ovmf.32.report:AutoGen Duration: 00:00:04 build.ovmf.32.report:GenFds Duration: 00:00:06 build.ovmf.3264.report:Build Duration: 00:00:14 build.ovmf.3264.report:AutoGen Duration: 00:00:06 build.ovmf.3264.report:GenFds Duration: 00:00:07 build.ovmf.report:Build Duration: 00:00:12 build.ovmf.report:AutoGen Duration: 00:00:05 build.ovmf.report:GenFds Duration: 00:00:06 With "--hash", the "Make" step disappeared entirely (which is very welcome, and I'll start using --hash today). However, the "AutoGen" step took just as long. Can you think of a reason why? (Above you mention that AutoGen should be skipped too.) Thanks! Laszlo >> -----Original Message----- >> From: Laszlo Ersek [mailto:lersek@redhat.com] >> Sent: Thursday, September 28, 2017 4:19 PM >> To: Zhu, Yonghong ; edk2-devel@lists.01.org >> Cc: Kinney, Michael D ; Shaw, Kevin W >> ; Gao, Liming >> Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary >> cache >> >> On 09/19/17 08:48, Yonghong Zhu wrote: >>> V2: >>> change the option name to --binary-destination and --binary-source. >>> >>> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 >> >> What are the benefits of a hash-based incremental build over a >> timestamp-based incremental build? >> >> Thank you, >> Laszlo >> >>> Cc: Liming Gao >>> Cc: Michael Kinney >>> Cc: Kevin W Shaw >>> Contributed-under: TianoCore Contribution Agreement 1.1 >>> Signed-off-by: Yonghong Zhu >>> --- >>> .../82_auto-generation_process.md | 20 >> ++++++++++++++++++++ >>> README.md | 1 + >>> appendix_d_buildexe_command/d4_usage.md | 19 >> +++++++++++++++---- >>> 3 files changed, 36 insertions(+), 4 deletions(-) >>> >>> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md >> b/8_pre-build_autogen_stage/82_auto-generation_process.md >>> index 671a7d5..f2ddf32 100644 >>> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md >>> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md >>> @@ -1031,10 +1031,30 @@ maximum command line length. The default >> value is 4096. >>> **Note:** The following `FLAGS` options are included in the response file: >>> `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, >> `ASLCC_FLAGS`, >>> and `ASM_FLAGS`. >>> ********** >>> >>> +#### 8.2.4.15 Build with Binary Cache >>> + >>> +**build** tool provides three new options for binary cache feature. >>> +--hash enables hash-based caching during build process. when --hash is >> enabled, >>> +build tool will base on the module hash value to do the incremental build, >> without >>> +--hash, build tool will base on the timestamp to do the incremental build. -- >> hash >>> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, >> build_rule.txt >>> +and build command are calculated as global hash value, Package DEC and its >> include >>> +header files are calculated as package hash value, Module source files and >> its INF >>> +file are calculated as module hash value. Library hash value will combine >> the global >>> +hash value and its dependent package hash value. Driver hash value will >> combine the >>> +global hash value, its dependent package hash value and its linked library >> hash value. >>> +When --hash and --binary-destination are specified, build tool will copy the >> generated >>> +binary files for each module into the directory specified by binary- >> destination at the >>> +build phase. Binary-destination directory caches all the generated binary >> files. >>> +When --hash and --binary-source are specified, build tool will try to get the >> binary >>> +files from the binary source directory at the build phase. If the cached >> binary has >>> +the same hash value, it will be directly used. Otherwise, build tool will >> compile the >>> +source files and generate the binary files. >>> + >>> ### 8.2.5 Post processing >>> >>> Once all files are parsed, the build tools will do following work for each EDK >>> II module: >>> >>> diff --git a/README.md b/README.md >>> index 52abb6a..ca59a35 100644 >>> --- a/README.md >>> +++ b/README.md >>> @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All rights >> reserved. >>> | | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523) Build >> spec: add EBNF for the --pcd syntax in the Section D.4 >> | | >>> | | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517) Build >> spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and >> BrotliCompress tool >> | | >>> | | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481) Build >> Spec: add clarification for not used Pcd that build tool will not do additional >> checks on its value >> | | >>> | | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518) Build >> Spec: Update Precedence of PCD Values >> | | >>> | | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669) Build >> Spec: Add multi-arg support to PREBUILD/POSTBUILD >> | | >>> +| | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689) Build >> spec: add description for build with binary cache >> | | >>> diff --git a/appendix_d_buildexe_command/d4_usage.md >> b/appendix_d_buildexe_command/d4_usage.md >>> index b71f2d0..c901266 100644 >>> --- a/appendix_d_buildexe_command/d4_usage.md >>> +++ b/appendix_d_buildexe_command/d4_usage.md >>> @@ -32,19 +32,20 @@ >>> ## D.4 Usage >>> >>> ```ini >>> Usage: build.exe [options] >>> [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run] >>> -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved. >>> +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved. >>> >>> Options: >>> --version show program's version number and exit >>> -h, --help show this help message and exit >>> -a TARGETARCH, --arch=TARGETARCH >>> - ARCHS is one of list: IA32, X64, IPF, ARM, or EBC, >>> - which overrides target.txt's TARGET_ARCH definition. >>> - To specify more archs, please repeat this option. >>> + ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or >>> + EBC, which overrides target.txt's TARGET_ARCH >>> + definition. To specify more archs, please repeat this >>> + option. >>> -p PLATFORMFILE, --platform=PLATFORMFILE >>> Build the platform specified by the DSC file name >>> argument, overriding target.txt's ACTIVE_PLATFORM >>> definition. >>> -m MODULEFILE, --module=MODULEFILE >>> @@ -112,10 +113,20 @@ Options: >>> -N, --no-cache Disable build cache mechanism >>> --conf=CONFDIRECTORY Specify the customized Conf directory. >>> --check-usage Check usage content of entries listed in INF file. >>> --ignore-sources Focus to a binary build and ignore all source files >>> --pcd=OPTIONPCD Set PCD value by command line. Format: >> "PcdName=Value" >>> + -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH >>> + Specify the maximum line length of build command. >>> + Default is 4096. >>> + --hash Enable hash-based caching during build process. >>> + --binary-destination=BINCACHEDEST >>> + Generate a cache of binary files in the specified >>> + directory. >>> + --binary-source=BINCACHESOURCE >>> + Consume a cache of binary files from the specified >>> + directory. >>> ``` >>> >>> ### D.4.1 Debug Levels >>> >>> The numeric debug levels are defined as integer values 0-9. >>> > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel >