Re: [Patch V2] Build spec: add description for build with binary cache

["Gao, Liming" <liming.gao@intel.com>]

Laszlo:
   I would like more people use it. Thank you! If you find any issue, please let me know. 

  When hash is enabled, AutoGen phase calculates hash value of every source files, INF, DSC and FDF, but has no AutoGen header and source file generation. So, AutoGen still takes some time. 

Thanks
Liming
>-----Original Message-----
>From: Laszlo Ersek [mailto:lersek@redhat.com]
>Sent: Thursday, September 28, 2017 7:15 PM
>To: Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong
><yonghong.zhu@intel.com>; edk2-devel@lists.01.org
>Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W
><kevin.w.shaw@intel.com>
>Subject: Re: [edk2] [Patch V2] Build spec: add description for build with binary
>cache
>
>On 09/28/17 11:02, Gao, Liming wrote:
>> Laszlo:
>>   Hash way may improve the incremental build performance. If hash value is
>not changed, module AutoGen and Make will be skipped.
>>   Time stamp way bases on Makefile. This way will run AutoGen every time.
>If no change, Makefile will not be updated.
>
>I did some tests, with ArmVirtQemu (AARCH64) and OVMF (Ia32 / Ia32X64 /
>X64).
>
>First I built all these platforms twice in a row, *without* --hash:
>- build ArmVirtQemu (AARCH64)
>- build OVMF (Ia32)
>- build OVMF (Ia32X64)
>- build OVMF (X64)
>- repeat all four builds in the same order, and now capture times in the
>report
>
>Then I built them twice in a row, *with* --hash:
>- build ArmVirtQemu (AARCH64)
>- build OVMF (Ia32)
>- build OVMF (Ia32X64)
>- build OVMF (X64)
>- repeat all four builds in the same order, and now capture times in the
>report
>
>The idea was that in the "repeat" builds, there was nothing to actually
>rebuild.
>
>Then we can compare how much time "--hash" saved, between the "repeat"
>builds:
>
>* without --hash:
>
>build.aa64virt.report:Build Duration:       00:00:15
>build.aa64virt.report:AutoGen Duration:     00:00:04
>build.aa64virt.report:Make Duration:        00:00:04
>build.aa64virt.report:GenFds Duration:      00:00:06
>
>build.ovmf.32.report:Build Duration:       00:00:16
>build.ovmf.32.report:AutoGen Duration:     00:00:04
>build.ovmf.32.report:Make Duration:        00:00:06
>build.ovmf.32.report:GenFds Duration:      00:00:06
>
>build.ovmf.3264.report:Build Duration:       00:00:20
>build.ovmf.3264.report:AutoGen Duration:     00:00:06
>build.ovmf.3264.report:Make Duration:        00:00:07
>build.ovmf.3264.report:GenFds Duration:      00:00:07
>
>build.ovmf.report:Build Duration:       00:00:18
>build.ovmf.report:AutoGen Duration:     00:00:04
>build.ovmf.report:Make Duration:        00:00:06
>build.ovmf.report:GenFds Duration:      00:00:06
>
>* With --hash:
>
>build.aa64virt.report:Build Duration:       00:00:11
>build.aa64virt.report:AutoGen Duration:     00:00:04
>build.aa64virt.report:GenFds Duration:      00:00:06
>
>build.ovmf.32.report:Build Duration:       00:00:11
>build.ovmf.32.report:AutoGen Duration:     00:00:04
>build.ovmf.32.report:GenFds Duration:      00:00:06
>
>build.ovmf.3264.report:Build Duration:       00:00:14
>build.ovmf.3264.report:AutoGen Duration:     00:00:06
>build.ovmf.3264.report:GenFds Duration:      00:00:07
>
>build.ovmf.report:Build Duration:       00:00:12
>build.ovmf.report:AutoGen Duration:     00:00:05
>build.ovmf.report:GenFds Duration:      00:00:06
>
>With "--hash", the "Make" step disappeared entirely (which is very
>welcome, and I'll start using --hash today).
>
>However, the "AutoGen" step took just as long. Can you think of a reason
>why? (Above you mention that AutoGen should be skipped too.)
>
>Thanks!
>Laszlo
>
>
>>> -----Original Message-----
>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>> Sent: Thursday, September 28, 2017 4:19 PM
>>> To: Zhu, Yonghong <yonghong.zhu@intel.com>; edk2-devel@lists.01.org
>>> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Shaw, Kevin W
>>> <kevin.w.shaw@intel.com>; Gao, Liming <liming.gao@intel.com>
>>> Subject: Re: [edk2] [Patch V2] Build spec: add description for build with
>binary
>>> cache
>>>
>>> On 09/19/17 08:48, Yonghong Zhu wrote:
>>>> V2:
>>>> change the option name to --binary-destination and --binary-source.
>>>>
>>>> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689
>>>
>>> What are the benefits of a hash-based incremental build over a
>>> timestamp-based incremental build?
>>>
>>> Thank you,
>>> Laszlo
>>>
>>>> Cc: Liming Gao <liming.gao@intel.com>
>>>> Cc: Michael Kinney <michael.d.kinney@intel.com>
>>>> Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
>>>> Contributed-under: TianoCore Contribution Agreement 1.1
>>>> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
>>>> ---
>>>>  .../82_auto-generation_process.md                    | 20
>>> ++++++++++++++++++++
>>>>  README.md                                            |  1 +
>>>>  appendix_d_buildexe_command/d4_usage.md              | 19
>>> +++++++++++++++----
>>>>  3 files changed, 36 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md
>>> b/8_pre-build_autogen_stage/82_auto-generation_process.md
>>>> index 671a7d5..f2ddf32 100644
>>>> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md
>>>> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md
>>>> @@ -1031,10 +1031,30 @@ maximum command line length.  The default
>>> value is 4096.
>>>>  **Note:** The following `FLAGS` options are included in the response
>file:
>>>>  `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`,
>>> `ASLCC_FLAGS`,
>>>>  and `ASM_FLAGS`.
>>>>  **********
>>>>
>>>> +#### 8.2.4.15 Build with Binary Cache
>>>> +
>>>> +**build** tool provides three new options for binary cache feature.
>>>> +--hash enables hash-based caching during build process. when --hash is
>>> enabled,
>>>> +build tool will base on the module hash value to do the incremental build,
>>> without
>>>> +--hash, build tool will base on the timestamp to do the incremental build.
>--
>>> hash
>>>> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt,
>>> build_rule.txt
>>>> +and build command are calculated as global hash value, Package DEC and
>its
>>> include
>>>> +header files are calculated as package hash value, Module source files
>and
>>> its INF
>>>> +file are calculated as module hash value. Library hash value will combine
>>> the global
>>>> +hash value and its dependent package hash value. Driver hash value will
>>> combine the
>>>> +global hash value, its dependent package hash value and its linked
>library
>>> hash value.
>>>> +When --hash and --binary-destination are specified, build tool will copy
>the
>>> generated
>>>> +binary files for each module into the directory specified by binary-
>>> destination at the
>>>> +build phase. Binary-destination directory caches all the generated binary
>>> files.
>>>> +When --hash and --binary-source are specified, build tool will try to get
>the
>>> binary
>>>> +files from the binary source directory at the build phase. If the cached
>>> binary has
>>>> +the same hash value, it will be directly used. Otherwise, build tool will
>>> compile the
>>>> +source files and generate the binary files.
>>>> +
>>>>  ### 8.2.5 Post processing
>>>>
>>>>  Once all files are parsed, the build tools will do following work for each
>EDK
>>>>  II module:
>>>>
>>>> diff --git a/README.md b/README.md
>>>> index 52abb6a..ca59a35 100644
>>>> --- a/README.md
>>>> +++ b/README.md
>>>> @@ -215,5 +215,6 @@ Copyright (c) 2008-2017, Intel Corporation. All
>rights
>>> reserved.
>>>>  |            | [#523](https://bugzilla.tianocore.org/show_bug.cgi?id=523)
>Build
>>> spec: add EBNF for the --pcd syntax in the Section D.4
>>> |               |
>>>>  |            | [#517](https://bugzilla.tianocore.org/show_bug.cgi?id=517)
>Build
>>> spec: chapter 5.2.2 Guided Tools add description for Pkcs7Sign tool and
>>> BrotliCompress tool
>>> |               |
>>>>  |            | [#481](https://bugzilla.tianocore.org/show_bug.cgi?id=481)
>Build
>>> Spec: add clarification for not used Pcd that build tool will not do additional
>>> checks on its value
>>> |               |
>>>>  |            | [#518](https://bugzilla.tianocore.org/show_bug.cgi?id=518)
>Build
>>> Spec: Update Precedence of PCD Values
>>> |               |
>>>>  |            | [#669](https://bugzilla.tianocore.org/show_bug.cgi?id=669)
>Build
>>> Spec: Add multi-arg support to PREBUILD/POSTBUILD
>>> |               |
>>>> +|            | [#689](https://bugzilla.tianocore.org/show_bug.cgi?id=689)
>Build
>>> spec: add description for build with binary cache
>>> |               |
>>>> diff --git a/appendix_d_buildexe_command/d4_usage.md
>>> b/appendix_d_buildexe_command/d4_usage.md
>>>> index b71f2d0..c901266 100644
>>>> --- a/appendix_d_buildexe_command/d4_usage.md
>>>> +++ b/appendix_d_buildexe_command/d4_usage.md
>>>> @@ -32,19 +32,20 @@
>>>>  ## D.4 Usage
>>>>
>>>>  ```ini
>>>>  Usage: build.exe [options]
>>>>  [all|fds|genc|genmake|clean|cleanall|cleanlib|modules|libraries|run]
>>>> -Copyright (c) 2007 - 2014, Intel Corporation All rights reserved.
>>>> +Copyright (c) 2007 - 2017, Intel Corporation All rights reserved.
>>>>
>>>>  Options:
>>>>    --version             show program's version number and exit
>>>>    -h, --help            show this help message and exit
>>>>    -a TARGETARCH, --arch=TARGETARCH
>>>> -                        ARCHS is one of list: IA32, X64, IPF, ARM, or EBC,
>>>> -                        which overrides target.txt's TARGET_ARCH definition.
>>>> -                        To specify more archs, please repeat this option.
>>>> +                        ARCHS is one of list: IA32, X64, IPF, ARM, AARCH64 or
>>>> +                        EBC, which overrides target.txt's TARGET_ARCH
>>>> +                        definition. To specify more archs, please repeat this
>>>> +                        option.
>>>>    -p PLATFORMFILE, --platform=PLATFORMFILE
>>>>                          Build the platform specified by the DSC file name
>>>>                          argument, overriding target.txt's ACTIVE_PLATFORM
>>>>                          definition.
>>>>    -m MODULEFILE, --module=MODULEFILE
>>>> @@ -112,10 +113,20 @@ Options:
>>>>    -N, --no-cache        Disable build cache mechanism
>>>>    --conf=CONFDIRECTORY  Specify the customized Conf directory.
>>>>    --check-usage         Check usage content of entries listed in INF file.
>>>>    --ignore-sources      Focus to a binary build and ignore all source files
>>>>    --pcd=OPTIONPCD       Set PCD value by command line. Format:
>>> "PcdName=Value"
>>>> +  -l COMMANDLENGTH, --cmd-len=COMMANDLENGTH
>>>> +                        Specify the maximum line length of build command.
>>>> +                        Default is 4096.
>>>> +  --hash                Enable hash-based caching during build process.
>>>> +  --binary-destination=BINCACHEDEST
>>>> +                        Generate a cache of binary files in the specified
>>>> +                        directory.
>>>> +  --binary-source=BINCACHESOURCE
>>>> +                        Consume a cache of binary files from the specified
>>>> +                        directory.
>>>>  ```
>>>>
>>>>  ### D.4.1 Debug Levels
>>>>
>>>>  The numeric debug levels are defined as integer values 0-9.
>>>>
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>>