Re: [Patch V2] Build spec: add description for build with binary cache

[Laszlo Ersek <lersek@redhat.com>]

On 09/19/17 08:48, Yonghong Zhu wrote:
> V2:
> change the option name to --binary-destination and --binary-source.
> 
> fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Michael Kinney <michael.d.kinney@intel.com>
> Cc: Kevin W Shaw <kevin.w.shaw@intel.com>
> Contributed-under: TianoCore Contribution Agreement 1.1
> Signed-off-by: Yonghong Zhu <yonghong.zhu@intel.com>
> ---
>  .../82_auto-generation_process.md                    | 20 ++++++++++++++++++++
>  README.md                                            |  1 +
>  appendix_d_buildexe_command/d4_usage.md              | 19 +++++++++++++++----
>  3 files changed, 36 insertions(+), 4 deletions(-)
> 
> diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md b/8_pre-build_autogen_stage/82_auto-generation_process.md
> index 671a7d5..f2ddf32 100644
> --- a/8_pre-build_autogen_stage/82_auto-generation_process.md
> +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md
> @@ -1031,10 +1031,30 @@ maximum command line length.  The default value is 4096.
>  **Note:** The following `FLAGS` options are included in the response file:
>  `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, `ASLCC_FLAGS`,
>  and `ASM_FLAGS`.
>  **********
>  
> +#### 8.2.4.15 Build with Binary Cache
> +
> +**build** tool provides three new options for binary cache feature.
> +--hash enables hash-based caching during build process. when --hash is enabled,
> +build tool will base on the module hash value to do the incremental build, without
> +--hash, build tool will base on the timestamp to do the incremental build. --hash
> +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, build_rule.txt
> +and build command are calculated as global hash value, Package DEC and its include
> +header files are calculated as package hash value, Module source files and its INF
> +file are calculated as module hash value. Library hash value will combine the global
> +hash value and its dependent package hash value. Driver hash value will combine the
> +global hash value, its dependent package hash value and its linked library hash value.
> +When --hash and --binary-destination are specified, build tool will copy the generated
> +binary files for each module into the directory specified by binary-destination at the
> +build phase. Binary-destination directory caches all the generated binary files.
> +When --hash and --binary-source are specified, build tool will try to get the binary
> +files from the binary source directory at the build phase. If the cached binary has
> +the same hash value, it will be directly used. Otherwise, build tool will compile the
> +source files and generate the binary files.
> +

I have another question about this feature: how can I invalidate the
binary cache, so that the next invocation with "--hash" fall back to the
timestamp-based incremental build?

Is it enough if I remove all "*.hash" files from the Build tree?

This question is relevant for bisection. Assume that we are bisecting a
commit range that straddles the introduction of "--hash" to BaseTools,
and that we use a build script that uses "--hash" whenever it is available.

Consider the following build order:

(1) Build the tree with "--hash" (because "--hash" is available at the
commit that we're testing). Assume this build displays the symptom we
are bisecting, so we enter "git bisect bad", and git checks out an
earlier commit.

(2) Assume BaseTools lacks "--hash" at this earlier commit, so we build
the tree without "--hash". Using the timestamp checks, the modules
changed or affected by the checkout done by git-bisect in step (1) will
be rebuilt correctly. We test this build and find that it works okay, so
we enter "git bisect good". Git checks out a later commit (such that it
is earlier than the one tested in (1)).

(3) Assume that at this commit, "--hash" is again available, so we build
the tree with "--hash". This is incorrect however, because the cached
values come from step (1), not step (2).


This means that every time the build script notices that "--hash" is
unavailable, it must invalidate (delete) all the cached values, so that
*next time* it returns to using "--hash", all the hash values are
recalculated from scratch, and only the timestamp based checks are used
for that build.

So, how can I invalidate all the cached values? Is it enough to delete
the *.hash files?

Thanks,
Laszlo