From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.132.183.28; helo=mx1.redhat.com; envelope-from=lersek@redhat.com; receiver=edk2-devel@lists.01.org Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 55F3D21F2AF77 for ; Thu, 28 Sep 2017 05:03:24 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 49630C01A739; Thu, 28 Sep 2017 12:06:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 49630C01A739 Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=lersek@redhat.com Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-73.rdu2.redhat.com [10.10.120.73]) by smtp.corp.redhat.com (Postfix) with ESMTP id B3BDA63BCF; Thu, 28 Sep 2017 12:06:36 +0000 (UTC) To: Yonghong Zhu , edk2-devel@lists.01.org Cc: Michael Kinney , Kevin W Shaw , Liming Gao References: <1505803680-15860-1-git-send-email-yonghong.zhu@intel.com> From: Laszlo Ersek Message-ID: <0dbfc584-f13c-c712-0ef5-bf1741b89bcd@redhat.com> Date: Thu, 28 Sep 2017 14:06:35 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1505803680-15860-1-git-send-email-yonghong.zhu@intel.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 28 Sep 2017 12:06:38 +0000 (UTC) Subject: Re: [Patch V2] Build spec: add description for build with binary cache X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Sep 2017 12:03:24 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 09/19/17 08:48, Yonghong Zhu wrote: > V2: > change the option name to --binary-destination and --binary-source. > > fixes:https://bugzilla.tianocore.org/show_bug.cgi?id=689 > Cc: Liming Gao > Cc: Michael Kinney > Cc: Kevin W Shaw > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Yonghong Zhu > --- > .../82_auto-generation_process.md | 20 ++++++++++++++++++++ > README.md | 1 + > appendix_d_buildexe_command/d4_usage.md | 19 +++++++++++++++---- > 3 files changed, 36 insertions(+), 4 deletions(-) > > diff --git a/8_pre-build_autogen_stage/82_auto-generation_process.md b/8_pre-build_autogen_stage/82_auto-generation_process.md > index 671a7d5..f2ddf32 100644 > --- a/8_pre-build_autogen_stage/82_auto-generation_process.md > +++ b/8_pre-build_autogen_stage/82_auto-generation_process.md > @@ -1031,10 +1031,30 @@ maximum command line length. The default value is 4096. > **Note:** The following `FLAGS` options are included in the response file: > `PP_FLAGS`, `CC_FLAGS`, `VFRPP_FLAGS`, `APP_FLAGS`, `ASLPP_FLAGS`, `ASLCC_FLAGS`, > and `ASM_FLAGS`. > ********** > > +#### 8.2.4.15 Build with Binary Cache > + > +**build** tool provides three new options for binary cache feature. > +--hash enables hash-based caching during build process. when --hash is enabled, > +build tool will base on the module hash value to do the incremental build, without > +--hash, build tool will base on the timestamp to do the incremental build. --hash > +option use md5 method to get every hash value, DSC/FDF, tools_def.txt, build_rule.txt > +and build command are calculated as global hash value, Package DEC and its include > +header files are calculated as package hash value, Module source files and its INF > +file are calculated as module hash value. Library hash value will combine the global > +hash value and its dependent package hash value. Driver hash value will combine the > +global hash value, its dependent package hash value and its linked library hash value. > +When --hash and --binary-destination are specified, build tool will copy the generated > +binary files for each module into the directory specified by binary-destination at the > +build phase. Binary-destination directory caches all the generated binary files. > +When --hash and --binary-source are specified, build tool will try to get the binary > +files from the binary source directory at the build phase. If the cached binary has > +the same hash value, it will be directly used. Otherwise, build tool will compile the > +source files and generate the binary files. > + I have another question about this feature: how can I invalidate the binary cache, so that the next invocation with "--hash" fall back to the timestamp-based incremental build? Is it enough if I remove all "*.hash" files from the Build tree? This question is relevant for bisection. Assume that we are bisecting a commit range that straddles the introduction of "--hash" to BaseTools, and that we use a build script that uses "--hash" whenever it is available. Consider the following build order: (1) Build the tree with "--hash" (because "--hash" is available at the commit that we're testing). Assume this build displays the symptom we are bisecting, so we enter "git bisect bad", and git checks out an earlier commit. (2) Assume BaseTools lacks "--hash" at this earlier commit, so we build the tree without "--hash". Using the timestamp checks, the modules changed or affected by the checkout done by git-bisect in step (1) will be rebuilt correctly. We test this build and find that it works okay, so we enter "git bisect good". Git checks out a later commit (such that it is earlier than the one tested in (1)). (3) Assume that at this commit, "--hash" is again available, so we build the tree with "--hash". This is incorrect however, because the cached values come from step (1), not step (2). This means that every time the build script notices that "--hash" is unavailable, it must invalidate (delete) all the cached values, so that *next time* it returns to using "--hash", all the hash values are recalculated from scratch, and only the timestamp based checks are used for that build. So, how can I invalidate all the cached values? Is it enough to delete the *.hash files? Thanks, Laszlo