From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.31, mailfrom: steven.shi@intel.com) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by groups.io with SMTP; Tue, 28 May 2019 17:59:21 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 May 2019 17:59:20 -0700 X-ExtLoop1: 1 Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by orsmga003.jf.intel.com with ESMTP; 28 May 2019 17:59:20 -0700 Received: from shsmsx106.ccr.corp.intel.com (10.239.4.159) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 28 May 2019 17:59:19 -0700 Received: from shsmsx102.ccr.corp.intel.com ([169.254.2.249]) by SHSMSX106.ccr.corp.intel.com ([169.254.10.213]) with mapi id 14.03.0415.000; Wed, 29 May 2019 08:59:18 +0800 From: "Steven Shi" To: "Rodriguez, Christian" , "devel@edk2.groups.io" CC: "Gao, Liming" , "Feng, Bob C" , "Fan, ZhijuX" Subject: Re: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache Thread-Topic: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache Thread-Index: AQHVFTKespSCsBBR+UyRCjIO0gFYPqaAo3uwgACkaEA= Date: Wed, 29 May 2019 00:59:17 +0000 Message-ID: <06C8AB66E78EE34A949939824ABE2B313FFDC3E0@shsmsx102.ccr.corp.intel.com> References: <20190528085139.14716-1-steven.shi@intel.com> <3A7DCC9A944C6149BF832E1C9B718ABC01F245AE@ORSMSX112.amr.corp.intel.com> In-Reply-To: <3A7DCC9A944C6149BF832E1C9B718ABC01F245AE@ORSMSX112.amr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjJjNGZkMzAtMjg2Ni00M2FjLWIwYjAtYzk5Y2Y1MGRhZjBiIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiOE5NU1MxXC9NRlpZeGpvaEdyenpaR0c2RjVlRVJCRXN2YmRwZjJBM2pvY0FvRWpoZXRzaitSeExmQjRFdmlXSVgifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: steven.shi@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable OK, will send the new version patch to simplify it. Thanks Steven Shi Intel\SSG\FID\Firmware Infrastructure > -----Original Message----- > From: Rodriguez, Christian > Sent: Tuesday, May 28, 2019 11:16 PM > To: devel@edk2.groups.io; Shi, Steven > Cc: Gao, Liming ; Feng, Bob C ; > Fan, ZhijuX > Subject: RE: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to > support library cache >=20 > Hi Steven, >=20 > After testing the bug reported, it seems that the dependency problem isn= 't > the root cause. We don't need to change the way it skips having to rebui= ld > libraries. The cache is missing the library build artifacts as you menti= oned > before and this is the root cause of the rebuild failure. This patch sho= uld fix > that, but you can simplify it by not adding a new tracking system and ke= eping > the dependency reduction as before. That will save the performance becau= se > it doesn't have to do extra checking. >=20 > Please see attached patch for minimal change needed. >=20 > Thanks, > Christian >=20 > >-----Original Message----- > >From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of > >Steven Shi > >Sent: Tuesday, May 28, 2019 1:52 AM > >To: devel@edk2.groups.io > >Cc: Gao, Liming ; Feng, Bob C > >; Rodriguez, Christian > >; Fan, ZhijuX > >Subject: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to supp= ort > >library cache > > > >https://bugzilla.tianocore.org/show_bug.cgi?id=3D1797 > > > >Current binary cache doesn't support to save and restore > >the library module. If a driver module cache miss happen, > >all its dependency library modules need rebuild which > >is very time-consuming. This patch is to entend the binary > >cache to support library. > > > >Cc: Liming Gao > >Cc: Bob Feng > >Cc: Christian Rodriguez > >Signed-off-by: Steven Shi > >--- > > BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++++++--- > > BaseTools/Source/Python/build/build.py | 3 ++- > > 2 files changed, 16 insertions(+), 4 deletions(-) > > > >diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py > >b/BaseTools/Source/Python/AutoGen/AutoGen.py > >index a5bef4f7c6..aeb63f52c5 100644 > >--- a/BaseTools/Source/Python/AutoGen/AutoGen.py > >+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py > >@@ -2578,7 +2578,7 @@ class ModuleAutoGen(AutoGen): > > self.AutoGenDepSet =3D set() > > self.ReferenceModules =3D [] > > self.ConstPcd =3D {} > >- > >+ self.CacheRestored =3D False > > > > def __repr__(self): > > return "%s [%s]" % (self.MetaFile, self.Arch) > >@@ -3906,6 +3906,12 @@ class ModuleAutoGen(AutoGen): > > ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf= ') > > if os.path.exists(ModuleFile): > > shutil.copy2(ModuleFile, FileDir) > >+ else: > >+ OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') > >+ DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') > >+ for Item in self.CodaTargetList: > >+ File =3D Item.Target.Path.replace('\\', '/').strip('/'= ).replace(DebugDir, > >'').replace(OutputDir, '').strip('/') > >+ self.OutputFile.add(File) > > if not self.OutputFile: > > Ma =3D self.BuildDatabase[self.MetaFile, self.Arch, self.B= uildTarget, > >self.ToolChain] > > self.OutputFile =3D Ma.Binaries > >@@ -3949,6 +3955,7 @@ class ModuleAutoGen(AutoGen): > > destination_dir =3D os.path.dirname(de= stination_file) > > CreateDirectory(destination_dir) > > shutil.copy2(File, destination_dir) > >+ self.CacheRestored =3D True > > if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D = "PcdDxe": > > CreatePcdDatabaseCode(self, TemplateString(), > >TemplateString()) > > return True > >@@ -3987,7 +3994,9 @@ class ModuleAutoGen(AutoGen): > > self.GenFfsList =3D GenFfsList > > if not self.IsLibrary and CreateLibraryMakeFile: > > for LibraryAutoGen in self.LibraryAutoGenList: > >- LibraryAutoGen.CreateMakeFile() > >+ # Only create makefile for libraries which have not be= en restored > >+ if not LibraryAutoGen.CacheRestored: > >+ LibraryAutoGen.CreateMakeFile() > > > > if self.CanSkip(): > > return > >@@ -4030,7 +4039,9 @@ class ModuleAutoGen(AutoGen): > > > > if not self.IsLibrary and CreateLibraryCodeFile: > > for LibraryAutoGen in self.LibraryAutoGenList: > >- LibraryAutoGen.CreateCodeFile() > >+ # Only create autogen code for libraries which have no= t been > >restored > >+ if not LibraryAutoGen.CacheRestored: > >+ LibraryAutoGen.CreateCodeFile() > > > > if self.CanSkip(): > > return > >diff --git a/BaseTools/Source/Python/build/build.py > >b/BaseTools/Source/Python/build/build.py > >index 80ceb98310..f1f4c07980 100644 > >--- a/BaseTools/Source/Python/build/build.py > >+++ b/BaseTools/Source/Python/build/build.py > >@@ -341,7 +341,8 @@ class ModuleMakeUnit(BuildUnit): > > # @param Target The build target name, one of gSupportedTa= rget > > # > > def __init__(self, Obj, Target): > >- Dependency =3D [ModuleMakeUnit(La, Target) for La in > >Obj.LibraryAutoGenList] > >+ # Skip the dependency modules which are already restored from = cache > >+ Dependency =3D [ModuleMakeUnit(La, Target) for La in > >Obj.LibraryAutoGenList if not La.CacheRestored] > > BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Depend= ency, > >Obj.MakeFileDir) > > if Target in [None, "", "all"]: > > self.Target =3D "tbuild" > >-- > >2.17.1.windows.2 > > > > > >