From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.65, mailfrom: christian.rodriguez@intel.com) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by groups.io with SMTP; Tue, 28 May 2019 08:15:45 -0700 X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 May 2019 08:15:44 -0700 X-ExtLoop1: 1 Received: from orsmsx104.amr.corp.intel.com ([10.22.225.131]) by fmsmga004.fm.intel.com with ESMTP; 28 May 2019 08:15:44 -0700 Received: from orsmsx112.amr.corp.intel.com ([169.254.3.79]) by ORSMSX104.amr.corp.intel.com ([169.254.4.200]) with mapi id 14.03.0415.000; Tue, 28 May 2019 08:15:43 -0700 From: "Christian Rodriguez" To: "devel@edk2.groups.io" , "Shi, Steven" CC: "Gao, Liming" , "Feng, Bob C" , "Fan, ZhijuX" Subject: Re: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache Thread-Topic: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache Thread-Index: AQHVFTKehti3KGVfH0yG9NGlWnhWeaaAo3uw Date: Tue, 28 May 2019 15:15:43 +0000 Message-ID: <3A7DCC9A944C6149BF832E1C9B718ABC01F245AE@ORSMSX112.amr.corp.intel.com> References: <20190528085139.14716-1-steven.shi@intel.com> In-Reply-To: <20190528085139.14716-1-steven.shi@intel.com> Accept-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjJjNGZkMzAtMjg2Ni00M2FjLWIwYjAtYzk5Y2Y1MGRhZjBiIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiOE5NU1MxXC9NRlpZeGpvaEdyenpaR0c2RjVlRVJCRXN2YmRwZjJBM2pvY0FvRWpoZXRzaitSeExmQjRFdmlXSVgifQ== dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.22.254.138] MIME-Version: 1.0 X-Groupsio-MsgNum: 41513 Content-Language: en-US Content-Type: multipart/mixed; boundary="_002_3A7DCC9A944C6149BF832E1C9B718ABC01F245AEORSMSX112amrcor_" --_002_3A7DCC9A944C6149BF832E1C9B718ABC01F245AEORSMSX112amrcor_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Steven, After testing the bug reported, it seems that the dependency problem isn't= the root cause. We don't need to change the way it skips having to rebuild= libraries. The cache is missing the library build artifacts as you mention= ed before and this is the root cause of the rebuild failure. This patch sho= uld fix that, but you can simplify it by not adding a new tracking system a= nd keeping the dependency reduction as before. That will save the performan= ce because it doesn't have to do extra checking. Please see attached patch for minimal change needed. Thanks, Christian >-----Original Message----- >From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of >Steven Shi >Sent: Tuesday, May 28, 2019 1:52 AM >To: devel@edk2.groups.io >Cc: Gao, Liming ; Feng, Bob C >; Rodriguez, Christian >; Fan, ZhijuX >Subject: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to suppor= t >library cache > >https://bugzilla.tianocore.org/show_bug.cgi?id=3D1797 > >Current binary cache doesn't support to save and restore >the library module. If a driver module cache miss happen, >all its dependency library modules need rebuild which >is very time-consuming. This patch is to entend the binary >cache to support library. > >Cc: Liming Gao >Cc: Bob Feng >Cc: Christian Rodriguez >Signed-off-by: Steven Shi >--- > BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++++++--- > BaseTools/Source/Python/build/build.py | 3 ++- > 2 files changed, 16 insertions(+), 4 deletions(-) > >diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py >b/BaseTools/Source/Python/AutoGen/AutoGen.py >index a5bef4f7c6..aeb63f52c5 100644 >--- a/BaseTools/Source/Python/AutoGen/AutoGen.py >+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py >@@ -2578,7 +2578,7 @@ class ModuleAutoGen(AutoGen): > self.AutoGenDepSet =3D set() > self.ReferenceModules =3D [] > self.ConstPcd =3D {} >- >+ self.CacheRestored =3D False > > def __repr__(self): > return "%s [%s]" % (self.MetaFile, self.Arch) >@@ -3906,6 +3906,12 @@ class ModuleAutoGen(AutoGen): > ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf') > if os.path.exists(ModuleFile): > shutil.copy2(ModuleFile, FileDir) >+ else: >+ OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') >+ DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') >+ for Item in self.CodaTargetList: >+ File =3D Item.Target.Path.replace('\\', '/').strip('/').= replace(DebugDir, >'').replace(OutputDir, '').strip('/') >+ self.OutputFile.add(File) > if not self.OutputFile: > Ma =3D self.BuildDatabase[self.MetaFile, self.Arch, self.Bui= ldTarget, >self.ToolChain] > self.OutputFile =3D Ma.Binaries >@@ -3949,6 +3955,7 @@ class ModuleAutoGen(AutoGen): > destination_dir =3D os.path.dirname(dest= ination_file) > CreateDirectory(destination_dir) > shutil.copy2(File, destination_dir) >+ self.CacheRestored =3D True > if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "P= cdDxe": > CreatePcdDatabaseCode(self, TemplateString(), >TemplateString()) > return True >@@ -3987,7 +3994,9 @@ class ModuleAutoGen(AutoGen): > self.GenFfsList =3D GenFfsList > if not self.IsLibrary and CreateLibraryMakeFile: > for LibraryAutoGen in self.LibraryAutoGenList: >- LibraryAutoGen.CreateMakeFile() >+ # Only create makefile for libraries which have not been= restored >+ if not LibraryAutoGen.CacheRestored: >+ LibraryAutoGen.CreateMakeFile() > > if self.CanSkip(): > return >@@ -4030,7 +4039,9 @@ class ModuleAutoGen(AutoGen): > > if not self.IsLibrary and CreateLibraryCodeFile: > for LibraryAutoGen in self.LibraryAutoGenList: >- LibraryAutoGen.CreateCodeFile() >+ # Only create autogen code for libraries which have not = been >restored >+ if not LibraryAutoGen.CacheRestored: >+ LibraryAutoGen.CreateCodeFile() > > if self.CanSkip(): > return >diff --git a/BaseTools/Source/Python/build/build.py >b/BaseTools/Source/Python/build/build.py >index 80ceb98310..f1f4c07980 100644 >--- a/BaseTools/Source/Python/build/build.py >+++ b/BaseTools/Source/Python/build/build.py >@@ -341,7 +341,8 @@ class ModuleMakeUnit(BuildUnit): > # @param Target The build target name, one of gSupportedTarg= et > # > def __init__(self, Obj, Target): >- Dependency =3D [ModuleMakeUnit(La, Target) for La in >Obj.LibraryAutoGenList] >+ # Skip the dependency modules which are already restored from ca= che >+ Dependency =3D [ModuleMakeUnit(La, Target) for La in >Obj.LibraryAutoGenList if not La.CacheRestored] > BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Dependen= cy, >Obj.MakeFileDir) > if Target in [None, "", "all"]: > self.Target =3D "tbuild" >-- >2.17.1.windows.2 > > > --_002_3A7DCC9A944C6149BF832E1C9B718ABC01F245AEORSMSX112amrcor_ Content-Type: application/octet-stream; name="0001-Add-library-copy-to-cache.patch" Content-Description: 0001-Add-library-copy-to-cache.patch Content-Disposition: attachment; filename="0001-Add-library-copy-to-cache.patch"; size=1335; creation-date="Tue, 28 May 2019 15:13:59 GMT"; modification-date="Tue, 28 May 2019 15:13:59 GMT" Content-Transfer-Encoding: base64 RnJvbSA5NzEyNzgyODM4NWMwOTUwMTNhOTBhMjBmZDhjNDdlN2UzZWEwYzJkIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBDaHJpc3RpYW4gUm9kcmlndWV6IDxjaHJpc3RpYW4ucm9kcmln dWV6QGludGVsLmNvbT4KRGF0ZTogVHVlLCAyOCBNYXkgMjAxOSAwNzo1ODowNSAtMDcwMApTdWJq ZWN0OiBbUEFUQ0hdIEFkZCBsaWJyYXJ5IGNvcHkgdG8gY2FjaGUKCi0tLQogQmFzZVRvb2xzL1Nv dXJjZS9QeXRob24vQXV0b0dlbi9BdXRvR2VuLnB5IHwgNiArKysrKysKIDEgZmlsZSBjaGFuZ2Vk LCA2IGluc2VydGlvbnMoKykKCmRpZmYgLS1naXQgYS9CYXNlVG9vbHMvU291cmNlL1B5dGhvbi9B dXRvR2VuL0F1dG9HZW4ucHkgYi9CYXNlVG9vbHMvU291cmNlL1B5dGhvbi9BdXRvR2VuL0F1dG9H ZW4ucHkKaW5kZXggYTM3NmJjMjRkNi4uNGU0Nzk0ZmQ5MyAxMDA2NDQKLS0tIGEvQmFzZVRvb2xz L1NvdXJjZS9QeXRob24vQXV0b0dlbi9BdXRvR2VuLnB5CisrKyBiL0Jhc2VUb29scy9Tb3VyY2Uv UHl0aG9uL0F1dG9HZW4vQXV0b0dlbi5weQpAQCAtMzkwNiw2ICszOTA2LDEyIEBAIGNsYXNzIE1v ZHVsZUF1dG9HZW4oQXV0b0dlbik6CiAgICAgICAgICAgICBNb2R1bGVGaWxlID0gcGF0aC5qb2lu KHNlbGYuT3V0cHV0RGlyLCBzZWxmLk5hbWUgKyAnLmluZicpDQogICAgICAgICAgICAgaWYgb3Mu cGF0aC5leGlzdHMoTW9kdWxlRmlsZSk6DQogICAgICAgICAgICAgICAgIHNodXRpbC5jb3B5MihN b2R1bGVGaWxlLCBGaWxlRGlyKQ0KKyAgICAgICAgZWxzZToNCisgICAgICAgICAgICBPdXRwdXRE aXIgPSBzZWxmLk91dHB1dERpci5yZXBsYWNlKCdcXCcsICcvJykuc3RyaXAoJy8nKQ0KKyAgICAg ICAgICAgIERlYnVnRGlyID0gc2VsZi5EZWJ1Z0Rpci5yZXBsYWNlKCdcXCcsICcvJykuc3RyaXAo Jy8nKQ0KKyAgICAgICAgICAgIGZvciBJdGVtIGluIHNlbGYuQ29kYVRhcmdldExpc3Q6DQorICAg ICAgICAgICAgICAgIEZpbGUgPSBJdGVtLlRhcmdldC5QYXRoLnJlcGxhY2UoJ1xcJywgJy8nKS5z dHJpcCgnLycpLnJlcGxhY2UoRGVidWdEaXIsICcnKS5yZXBsYWNlKE91dHB1dERpciwgJycpLnN0 cmlwKCcvJykNCisgICAgICAgICAgICAgICAgc2VsZi5PdXRwdXRGaWxlLmFkZChGaWxlKQ0KICAg ICAgICAgaWYgbm90IHNlbGYuT3V0cHV0RmlsZToNCiAgICAgICAgICAgICBNYSA9IHNlbGYuQnVp bGREYXRhYmFzZVtzZWxmLk1ldGFGaWxlLCBzZWxmLkFyY2gsIHNlbGYuQnVpbGRUYXJnZXQsIHNl bGYuVG9vbENoYWluXQ0KICAgICAgICAgICAgIHNlbGYuT3V0cHV0RmlsZSA9IE1hLkJpbmFyaWVz DQotLSAKMi4yMS4wLndpbmRvd3MuMQoK --_002_3A7DCC9A944C6149BF832E1C9B718ABC01F245AEORSMSX112amrcor_--