Re: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache

["Christian Rodriguez" <christian.rodriguez@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 5532 bytes --]

Hi Steven,

After testing the bug reported, it seems that the dependency problem isn't the root cause. We don't need to change the way it skips having to rebuild libraries. The cache is missing the library build artifacts as you mentioned before and this is the root cause of the rebuild failure. This patch should fix that, but you can simplify it by not adding a new tracking system and keeping the dependency reduction as before. That will save the performance because it doesn't have to do extra checking.

Please see attached patch for minimal change needed.

Thanks,
Christian

>-----Original Message-----
>From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
>Steven Shi
>Sent: Tuesday, May 28, 2019 1:52 AM
>To: devel@edk2.groups.io
>Cc: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
><bob.c.feng@intel.com>; Rodriguez, Christian
><christian.rodriguez@intel.com>; Fan, ZhijuX <zhijux.fan@intel.com>
>Subject: [edk2-devel] [PATCH] BaseTools:Extend the binary cache to support
>library cache
>
>https://bugzilla.tianocore.org/show_bug.cgi?id=1797
>
>Current binary cache doesn't support to save and restore
>the library module. If a driver module cache miss happen,
>all its dependency library modules need rebuild which
>is very time-consuming. This patch is to entend the binary
>cache to support library.
>
>Cc: Liming Gao <liming.gao@intel.com>
>Cc: Bob Feng <bob.c.feng@intel.com>
>Cc: Christian Rodriguez <christian.rodriguez@intel.com>
>Signed-off-by: Steven Shi <steven.shi@intel.com>
>---
> BaseTools/Source/Python/AutoGen/AutoGen.py | 17 ++++++++++++++---
> BaseTools/Source/Python/build/build.py     |  3 ++-
> 2 files changed, 16 insertions(+), 4 deletions(-)
>
>diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py
>b/BaseTools/Source/Python/AutoGen/AutoGen.py
>index a5bef4f7c6..aeb63f52c5 100644
>--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
>+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
>@@ -2578,7 +2578,7 @@ class ModuleAutoGen(AutoGen):
>         self.AutoGenDepSet = set()
>         self.ReferenceModules = []
>         self.ConstPcd                  = {}
>-
>+        self.CacheRestored             = False
>
>     def __repr__(self):
>         return "%s [%s]" % (self.MetaFile, self.Arch)
>@@ -3906,6 +3906,12 @@ class ModuleAutoGen(AutoGen):
>             ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
>             if os.path.exists(ModuleFile):
>                 shutil.copy2(ModuleFile, FileDir)
>+        else:
>+            OutputDir = self.OutputDir.replace('\\', '/').strip('/')
>+            DebugDir = self.DebugDir.replace('\\', '/').strip('/')
>+            for Item in self.CodaTargetList:
>+                File = Item.Target.Path.replace('\\', '/').strip('/').replace(DebugDir,
>'').replace(OutputDir, '').strip('/')
>+                self.OutputFile.add(File)
>         if not self.OutputFile:
>             Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget,
>self.ToolChain]
>             self.OutputFile = Ma.Binaries
>@@ -3949,6 +3955,7 @@ class ModuleAutoGen(AutoGen):
>                                 destination_dir = os.path.dirname(destination_file)
>                                 CreateDirectory(destination_dir)
>                                 shutil.copy2(File, destination_dir)
>+                    self.CacheRestored = True
>                     if self.Name == "PcdPeim" or self.Name == "PcdDxe":
>                         CreatePcdDatabaseCode(self, TemplateString(),
>TemplateString())
>                     return True
>@@ -3987,7 +3994,9 @@ class ModuleAutoGen(AutoGen):
>         self.GenFfsList = GenFfsList
>         if not self.IsLibrary and CreateLibraryMakeFile:
>             for LibraryAutoGen in self.LibraryAutoGenList:
>-                LibraryAutoGen.CreateMakeFile()
>+                # Only create makefile for libraries which have not been restored
>+                if not LibraryAutoGen.CacheRestored:
>+                    LibraryAutoGen.CreateMakeFile()
>
>         if self.CanSkip():
>             return
>@@ -4030,7 +4039,9 @@ class ModuleAutoGen(AutoGen):
>
>         if not self.IsLibrary and CreateLibraryCodeFile:
>             for LibraryAutoGen in self.LibraryAutoGenList:
>-                LibraryAutoGen.CreateCodeFile()
>+                # Only create autogen code for libraries which have not been
>restored
>+                if not LibraryAutoGen.CacheRestored:
>+                    LibraryAutoGen.CreateCodeFile()
>
>         if self.CanSkip():
>             return
>diff --git a/BaseTools/Source/Python/build/build.py
>b/BaseTools/Source/Python/build/build.py
>index 80ceb98310..f1f4c07980 100644
>--- a/BaseTools/Source/Python/build/build.py
>+++ b/BaseTools/Source/Python/build/build.py
>@@ -341,7 +341,8 @@ class ModuleMakeUnit(BuildUnit):
>     #   @param  Target      The build target name, one of gSupportedTarget
>     #
>     def __init__(self, Obj, Target):
>-        Dependency = [ModuleMakeUnit(La, Target) for La in
>Obj.LibraryAutoGenList]
>+        # Skip the dependency modules which are already restored from cache
>+        Dependency = [ModuleMakeUnit(La, Target) for La in
>Obj.LibraryAutoGenList if not La.CacheRestored]
>         BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Dependency,
>Obj.MakeFileDir)
>         if Target in [None, "", "all"]:
>             self.Target = "tbuild"
>--
>2.17.1.windows.2
>
>
>


[-- Attachment #2: 0001-Add-library-copy-to-cache.patch --]
[-- Type: application/octet-stream, Size: 1335 bytes --]

From 97127828385c095013a90a20fd8c47e7e3ea0c2d Mon Sep 17 00:00:00 2001
From: Christian Rodriguez <christian.rodriguez@intel.com>
Date: Tue, 28 May 2019 07:58:05 -0700
Subject: [PATCH] Add library copy to cache

---
 BaseTools/Source/Python/AutoGen/AutoGen.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index a376bc24d6..4e4794fd93 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -3906,6 +3906,12 @@ class ModuleAutoGen(AutoGen):
             ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
             if os.path.exists(ModuleFile):
                 shutil.copy2(ModuleFile, FileDir)
+        else:
+            OutputDir = self.OutputDir.replace('\\', '/').strip('/')
+            DebugDir = self.DebugDir.replace('\\', '/').strip('/')
+            for Item in self.CodaTargetList:
+                File = Item.Target.Path.replace('\\', '/').strip('/').replace(DebugDir, '').replace(OutputDir, '').strip('/')
+                self.OutputFile.add(File)
         if not self.OutputFile:
             Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget, self.ToolChain]
             self.OutputFile = Ma.Binaries
-- 
2.21.0.windows.1