Re: [PATCH] BaseTools: Cannot store library cache of different arch together

["Christian Rodriguez" <christian.rodriguez@intel.com>]

The change looks good overall, but I'm concerned about the performance if there are multiple of the same library used by different modules. In this case the library will be copied once per arch per module. I'm not sure if it would make a giant impact, but just something to think about because you would be doing extra IO accesses for each duplicate library autogen.

Possible suggestion:
Is it possible to change the hash to include the arch? Or you can store a unique tuple pair like (lib.arch, lib) in the set to reduce the libraries being copied to a unique subset.

Thanks,
Christian

>-----Original Message-----
>From: Shi, Steven
>Sent: Monday, June 10, 2019 10:53 PM
>To: devel@edk2.groups.io
>Cc: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
><bob.c.feng@intel.com>; Rodriguez, Christian
><christian.rodriguez@intel.com>
>Subject: [PATCH] BaseTools: Cannot store library cache of different arch
>together
>
>https://bugzilla.tianocore.org/show_bug.cgi?id=1895
>
>Build cache cannot store cache for the same library modules in different arch
>together. E.g. Both the below IA32 and X64 arch BaseLib caches should exist
>after build Ovmf3264, but now only the one in X64 arch exist.
>The reason is the current Basetool use a set() to same all library AutoGen
>objects, but the different arch lib AutoGen objects have same __hash_ value
>which comes from the lib MetaFile(The path of module file):
>    def __hash__(self):
>        return hash(self.MetaFile)
>
>So the different arch lib AutoGen objects are duplicated one to the set() and
>only one can exist. This is why the Basetool can only store one arch cache for
>library.
>This patch avoid to use the set() as the container to save the library and
>module objects because the objects might have same __hash__ value.
>
>Cc: Liming Gao <liming.gao@intel.com>
>Cc: Bob Feng <bob.c.feng@intel.com>
>Cc: Christian Rodriguez <christian.rodriguez@intel.com>
>Signed-off-by: Steven Shi <steven.shi@intel.com>
>---
> BaseTools/Source/Python/build/build.py | 13 +++----------
> 1 file changed, 3 insertions(+), 10 deletions(-)
>
>diff --git a/BaseTools/Source/Python/build/build.py
>b/BaseTools/Source/Python/build/build.py
>index 0855d4561c..f7a79cbbab 100644
>--- a/BaseTools/Source/Python/build/build.py
>+++ b/BaseTools/Source/Python/build/build.py
>@@ -2203,21 +2203,14 @@ class Build():
>             RemoveDirectory(os.path.dirname(GlobalData.gDatabasePath), True)
>
>     def CreateAsBuiltInf(self):
>-        all_lib_set = set()
>-        all_mod_set = set()
>         for Module in self.BuildModules:
>             Module.CreateAsBuiltInf()
>-            all_mod_set.add(Module)
>+            for lib in Module.LibraryAutoGenList:
>+                lib.CreateAsBuiltInf(True)
>         for Module in self.HashSkipModules:
>             Module.CreateAsBuiltInf(True)
>-            all_mod_set.add(Module)
>-        for Module in all_mod_set:
>             for lib in Module.LibraryAutoGenList:
>-                all_lib_set.add(lib)
>-        for lib in all_lib_set:
>-            lib.CreateAsBuiltInf(True)
>-        all_lib_set.clear()
>-        all_mod_set.clear()
>+                lib.CreateAsBuiltInf(True)
>         self.BuildModules = []
>         self.HashSkipModules = []
>     ## Do some clean-up works when error occurred
>--
>2.17.1.windows.2