* [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature
@ 2019-04-04 16:04 Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 1/4] " Christian Rodriguez
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Christian Rodriguez @ 2019-04-04 16:04 UTC (permalink / raw)
To: devel
Patch 1: Consider modules with .inc source files as Binary Modules and do not Skip by hash.
Patch 2: Re-order hashing operations so we don't do redundant hashes.
Patch 3: Respect artifact location within directory structure.
Patch 4: Re-use libraries, since they have already been hashed.
Christian Rodriguez (4):
BaseTools: Fix corner-cases of --hash feature
BaseTools: Fix corner-cases of --hash feature
BaseTools: Fix corner-cases of --hash feature
BaseTools: Fix corner-cases of --hash feature
BaseTools/Source/Python/AutoGen/AutoGen.py | 47 +++++++++++++++++-----
BaseTools/Source/Python/AutoGen/GenMake.py | 6 +--
BaseTools/Source/Python/build/build.py | 14 ++++++-
3 files changed, 52 insertions(+), 15 deletions(-)
--
2.19.1.windows.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Patch V2 1/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature Christian Rodriguez
@ 2019-04-04 16:04 ` Christian Rodriguez
2019-04-09 5:09 ` BobCF
2019-04-04 16:04 ` [Patch V2 2/4] " Christian Rodriguez
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Christian Rodriguez @ 2019-04-04 16:04 UTC (permalink / raw)
To: devel; +Cc: Bob Feng, Liming Gao, Yonghong Zhu
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Consider modules with .inc source files as Binary Modules
and do not Skip by hash.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 8c7c20a386..792bc99f54 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -3923,8 +3923,13 @@ class ModuleAutoGen(AutoGen):
shutil.copy2(File, FileDir)
def AttemptModuleCacheCopy(self):
+ # If library or Module is binary do not skip by hash
if self.IsBinaryModule:
return False
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
FileDir = path.join(GlobalData.gBinCacheSource, self.Arch, self.SourceDir, self.MetaFile.BaseName)
HashFile = path.join(FileDir, self.Name + '.hash')
if os.path.exists(HashFile):
@@ -4126,7 +4131,16 @@ class ModuleAutoGen(AutoGen):
## Decide whether we can skip the ModuleAutoGen process
def CanSkipbyHash(self):
+ # If library or Module is binary do not skip by hash
+ if self.IsBinaryModule:
+ return False
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
if GlobalData.gUseHashCache:
+ # If there is a valid hash or function generated a valid hash; function will return False
+ # and the statement below will return True
return not self.GenModuleHash()
return False
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Patch V2 2/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 1/4] " Christian Rodriguez
@ 2019-04-04 16:04 ` Christian Rodriguez
2019-04-09 5:09 ` BobCF
2019-04-04 16:04 ` [Patch V2 3/4] " Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 4/4] " Christian Rodriguez
3 siblings, 1 reply; 9+ messages in thread
From: Christian Rodriguez @ 2019-04-04 16:04 UTC (permalink / raw)
To: devel; +Cc: Bob Feng, Liming Gao, Yonghong Zhu
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Re-order hashing operations so we don't do redundant hashes.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 792bc99f54..b516404696 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -661,7 +661,7 @@ class WorkspaceAutoGen(AutoGen):
#
# Generate Package level hash value
#
- GlobalData.gPackageHash[Arch] = {}
+ GlobalData.gPackageHash = {}
if GlobalData.gUseHashCache:
for Pkg in Pkgs:
self._GenPkgLevelHash(Pkg)
@@ -747,7 +747,7 @@ class WorkspaceAutoGen(AutoGen):
return True
def _GenPkgLevelHash(self, Pkg):
- if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]:
+ if Pkg.PackageName in GlobalData.gPackageHash:
return
PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName)
@@ -770,7 +770,7 @@ class WorkspaceAutoGen(AutoGen):
f.close()
m.update(Content)
SaveFileOnChange(HashFile, m.hexdigest(), False)
- GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest()
+ GlobalData.gPackageHash[Pkg.PackageName] = m.hexdigest()
def _GetMetaFiles(self, Target, Toolchain, Arch):
AllWorkSpaceMetaFiles = set()
@@ -4092,14 +4092,16 @@ class ModuleAutoGen(AutoGen):
def GenModuleHash(self):
if self.Arch not in GlobalData.gModuleHash:
GlobalData.gModuleHash[self.Arch] = {}
+ if self.Name in GlobalData.gModuleHash[self.Arch] and GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
+ return False
m = hashlib.md5()
# Add Platform level hash
m.update(GlobalData.gPlatformHash.encode('utf-8'))
# Add Package level hash
if self.DependentPackageList:
for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName):
- if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]:
- m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName].encode('utf-8'))
+ if Pkg.PackageName in GlobalData.gPackageHash:
+ m.update(GlobalData.gPackageHash[Pkg.PackageName].encode('utf-8'))
# Add Library hash
if self.LibraryAutoGenList:
@@ -4124,9 +4126,8 @@ class ModuleAutoGen(AutoGen):
ModuleHashFile = path.join(self.BuildDir, self.Name + ".hash")
if self.Name not in GlobalData.gModuleHash[self.Arch]:
GlobalData.gModuleHash[self.Arch][self.Name] = m.hexdigest()
- if GlobalData.gBinCacheSource:
- if self.AttemptModuleCacheCopy():
- return False
+ if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
+ return False
return SaveFileOnChange(ModuleHashFile, m.hexdigest(), False)
## Decide whether we can skip the ModuleAutoGen process
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Patch V2 3/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 1/4] " Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 2/4] " Christian Rodriguez
@ 2019-04-04 16:04 ` Christian Rodriguez
2019-04-09 5:08 ` BobCF
2019-04-04 16:04 ` [Patch V2 4/4] " Christian Rodriguez
3 siblings, 1 reply; 9+ messages in thread
From: Christian Rodriguez @ 2019-04-04 16:04 UTC (permalink / raw)
To: devel; +Cc: Bob Feng, Liming Gao, Yonghong Zhu
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Respect artifact location within directory structure.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index b516404696..d087ca5e0e 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -3906,9 +3906,11 @@ class ModuleAutoGen(AutoGen):
FileDir = path.join(GlobalData.gBinCacheDest, self.Arch, self.SourceDir, self.MetaFile.BaseName)
CreateDirectory (FileDir)
HashFile = path.join(self.BuildDir, self.Name + '.hash')
- ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
if os.path.exists(HashFile):
shutil.copy2(HashFile, FileDir)
+ if self.IsLibrary:
+ return
+ ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
if os.path.exists(ModuleFile):
shutil.copy2(ModuleFile, FileDir)
if not self.OutputFile:
@@ -3920,7 +3922,11 @@ class ModuleAutoGen(AutoGen):
if not os.path.isabs(File):
File = os.path.join(self.OutputDir, File)
if os.path.exists(File):
- shutil.copy2(File, FileDir)
+ sub_dir = os.path.relpath(File, self.OutputDir)
+ destination_file = os.path.join(FileDir, sub_dir)
+ destination_dir = os.path.dirname(destination_file)
+ CreateDirectory(destination_dir)
+ shutil.copy2(File, destination_dir)
def AttemptModuleCacheCopy(self):
# If library or Module is binary do not skip by hash
@@ -3944,7 +3950,11 @@ class ModuleAutoGen(AutoGen):
shutil.copy2(HashFile, self.BuildDir)
else:
File = path.join(root, f)
- shutil.copy2(File, self.OutputDir)
+ sub_dir = os.path.relpath(File, FileDir)
+ destination_file = os.path.join(self.OutputDir, sub_dir)
+ destination_dir = os.path.dirname(destination_file)
+ CreateDirectory(destination_dir)
+ shutil.copy2(File, destination_dir)
if self.Name == "PcdPeim" or self.Name == "PcdDxe":
CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
return True
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [Patch V2 4/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature Christian Rodriguez
` (2 preceding siblings ...)
2019-04-04 16:04 ` [Patch V2 3/4] " Christian Rodriguez
@ 2019-04-04 16:04 ` Christian Rodriguez
2019-04-09 5:09 ` [edk2-devel] " BobCF
3 siblings, 1 reply; 9+ messages in thread
From: Christian Rodriguez @ 2019-04-04 16:04 UTC (permalink / raw)
To: devel; +Cc: Bob Feng, Liming Gao, Yonghong Zhu
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Re-use libraries, since they have already been hashed.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++---
BaseTools/Source/Python/build/build.py | 14 +++++++++++++-
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index b441817b52..537314e33b 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -983,7 +983,7 @@ cleanlib:
## For creating makefile targets for dependent libraries
def ProcessDependentLibrary(self):
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
self.LibraryBuildDirectoryList.append(self.PlaceMacro(LibraryAutoGen.BuildDir, self.Macros))
## Return a list containing source file's dependencies
@@ -1486,7 +1486,7 @@ cleanlib:
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
@@ -1622,7 +1622,7 @@ class TopLevelMakefile(BuildFile):
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index de641fb452..8c59fc9e84 100644
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -1775,7 +1775,8 @@ class Build():
for Module in Pa.Platform.Modules:
if self.ModuleFile.Dir == Module.Dir and self.ModuleFile.Name == Module.Name:
Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile)
- if Ma is None: continue
+ if Ma is None:
+ continue
MaList.append(Ma)
if Ma.CanSkipbyHash():
self.HashSkipModules.append(Ma)
@@ -2149,10 +2150,21 @@ class Build():
RemoveDirectory(os.path.dirname(GlobalData.gDatabasePath), True)
def CreateAsBuiltInf(self):
+ all_lib_set = set()
+ all_mod_set = set()
for Module in self.BuildModules:
Module.CreateAsBuiltInf()
+ all_mod_set.add(Module)
for Module in self.HashSkipModules:
Module.CreateAsBuiltInf(True)
+ all_mod_set.add(Module)
+ for Module in all_mod_set:
+ for lib in Module.LibraryAutoGenList:
+ all_lib_set.add(lib)
+ for lib in all_lib_set:
+ lib.CreateAsBuiltInf(True)
+ all_lib_set.clear()
+ all_mod_set.clear()
self.BuildModules = []
self.HashSkipModules = []
## Do some clean-up works when error occurred
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Patch V2 3/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 ` [Patch V2 3/4] " Christian Rodriguez
@ 2019-04-09 5:08 ` BobCF
0 siblings, 0 replies; 9+ messages in thread
From: BobCF @ 2019-04-09 5:08 UTC (permalink / raw)
To: Rodriguez, Christian, devel@edk2.groups.io; +Cc: Gao, Liming, Zhu, Yonghong
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: Rodriguez, Christian
Sent: Friday, April 5, 2019 12:04 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: [Patch V2 3/4] BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Respect artifact location within directory structure.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index b516404696..d087ca5e0e 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -3906,9 +3906,11 @@ class ModuleAutoGen(AutoGen):
FileDir = path.join(GlobalData.gBinCacheDest, self.Arch, self.SourceDir, self.MetaFile.BaseName)
CreateDirectory (FileDir)
HashFile = path.join(self.BuildDir, self.Name + '.hash')
- ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
if os.path.exists(HashFile):
shutil.copy2(HashFile, FileDir)
+ if self.IsLibrary:
+ return
+ ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
if os.path.exists(ModuleFile):
shutil.copy2(ModuleFile, FileDir)
if not self.OutputFile:
@@ -3920,7 +3922,11 @@ class ModuleAutoGen(AutoGen):
if not os.path.isabs(File):
File = os.path.join(self.OutputDir, File)
if os.path.exists(File):
- shutil.copy2(File, FileDir)
+ sub_dir = os.path.relpath(File, self.OutputDir)
+ destination_file = os.path.join(FileDir, sub_dir)
+ destination_dir = os.path.dirname(destination_file)
+ CreateDirectory(destination_dir)
+ shutil.copy2(File, destination_dir)
def AttemptModuleCacheCopy(self):
# If library or Module is binary do not skip by hash @@ -3944,7 +3950,11 @@ class ModuleAutoGen(AutoGen):
shutil.copy2(HashFile, self.BuildDir)
else:
File = path.join(root, f)
- shutil.copy2(File, self.OutputDir)
+ sub_dir = os.path.relpath(File, FileDir)
+ destination_file = os.path.join(self.OutputDir, sub_dir)
+ destination_dir = os.path.dirname(destination_file)
+ CreateDirectory(destination_dir)
+ shutil.copy2(File, destination_dir)
if self.Name == "PcdPeim" or self.Name == "PcdDxe":
CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
return True
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Patch V2 2/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 ` [Patch V2 2/4] " Christian Rodriguez
@ 2019-04-09 5:09 ` BobCF
0 siblings, 0 replies; 9+ messages in thread
From: BobCF @ 2019-04-09 5:09 UTC (permalink / raw)
To: Rodriguez, Christian, devel@edk2.groups.io; +Cc: Gao, Liming, Zhu, Yonghong
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: Rodriguez, Christian
Sent: Friday, April 5, 2019 12:04 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: [Patch V2 2/4] BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Re-order hashing operations so we don't do redundant hashes.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 792bc99f54..b516404696 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -661,7 +661,7 @@ class WorkspaceAutoGen(AutoGen):
#
# Generate Package level hash value
#
- GlobalData.gPackageHash[Arch] = {}
+ GlobalData.gPackageHash = {}
if GlobalData.gUseHashCache:
for Pkg in Pkgs:
self._GenPkgLevelHash(Pkg) @@ -747,7 +747,7 @@ class WorkspaceAutoGen(AutoGen):
return True
def _GenPkgLevelHash(self, Pkg):
- if Pkg.PackageName in GlobalData.gPackageHash[Pkg.Arch]:
+ if Pkg.PackageName in GlobalData.gPackageHash:
return
PkgDir = os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) @@ -770,7 +770,7 @@ class WorkspaceAutoGen(AutoGen):
f.close()
m.update(Content)
SaveFileOnChange(HashFile, m.hexdigest(), False)
- GlobalData.gPackageHash[Pkg.Arch][Pkg.PackageName] = m.hexdigest()
+ GlobalData.gPackageHash[Pkg.PackageName] = m.hexdigest()
def _GetMetaFiles(self, Target, Toolchain, Arch):
AllWorkSpaceMetaFiles = set()
@@ -4092,14 +4092,16 @@ class ModuleAutoGen(AutoGen):
def GenModuleHash(self):
if self.Arch not in GlobalData.gModuleHash:
GlobalData.gModuleHash[self.Arch] = {}
+ if self.Name in GlobalData.gModuleHash[self.Arch] and GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
+ return False
m = hashlib.md5()
# Add Platform level hash
m.update(GlobalData.gPlatformHash.encode('utf-8'))
# Add Package level hash
if self.DependentPackageList:
for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName):
- if Pkg.PackageName in GlobalData.gPackageHash[self.Arch]:
- m.update(GlobalData.gPackageHash[self.Arch][Pkg.PackageName].encode('utf-8'))
+ if Pkg.PackageName in GlobalData.gPackageHash:
+
+ m.update(GlobalData.gPackageHash[Pkg.PackageName].encode('utf-8'))
# Add Library hash
if self.LibraryAutoGenList:
@@ -4124,9 +4126,8 @@ class ModuleAutoGen(AutoGen):
ModuleHashFile = path.join(self.BuildDir, self.Name + ".hash")
if self.Name not in GlobalData.gModuleHash[self.Arch]:
GlobalData.gModuleHash[self.Arch][self.Name] = m.hexdigest()
- if GlobalData.gBinCacheSource:
- if self.AttemptModuleCacheCopy():
- return False
+ if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
+ return False
return SaveFileOnChange(ModuleHashFile, m.hexdigest(), False)
## Decide whether we can skip the ModuleAutoGen process
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Patch V2 1/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 ` [Patch V2 1/4] " Christian Rodriguez
@ 2019-04-09 5:09 ` BobCF
0 siblings, 0 replies; 9+ messages in thread
From: BobCF @ 2019-04-09 5:09 UTC (permalink / raw)
To: Rodriguez, Christian, devel@edk2.groups.io; +Cc: Gao, Liming, Zhu, Yonghong
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: Rodriguez, Christian
Sent: Friday, April 5, 2019 12:04 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: [Patch V2 1/4] BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Consider modules with .inc source files as Binary Modules and do not Skip by hash.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 8c7c20a386..792bc99f54 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -3923,8 +3923,13 @@ class ModuleAutoGen(AutoGen):
shutil.copy2(File, FileDir)
def AttemptModuleCacheCopy(self):
+ # If library or Module is binary do not skip by hash
if self.IsBinaryModule:
return False
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
FileDir = path.join(GlobalData.gBinCacheSource, self.Arch, self.SourceDir, self.MetaFile.BaseName)
HashFile = path.join(FileDir, self.Name + '.hash')
if os.path.exists(HashFile):
@@ -4126,7 +4131,16 @@ class ModuleAutoGen(AutoGen):
## Decide whether we can skip the ModuleAutoGen process
def CanSkipbyHash(self):
+ # If library or Module is binary do not skip by hash
+ if self.IsBinaryModule:
+ return False
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
if GlobalData.gUseHashCache:
+ # If there is a valid hash or function generated a valid hash; function will return False
+ # and the statement below will return True
return not self.GenModuleHash()
return False
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [edk2-devel] [Patch V2 4/4] BaseTools: Fix corner-cases of --hash feature
2019-04-04 16:04 ` [Patch V2 4/4] " Christian Rodriguez
@ 2019-04-09 5:09 ` BobCF
0 siblings, 0 replies; 9+ messages in thread
From: BobCF @ 2019-04-09 5:09 UTC (permalink / raw)
To: devel@edk2.groups.io, Rodriguez, Christian; +Cc: Gao, Liming, Zhu, Yonghong
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Christian Rodriguez
Sent: Friday, April 5, 2019 12:04 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: [edk2-devel] [Patch V2 4/4] BaseTools: Fix corner-cases of --hash feature
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1680
Re-use libraries, since they have already been hashed.
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christian Rodriguez <christian.rodriguez@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
---
BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++---
BaseTools/Source/Python/build/build.py | 14 +++++++++++++-
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index b441817b52..537314e33b 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -983,7 +983,7 @@ cleanlib:
## For creating makefile targets for dependent libraries
def ProcessDependentLibrary(self):
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
self.LibraryBuildDirectoryList.append(self.PlaceMacro(LibraryAutoGen.BuildDir, self.Macros))
## Return a list containing source file's dependencies @@ -1486,7 +1486,7 @@ cleanlib:
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
@@ -1622,7 +1622,7 @@ class TopLevelMakefile(BuildFile):
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule:
+ if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index de641fb452..8c59fc9e84 100644
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -1775,7 +1775,8 @@ class Build():
for Module in Pa.Platform.Modules:
if self.ModuleFile.Dir == Module.Dir and self.ModuleFile.Name == Module.Name:
Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile)
- if Ma is None: continue
+ if Ma is None:
+ continue
MaList.append(Ma)
if Ma.CanSkipbyHash():
self.HashSkipModules.append(Ma) @@ -2149,10 +2150,21 @@ class Build():
RemoveDirectory(os.path.dirname(GlobalData.gDatabasePath), True)
def CreateAsBuiltInf(self):
+ all_lib_set = set()
+ all_mod_set = set()
for Module in self.BuildModules:
Module.CreateAsBuiltInf()
+ all_mod_set.add(Module)
for Module in self.HashSkipModules:
Module.CreateAsBuiltInf(True)
+ all_mod_set.add(Module)
+ for Module in all_mod_set:
+ for lib in Module.LibraryAutoGenList:
+ all_lib_set.add(lib)
+ for lib in all_lib_set:
+ lib.CreateAsBuiltInf(True)
+ all_lib_set.clear()
+ all_mod_set.clear()
self.BuildModules = []
self.HashSkipModules = []
## Do some clean-up works when error occurred
--
2.19.1.windows.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-04-09 5:09 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-04-04 16:04 [Patch V2 0/4] BaseTools: Fix corner-cases of --hash feature Christian Rodriguez
2019-04-04 16:04 ` [Patch V2 1/4] " Christian Rodriguez
2019-04-09 5:09 ` BobCF
2019-04-04 16:04 ` [Patch V2 2/4] " Christian Rodriguez
2019-04-09 5:09 ` BobCF
2019-04-04 16:04 ` [Patch V2 3/4] " Christian Rodriguez
2019-04-09 5:08 ` BobCF
2019-04-04 16:04 ` [Patch V2 4/4] " Christian Rodriguez
2019-04-09 5:09 ` [edk2-devel] " BobCF
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox