* BaseTools --hash malfunction after migrate from stable201905 to stable201908
@ 2019-09-20 6:54 Lin, Derek (HPS SW)
2019-09-23 7:51 ` Steven Shi
2019-09-24 8:12 ` Steven Shi
0 siblings, 2 replies; 6+ messages in thread
From: Lin, Derek (HPS SW) @ 2019-09-20 6:54 UTC (permalink / raw)
To: devel@edk2.groups.io; +Cc: Feng, Bob C, steven.shi@intel.com
[-- Attachment #1: Type: text/plain, Size: 1692 bytes --]
Hi BaseTools experts,
We saw the clean build performance improvement after updating from edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.
We're building server platform. And we use --hash to speed up incremental build as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an incremental build without any code change.
Dig into the issue, it is caused by commit https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and --binary-source to perform cache based incremental build. However, when we only have --hash in build flag, it is not functional.
Here's a build time log retrieved from -y report.log.
201905 incremental | 201908 incremental | 201905 clean | 201908 clean
Build Duration: 00:03:44 00:04:40 00:07:12 00:06:19
AutoGen Duration: 00:02:40 00:03:15 00:04:28 00:03:09
Make Duration: 00:00:36 00:00:39 00:01:21 00:01:34
GenFds Duration: 00:00:27 00:00:28 00:01:22 00:01:18
We could see the AutoGen Duration has almost no change between clean and incremental build in 201908.
Could we fix this?
Thanks,
Derek
[-- Attachment #2: Type: text/html, Size: 6627 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BaseTools --hash malfunction after migrate from stable201905 to stable201908
2019-09-20 6:54 BaseTools --hash malfunction after migrate from stable201905 to stable201908 Lin, Derek (HPS SW)
@ 2019-09-23 7:51 ` Steven Shi
2019-09-24 8:12 ` Steven Shi
1 sibling, 0 replies; 6+ messages in thread
From: Steven Shi @ 2019-09-23 7:51 UTC (permalink / raw)
To: Lin, Derek (HPS SW), devel@edk2.groups.io; +Cc: Feng, Bob C, Gao, Liming
[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]
Hi Derek,
Thank you to raise this issue. We will fix it and add back the --hash original functionality.
Thanks
Steven
From: Lin, Derek (HPS SW) [mailto:derek.lin2@hpe.com]
Sent: Friday, September 20, 2019 2:54 PM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Shi, Steven <steven.shi@intel.com>
Subject: BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi BaseTools experts,
We saw the clean build performance improvement after updating from edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.
We're building server platform. And we use --hash to speed up incremental build as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an incremental build without any code change.
Dig into the issue, it is caused by commit https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and --binary-source to perform cache based incremental build. However, when we only have --hash in build flag, it is not functional.
Here's a build time log retrieved from -y report.log.
201905 incremental | 201908 incremental | 201905 clean | 201908 clean
Build Duration: 00:03:44 00:04:40 00:07:12 00:06:19
AutoGen Duration: 00:02:40 00:03:15 00:04:28 00:03:09
Make Duration: 00:00:36 00:00:39 00:01:21 00:01:34
GenFds Duration: 00:00:27 00:00:28 00:01:22 00:01:18
We could see the AutoGen Duration has almost no change between clean and incremental build in 201908.
Could we fix this?
Thanks,
Derek
[-- Attachment #2: Type: text/html, Size: 8031 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BaseTools --hash malfunction after migrate from stable201905 to stable201908
2019-09-20 6:54 BaseTools --hash malfunction after migrate from stable201905 to stable201908 Lin, Derek (HPS SW)
2019-09-23 7:51 ` Steven Shi
@ 2019-09-24 8:12 ` Steven Shi
2019-09-27 7:57 ` [edk2-devel] " Lin, Derek (HPS SW)
1 sibling, 1 reply; 6+ messages in thread
From: Steven Shi @ 2019-09-24 8:12 UTC (permalink / raw)
To: Lin, Derek (HPS SW), devel@edk2.groups.io; +Cc: Feng, Bob C, Gao, Liming
[-- Attachment #1.1: Type: text/plain, Size: 3177 bytes --]
Hi Derek,
I took a look at this issue, and add back the -hash functionality in this branch: https://github.com/shijunjing/edk2/tree/hashcache_v1. The attachment is the patch based on latest edk2.
The current -hash fix performance is not as good as the edk2-stable201905, because the edk2-stable201905 -hash doesn't parse source code header files and not include the header files as the module's hash dependency which is not sound. After add the header files dependency, the -hash become slow.
Let me know whether the attached fix works for you. I might continue to tune its performance.
Thanks
Steven Shi
Intel\SSG\FID\Firmware Infrastructure
From: Shi, Steven
Sent: Monday, September 23, 2019 3:52 PM
To: 'Lin, Derek (HPS SW)' <derek.lin2@hpe.com>; devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming (liming.gao@intel.com) <liming.gao@intel.com>
Subject: RE: BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi Derek,
Thank you to raise this issue. We will fix it and add back the --hash original functionality.
Thanks
Steven
From: Lin, Derek (HPS SW) [mailto:derek.lin2@hpe.com]
Sent: Friday, September 20, 2019 2:54 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Subject: BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi BaseTools experts,
We saw the clean build performance improvement after updating from edk2-stable201905 to edk2-stable201908, which is promising.
But we found the incremental build time increase 25%~35%, which is not good.
We're building server platform. And we use --hash to speed up incremental build as described in https://github.com/BobCF/edk2/wiki/Incremental-Build
After upgrading to edk2-stable201908, none of AutoGen is skipped when I did an incremental build without any code change.
Dig into the issue, it is caused by commit https://github.com/tianocore/edk2/commit/0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
>From the code change logic, it seems like it require both --hash and --binary-source to perform cache based incremental build. However, when we only have --hash in build flag, it is not functional.
Here's a build time log retrieved from -y report.log.
201905 incremental | 201908 incremental | 201905 clean | 201908 clean
Build Duration: 00:03:44 00:04:40 00:07:12 00:06:19
AutoGen Duration: 00:02:40 00:03:15 00:04:28 00:03:09
Make Duration: 00:00:36 00:00:39 00:01:21 00:01:34
GenFds Duration: 00:00:27 00:00:28 00:01:22 00:01:18
We could see the AutoGen Duration has almost no change between clean and incremental build in 201908.
Could we fix this?
Thanks,
Derek
[-- Attachment #1.2: Type: text/html, Size: 10923 bytes --]
[-- Attachment #2: 0001-Fix-the-hash-functionality-for-increamental-build.patch --]
[-- Type: application/octet-stream, Size: 23859 bytes --]
From 24f30a6bfa2cc31eab58f8699e475a0329fd4659 Mon Sep 17 00:00:00 2001
From: Steven <steven.shi@intel.com>
Date: Tue, 24 Sep 2019 15:55:56 +0800
Subject: [PATCH] Fix the --hash functionality for increamental build
Current --hash option has no effect and cannot skip module build
in the increamental build. This patch add back the original
functionality.
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
.../Source/Python/AutoGen/AutoGenWorker.py | 5 +
BaseTools/Source/Python/AutoGen/CacheIR.py | 3 +
BaseTools/Source/Python/AutoGen/DataPipe.py | 2 +
.../Source/Python/AutoGen/ModuleAutoGen.py | 114 ++++++++++--------
BaseTools/Source/Python/Common/GlobalData.py | 5 -
BaseTools/Source/Python/build/build.py | 81 ++++---------
6 files changed, 98 insertions(+), 112 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
index 94ea61a487..aec843a444 100755
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -184,6 +184,7 @@ class AutoGenWorkerInProcess(mp.Process):
GlobalData.gDisableIncludePathCheck = False
GlobalData.gFdfParser = self.data_pipe.Get("FdfParser")
GlobalData.gDatabasePath = self.data_pipe.Get("DatabasePath")
+ GlobalData.gUseHashCache = self.data_pipe.Get("UseHashCache")
GlobalData.gBinCacheSource = self.data_pipe.Get("BinCacheSource")
GlobalData.gBinCacheDest = self.data_pipe.Get("BinCacheDest")
GlobalData.gCacheIR = self.share_data
@@ -240,6 +241,10 @@ class AutoGenWorkerInProcess(mp.Process):
Ma.GenPreMakefileHash(GlobalData.gCacheIR)
if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheIR):
continue
+ elif GlobalData.gUseHashCache and CommandTarget in [None, "", "all"]:
+ Ma.GenModuleIncrmtlHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyIncrmtlCache(GlobalData.gCacheIR):
+ continue
Ma.CreateCodeFile(False)
Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.Path, Ma.Arch),[]))
diff --git a/BaseTools/Source/Python/AutoGen/CacheIR.py b/BaseTools/Source/Python/AutoGen/CacheIR.py
index 715be5273c..71012aafbb 100755
--- a/BaseTools/Source/Python/AutoGen/CacheIR.py
+++ b/BaseTools/Source/Python/AutoGen/CacheIR.py
@@ -27,3 +27,6 @@ class ModuleBuildCacheIR():
self.CacheCrash = False
self.PreMakeCacheHit = False
self.MakeCacheHit = False
+ # Hash Cache used for increamental build
+ self.IncrmtlCacheHit = False
+
diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source/Python/AutoGen/DataPipe.py
index 8b8cfd1c51..4baeeb1d85 100755
--- a/BaseTools/Source/Python/AutoGen/DataPipe.py
+++ b/BaseTools/Source/Python/AutoGen/DataPipe.py
@@ -159,6 +159,8 @@ class MemoryDataPipe(DataPipe):
self.DataContainer = {"LogLevel": EdkLogger.GetLevel()}
+ self.DataContainer = {"UseHashCache":GlobalData.gUseHashCache}
+
self.DataContainer = {"BinCacheSource":GlobalData.gBinCacheSource}
self.DataContainer = {"BinCacheDest":GlobalData.gBinCacheDest}
diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
index fad5bab0f2..bb650b61b4 100755
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -1731,6 +1731,12 @@ class ModuleAutoGen(AutoGen):
for File in gDict[(self.MetaFile.Path, self.Arch)].AutoGenFileList:
CopyFileOnChange(str(File), CacheDebugDir)
+ for Root, Dirs, Files in os.walk(self.DebugDir):
+ for File in Files:
+ if File.lower().endswith('.dll'):
+ NewFile = path.join(self.DebugDir, File)
+ CopyFileOnChange(NewFile, CacheDebugDir)
+
return True
## Create makefile for the module and its dependent libraries
@@ -1914,50 +1920,15 @@ class ModuleAutoGen(AutoGen):
self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)
return RetVal
- def GenModuleHash(self):
- # Initialize a dictionary for each arch type
- if self.Arch not in GlobalData.gModuleHash:
- GlobalData.gModuleHash[self.Arch] = {}
-
- # Early exit if module or library has been hashed and is in memory
- if self.Name in GlobalData.gModuleHash[self.Arch]:
- return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8')
-
- # Initialze hash object
- m = hashlib.md5()
-
- # Add Platform level hash
- m.update(GlobalData.gPlatformHash.encode('utf-8'))
-
- # Add Package level hash
- if self.DependentPackageList:
- for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName):
- if Pkg.PackageName in GlobalData.gPackageHash:
- m.update(GlobalData.gPackageHash[Pkg.PackageName].encode('utf-8'))
-
- # Add Library hash
- if self.LibraryAutoGenList:
- for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name):
- if Lib.Name not in GlobalData.gModuleHash[self.Arch]:
- Lib.GenModuleHash()
- m.update(GlobalData.gModuleHash[self.Arch][Lib.Name].encode('utf-8'))
-
- # Add Module self
- with open(str(self.MetaFile), 'rb') as f:
- Content = f.read()
- m.update(Content)
-
- # Add Module's source files
- if self.SourceFileList:
- for File in sorted(self.SourceFileList, key=lambda x: str(x)):
- f = open(str(File), 'rb')
- Content = f.read()
- f.close()
- m.update(Content)
-
- GlobalData.gModuleHash[self.Arch][self.Name] = m.hexdigest()
-
- return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8')
+ def GenModuleIncrmtlHash(self, gDict):
+ self.GenModuleFilesHash(gDict)
+ self.GenPreMakefileHash(gDict)
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate PreMakefileHashHexDigest for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return None
+ else:
+ return gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest.encode('utf-8')
def GenModuleFilesHash(self, gDict):
# Early exit if module or library has been hashed and is in memory
@@ -2077,8 +2048,6 @@ class ModuleAutoGen(AutoGen):
for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName):
if (Pkg.PackageName, 'PackageHash') in gDict:
m.update(gDict[(Pkg.PackageName, 'PackageHash')].encode('utf-8'))
- else:
- EdkLogger.quiet("[cache warning]: %s PackageHash needed by %s[%s] is missing" %(Pkg.PackageName, self.MetaFile.Name, self.Arch))
# Add Library hash
if self.LibraryAutoGenList:
@@ -2241,6 +2210,53 @@ class ModuleAutoGen(AutoGen):
return gDict[(self.MetaFile.Path, self.Arch)]
+
+ ## Decide whether we can skip the left autogen and make process
+ def CanSkipbyIncrmtlCache(self, gDict):
+ if not GlobalData.gUseHashCache:
+ return False
+
+ # Disable incremental cache if binary cache is enabled
+ if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest:
+ return False
+
+ if not (self.MetaFile.Path, self.Arch) in gDict:
+ return False
+
+ if gDict[(self.MetaFile.Path, self.Arch)].IncrmtlCacheHit:
+ return True
+
+ if gDict[(self.MetaFile.Path, self.Arch)].CacheCrash:
+ return False
+
+ # If Module is binary, do not skip by cache
+ if self.IsBinaryModule:
+ return False
+
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
+
+ # Early exit for libraries that haven't yet finished building
+ HashFile = path.join(self.BuildDir, self.Name + ".hash")
+ if not os.path.exists(HashFile):
+ return False
+
+ with open(HashFile, "rb") as f:
+ if self.GenModuleIncrmtlHash(gDict) == f.read():
+ IncrmtlCacheHit = True
+ else:
+ IncrmtlCacheHit = False
+
+ if IncrmtlCacheHit:
+ with GlobalData.cache_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.IncrmtlCacheHit = IncrmtlCacheHit
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ print("[hash hit]:", self.MetaFile.Path, self.Arch)
+ return IncrmtlCacheHit
+
## Decide whether we can skip the left autogen and make process
def CanSkipbyPreMakefileCache(self, gDict):
if not GlobalData.gBinCacheSource:
@@ -2500,7 +2516,7 @@ class ModuleAutoGen(AutoGen):
## Decide whether we can skip the ModuleAutoGen process
def CanSkipbyCache(self, gDict):
# Hashing feature is off
- if not GlobalData.gBinCacheSource:
+ if not GlobalData.gUseHashCache:
return False
if self in GlobalData.gBuildHashSkipTracking:
@@ -2520,6 +2536,10 @@ class ModuleAutoGen(AutoGen):
if not (self.MetaFile.Path, self.Arch) in gDict:
return False
+ if gDict[(self.MetaFile.Path, self.Arch)].IncrmtlCacheHit:
+ GlobalData.gBuildHashSkipTracking[self] = True
+ return True
+
if gDict[(self.MetaFile.Path, self.Arch)].PreMakeCacheHit:
GlobalData.gBuildHashSkipTracking[self] = True
return True
diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Source/Python/Common/GlobalData.py
index 8eb72aa1d6..34f76e595b 100755
--- a/BaseTools/Source/Python/Common/GlobalData.py
+++ b/BaseTools/Source/Python/Common/GlobalData.py
@@ -109,11 +109,6 @@ gModuleHash = {}
gEnableGenfdsMultiThread = True
gSikpAutoGenCache = set()
-# Dictionary for tracking Module build status as success or failure
-# Top Dict: Key: Arch Type Value: Dictionary
-# Second Dict: Key: AutoGen Obj Value: 'SUCCESS'\'FAIL'\'FAIL_METAFILE'
-gModuleBuildTracking = dict()
-
# Dictionary of booleans that dictate whether a module or
# library can be skiped
# Top Dict: Key: Arch Type Value: Dictionary
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index bcd832c525..ecd2dfa3b5 100755
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -612,9 +612,9 @@ class BuildTask:
self.CompleteFlag = True
# Run hash operation post dependency, to account for libs
- if GlobalData.gUseHashCache and self.BuildItem.BuildObject.IsLibrary:
+ if GlobalData.gUseHashCache:
HashFile = path.join(self.BuildItem.BuildObject.BuildDir, self.BuildItem.BuildObject.Name + ".hash")
- SaveFileOnChange(HashFile, self.BuildItem.BuildObject.GenModuleHash(), True)
+ SaveFileOnChange(HashFile, self.BuildItem.BuildObject.GenModuleIncrmtlHash(GlobalData.gCacheIR), True)
except:
#
# TRICK: hide the output of threads left running, so that the user can
@@ -631,14 +631,6 @@ class BuildTask:
BuildTask._ErrorMessage = "%s broken\n %s [%s]" % \
(threading.currentThread().getName(), Command, WorkingDir)
- # Set the value used by hash invalidation flow in GlobalData.gModuleBuildTracking to 'SUCCESS'
- # If Module or Lib is being tracked, it did not fail header check test, and built successfully
- if (self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking and
- GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] != 'FAIL_METAFILE' and
- not BuildTask._ErrorFlag.isSet()
- ):
- GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] = 'SUCCESS'
-
# indicate there's a thread is available for another build task
BuildTask._RunningQueueLock.acquire()
BuildTask._RunningQueue.pop(self.BuildItem)
@@ -841,6 +833,10 @@ class Build():
PcdMa.GenPreMakefileHash(share_data)
if PcdMa.CanSkipbyPreMakefileCache(share_data):
continue
+ elif GlobalData.gUseHashCache and self.Target in [None, "", "all"]:
+ PcdMa.GenModuleIncrmtlHash(share_data)
+ if PcdMa.CanSkipbyIncrmtlCache(share_data):
+ continue
PcdMa.CreateCodeFile(False)
PcdMa.CreateMakeFile(False,GenFfsList = DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.Path, PcdMa.Arch),[]))
@@ -1160,38 +1156,6 @@ class Build():
EdkLogger.error("Postbuild", POSTBUILD_ERROR, 'Postbuild process is not success!')
EdkLogger.info("\n- Postbuild Done -\n")
- ## Error handling for hash feature
- #
- # On BuildTask error, iterate through the Module Build tracking
- # dictionary to determine wheather a module failed to build. Invalidate
- # the hash associated with that module by removing it from storage.
- #
- #
- def invalidateHash(self):
- # Only for hashing feature
- if not GlobalData.gUseHashCache:
- return
-
- # GlobalData.gModuleBuildTracking contains only modules or libs that cannot be skipped by hash
- for Ma in GlobalData.gModuleBuildTracking:
- # Skip invalidating for Successful Module/Lib builds
- if GlobalData.gModuleBuildTracking[Ma] == 'SUCCESS':
- continue
-
- # The module failed to build, failed to start building, or failed the header check test from this point on
-
- # Remove .hash from build
- ModuleHashFile = os.path.join(Ma.BuildDir, Ma.Name + ".hash")
- if os.path.exists(ModuleHashFile):
- os.remove(ModuleHashFile)
-
- # Remove .hash file from cache
- if GlobalData.gBinCacheDest:
- FileDir = os.path.join(GlobalData.gBinCacheDest, Ma.PlatformInfo.OutputDir, Ma.BuildTarget + "_" + Ma.ToolChain, Ma.Arch, Ma.SourceDir, Ma.MetaFile.BaseName)
- HashFile = os.path.join(FileDir, Ma.Name + '.hash')
- if os.path.exists(HashFile):
- os.remove(HashFile)
-
## Build a module or platform
#
# Create autogen code and makefile for a module or platform, and the launch
@@ -1836,7 +1800,7 @@ class Build():
CmdListDict = self._GenFfsCmd(Wa.ArchList)
# Add Platform and Package level hash in share_data for module hash calculation later
- if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest:
+ if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest or GlobalData.gUseHashCache:
GlobalData.gCacheIR[('PlatformHash')] = GlobalData.gPlatformHash
for PkgName in GlobalData.gPackageHash.keys():
GlobalData.gCacheIR[(PkgName, 'PackageHash')] = GlobalData.gPackageHash[PkgName]
@@ -1872,6 +1836,10 @@ class Build():
self.HashSkipModules.append(Ma)
EdkLogger.quiet("cache hit: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
continue
+ elif GlobalData.gUseHashCache and self.Target in [None, "", "all"]:
+ Ma.GenModuleIncrmtlHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyIncrmtlCache(GlobalData.gCacheIR):
+ continue
# Not to auto-gen for targets 'clean', 'cleanlib', 'cleanall', 'run', 'fds'
if self.Target not in ['clean', 'cleanlib', 'cleanall', 'run', 'fds']:
@@ -1905,8 +1873,6 @@ class Build():
Ma.PrintFirstMakeCacheMissFile(GlobalData.gCacheIR)
self.BuildModules.append(Ma)
- # Initialize all modules in tracking to 'FAIL'
- GlobalData.gModuleBuildTracking[Ma] = 'FAIL'
self.AutoGenTime += int(round((time.time() - AutoGenStart)))
MakeStart = time.time()
for Ma in self.BuildModules:
@@ -1917,7 +1883,6 @@ class Build():
# we need a full version of makefile for platform
ExitFlag.set()
BuildTask.WaitForComplete()
- self.invalidateHash()
Pa.CreateMakeFile(False)
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
# Start task scheduler
@@ -1927,7 +1892,6 @@ class Build():
# in case there's an interruption. we need a full version of makefile for platform
Pa.CreateMakeFile(False)
if BuildTask.HasError():
- self.invalidateHash()
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
self.MakeTime += int(round((time.time() - MakeStart)))
@@ -1940,7 +1904,6 @@ class Build():
self.BuildModules = []
self.MakeTime += int(round((time.time() - MakeContiue)))
if BuildTask.HasError():
- self.invalidateHash()
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
self.BuildReport.AddPlatformReport(Wa, MaList)
@@ -1993,7 +1956,6 @@ class Build():
# Save MAP buffer into MAP file.
#
self._SaveMapFile (MapBuffer, Wa)
- self.invalidateHash()
def _GenFfsCmd(self,ArchList):
# convert dictionary of Cmd:(Inf,Arch)
@@ -2105,7 +2067,7 @@ class Build():
CmdListDict = self._GenFfsCmd(Wa.ArchList)
# Add Platform and Package level hash in share_data for module hash calculation later
- if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest:
+ if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest or GlobalData.gUseHashCache:
GlobalData.gCacheIR[('PlatformHash')] = GlobalData.gPlatformHash
for PkgName in GlobalData.gPackageHash.keys():
GlobalData.gCacheIR[(PkgName, 'PackageHash')] = GlobalData.gPackageHash[PkgName]
@@ -2142,7 +2104,7 @@ class Build():
ModuleCodaFile[(ma.MetaFile.File,ma.MetaFile.Root,ma.Arch,ma.MetaFile.Path)] = [item.Target for item in ma.CodaTargetList]
Pa.DataPipe.DataContainer = {"ModuleCodaFile":ModuleCodaFile}
for Module in ModuleList:
- # Get ModuleAutoGen object to generate C code file and makefile
+ # Get ModuleAutoGen object to generate C code file and makefile
Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile,Pa.DataPipe)
if Ma is None:
@@ -2152,9 +2114,6 @@ class Build():
Ma.Workspace = Wa
PcdMaList.append(Ma)
TotalModules.append(Ma)
- # Initialize all modules in tracking to 'FAIL'
- GlobalData.gModuleBuildTracking[Ma] = 'FAIL'
-
mqueue = mp.Queue()
for m in Pa.GetAllModuleInfo:
@@ -2165,8 +2124,12 @@ class Build():
autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList,GlobalData.gCacheIR)
# Skip cache hit modules
- if GlobalData.gBinCacheSource:
+ if GlobalData.gUseHashCache:
for Ma in TotalModules:
+ if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCacheIR and \
+ GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)].IncrmtlCacheHit:
+ self.HashSkipModules.append(Ma)
+ continue
if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCacheIR and \
GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)].PreMakeCacheHit:
self.HashSkipModules.append(Ma)
@@ -2224,7 +2187,9 @@ class Build():
for IR in GlobalData.gCacheIR.keys():
if 'PlatformHash' in IR or 'PackageHash' in IR:
continue
- if GlobalData.gCacheIR[IR].PreMakeCacheHit or GlobalData.gCacheIR[IR].MakeCacheHit:
+ if GlobalData.gCacheIR[IR].PreMakeCacheHit or \
+ GlobalData.gCacheIR[IR].MakeCacheHit or \
+ GlobalData.gCacheIR[IR].IncrmtlCacheHit:
CacheHitMa.add(IR)
else:
# There might be binary module or module which has .inc files, not count for cache miss
@@ -2242,7 +2207,6 @@ class Build():
# we need a full version of makefile for platform
ExitFlag.set()
BuildTask.WaitForComplete()
- self.invalidateHash()
Pa.CreateMakeFile(False)
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
# Start task scheduler
@@ -2252,7 +2216,6 @@ class Build():
# in case there's an interruption. we need a full version of makefile for platform
if BuildTask.HasError():
- self.invalidateHash()
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
self.MakeTime += int(round((time.time() - MakeStart)))
@@ -2274,7 +2237,6 @@ class Build():
# has been signaled.
#
if BuildTask.HasError():
- self.invalidateHash()
EdkLogger.error("build", BUILD_ERROR, "Failed to build module", ExtraData=GlobalData.gBuildingModule)
# Create MAP file when Load Fix Address is enabled.
@@ -2315,7 +2277,6 @@ class Build():
#
self._SaveMapFile(MapBuffer, Wa)
self.CreateGuidedSectionToolsFile(Wa)
- self.invalidateHash()
## Generate GuidedSectionTools.txt in the FV directories.
#
def CreateGuidedSectionToolsFile(self,Wa):
--
2.22.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
2019-09-24 8:12 ` Steven Shi
@ 2019-09-27 7:57 ` Lin, Derek (HPS SW)
2019-09-29 2:31 ` Steven Shi
0 siblings, 1 reply; 6+ messages in thread
From: Lin, Derek (HPS SW) @ 2019-09-27 7:57 UTC (permalink / raw)
To: Steven Shi, devel
[-- Attachment #1.1: Type: text/plain, Size: 820 bytes --]
Hi Steven,
Thanks for the fix!
I've tried your patch with a minor fix as attached.
1 2 3 4
Build Duration: 00:05:51 00:03:47 00:05:15 00:04:31
AutoGen Duration: 00:03:32 00:03:08 00:03:14 00:03:16
Make Duration: 00:01:50 00:00:16 00:01:29 00:00:50
GenFds Duration: 00:00:30 00:00:22 00:00:31 00:00:23
1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build
BTW, this data cannot compare with previous data I provided because they are built in different environment.
In general, the result is as you said that it fix --hash functionality, but not as quick as before.
Also the cost of --hash in clean build is increased.
[-- Attachment #1.2: Type: text/html, Size: 1248 bytes --]
[-- Attachment #2: build.patch --]
[-- Type: application/octet-stream, Size: 1067 bytes --]
From 785f105b6ae6582996cb1db36f182d796729590c Mon Sep 17 00:00:00 2001
From: Derek Lin <derek.lin2@hpe.com>
Date: Fri, 27 Sep 2019 14:06:04 +0800
Subject: [PATCH] Remove gModuleBuildTracking
---
BaseTools/Source/Python/AutoGen/GenMake.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 584156dab97..4ed46de888c 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -968,8 +968,6 @@ def ProcessBuildTargetList(self):
for aFile in headerFileDependencySet:
if aFile in headerFilesInMetaFileSet:
continue
- if GlobalData.gUseHashCache:
- GlobalData.gModuleBuildTracking[self._AutoGenObject] = 'FAIL_METAFILE'
EdkLogger.warn("build","Module MetaFile [Sources] is missing local header!",
ExtraData = "Local Header: " + aFile + " not found in " + self._AutoGenObject.MetaFile.Path
)
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
2019-09-27 7:57 ` [edk2-devel] " Lin, Derek (HPS SW)
@ 2019-09-29 2:31 ` Steven Shi
2019-10-21 5:27 ` Steven Shi
0 siblings, 1 reply; 6+ messages in thread
From: Steven Shi @ 2019-09-29 2:31 UTC (permalink / raw)
To: Lin, Derek (HPS SW), devel@edk2.groups.io
[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]
Hi Derek,
Thank you for the data. I will continue to enhance the –hash performance, and will let you know after I figure out new solution.
Thanks
Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure
From: Lin, Derek (HPS SW) [mailto:derek.lin2@hpe.com]
Sent: Friday, September 27, 2019 3:57 PM
To: Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io
Subject: Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi Steven,
Thanks for the fix!
I've tried your patch with a minor fix as attached.
1 2 3 4
Build Duration: 00:05:51 00:03:47 00:05:15 00:04:31
AutoGen Duration: 00:03:32 00:03:08 00:03:14 00:03:16
Make Duration: 00:01:50 00:00:16 00:01:29 00:00:50
GenFds Duration: 00:00:30 00:00:22 00:00:31 00:00:23
1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build
BTW, this data cannot compare with previous data I provided because they are built in different environment.
In general, the result is as you said that it fix --hash functionality, but not as quick as before.
Also the cost of --hash in clean build is increased.
[-- Attachment #2: Type: text/html, Size: 5782 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
2019-09-29 2:31 ` Steven Shi
@ 2019-10-21 5:27 ` Steven Shi
0 siblings, 0 replies; 6+ messages in thread
From: Steven Shi @ 2019-10-21 5:27 UTC (permalink / raw)
To: 'Lin, Derek (HPS SW)', 'devel@edk2.groups.io'
[-- Attachment #1: Type: text/plain, Size: 2488 bytes --]
Hi Derek,
I figure out a new solution for –hash cache as below. Please try it again.
https://github.com/shijunjing/edk2/tree/hashcache_v2
It is based on latest edk2 master branch, and what you need porting are below files. Let me know if you need me to create a patch based on edk2-stable201908.
$ git status
On branch hashcache_v2
$ git show HEAD --name-only
BaseTools/Conf/build_rule.template
BaseTools/Conf/tools_def.template
BaseTools/Source/Python/AutoGen/AutoGenWorker.py
BaseTools/Source/Python/AutoGen/DataPipe.py
BaseTools/Source/Python/AutoGen/GenMake.py
BaseTools/Source/Python/AutoGen/IncludesAutoGen.py
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
BaseTools/Source/Python/Common/GlobalData.py
BaseTools/Source/Python/build/build.py
Thanks
Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure
From: Shi, Steven
Sent: Sunday, September 29, 2019 10:32 AM
To: Lin, Derek (HPS SW) <derek.lin2@hpe.com>; devel@edk2.groups.io
Subject: RE: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi Derek,
Thank you for the data. I will continue to enhance the –hash performance, and will let you know after I figure out new solution.
Thanks
Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure
From: Lin, Derek (HPS SW) [mailto:derek.lin2@hpe.com]
Sent: Friday, September 27, 2019 3:57 PM
To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: Re: [edk2-devel] BaseTools --hash malfunction after migrate from stable201905 to stable201908
Hi Steven,
Thanks for the fix!
I've tried your patch with a minor fix as attached.
1 2 3 4
Build Duration: 00:05:51 00:03:47 00:05:15 00:04:31
AutoGen Duration: 00:03:32 00:03:08 00:03:14 00:03:16
Make Duration: 00:01:50 00:00:16 00:01:29 00:00:50
GenFds Duration: 00:00:30 00:00:22 00:00:31 00:00:23
1: clean build with patch and --hash
2: incremental build
3. clean build without patch and without --hash
4. incremental build
BTW, this data cannot compare with previous data I provided because they are built in different environment.
In general, the result is as you said that it fix --hash functionality, but not as quick as before.
Also the cost of --hash in clean build is increased.
[-- Attachment #2: Type: text/html, Size: 11146 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-10-21 5:27 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-09-20 6:54 BaseTools --hash malfunction after migrate from stable201905 to stable201908 Lin, Derek (HPS SW)
2019-09-23 7:51 ` Steven Shi
2019-09-24 8:12 ` Steven Shi
2019-09-27 7:57 ` [edk2-devel] " Lin, Derek (HPS SW)
2019-09-29 2:31 ` Steven Shi
2019-10-21 5:27 ` Steven Shi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox