* [PATCH 0/4] Build cache enhancement
@ 2019-08-08 12:25 Steven Shi
2019-08-08 12:25 ` [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache Steven Shi
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: Steven Shi @ 2019-08-08 12:25 UTC (permalink / raw)
To: devel
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson,
Shi, Steven
From: "Shi, Steven" <steven.shi@intel.com>
Enhance the edk2 build cache with below patches:
Patch 01/04: Improve the cache hit rate through new cache checkpoint and hash algorithm
Patch 02/04: Print more info to explain why a module build cache miss
Patch 03/04: Fix the unsafe [self.Arch][self.Name] key usage in build cache
Patch 04/04 Add the GenFds multi-thread support in build cache
This patch set is based on patch set of [Patch 00/10 V8] Enable multiple process AutoGen
https://edk2.groups.io/g/devel/topic/patch_00_10_v8_enable/32779325?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,140,32779325
You can directly try this patch set in the branch:
https://github.com/shijunjing/edk2/tree/build_cache_improve_v1
Shi, Steven (4):
BaseTools: Improve the cache hit in the edk2 build cache
BaseTools: Print first cache missing file for build cachle
BaseTools: Change the [Arch][Name] module key in Build cache
BaseTools: Add GenFds multi-thread support in build cache
.../Source/Python/AutoGen/AutoGenWorker.py | 23 +
BaseTools/Source/Python/AutoGen/CacheIR.py | 28 +
BaseTools/Source/Python/AutoGen/DataPipe.py | 8 +
BaseTools/Source/Python/AutoGen/GenMake.py | 229 +++---
.../Source/Python/AutoGen/ModuleAutoGen.py | 742 ++++++++++++++++--
BaseTools/Source/Python/Common/GlobalData.py | 9 +
BaseTools/Source/Python/build/build.py | 171 ++--
7 files changed, 979 insertions(+), 231 deletions(-)
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
create mode 100755 BaseTools/Source/Python/AutoGen/CacheIR.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/DataPipe.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/GenMake.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
mode change 100644 => 100755 BaseTools/Source/Python/Common/GlobalData.py
mode change 100644 => 100755 BaseTools/Source/Python/build/build.py
--
2.17.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
@ 2019-08-08 12:25 ` Steven Shi
2019-09-06 20:45 ` commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error vladimir.olovyannikov
2019-08-08 12:25 ` [PATCH 2/4] BaseTools: Print first cache missing file for build cachle Steven Shi
` (3 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: Steven Shi @ 2019-08-08 12:25 UTC (permalink / raw)
To: devel
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson,
Shi, Steven
From: "Shi, Steven" <steven.shi@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1927
Current cache hash algorithm does not parse and generate
the makefile to get the accurate dependency files for a
module. It instead use the platform and package meta files
to get the module depenedency in a quick but over approximate
way. These meta files are monolithic and involve many redundant
dependency for the module, which cause the module build
cache miss easily.
This patch introduces one more cache checkpoint and a new
hash algorithm besides the current quick one. The new hash
algorithm leverages the module makefile to achieve more
accurate and precise dependency info for a module. When
the build cache miss with the first quick hash, the
Basetool will caculate new one after makefile is generated
and then check again.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 21 +++++++++++++++++++++
BaseTools/Source/Python/AutoGen/CacheIR.py | 28 ++++++++++++++++++++++++++++
BaseTools/Source/Python/AutoGen/DataPipe.py | 8 ++++++++
BaseTools/Source/Python/AutoGen/GenMake.py | 223 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------------------------------------------------------------------------
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 643 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------------------
BaseTools/Source/Python/Common/GlobalData.py | 9 +++++++++
BaseTools/Source/Python/build/build.py | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------
7 files changed, 862 insertions(+), 192 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
old mode 100644
new mode 100755
index e583828741..a84ed46f2e
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -182,6 +182,12 @@ class AutoGenWorkerInProcess(mp.Process):
GlobalData.gDisableIncludePathCheck = False
GlobalData.gFdfParser = self.data_pipe.Get("FdfParser")
GlobalData.gDatabasePath = self.data_pipe.Get("DatabasePath")
+ GlobalData.gBinCacheSource = self.data_pipe.Get("BinCacheSource")
+ GlobalData.gBinCacheDest = self.data_pipe.Get("BinCacheDest")
+ GlobalData.gCacheIR = self.data_pipe.Get("CacheIR")
+ GlobalData.gEnableGenfdsMultiThread = self.data_pipe.Get("EnableGenfdsMultiThread")
+ GlobalData.file_lock = self.file_lock
+ CommandTarget = self.data_pipe.Get("CommandTarget")
pcd_from_build_option = []
for pcd_tuple in self.data_pipe.Get("BuildOptPcd"):
pcd_id = ".".join((pcd_tuple[0],pcd_tuple[1]))
@@ -193,10 +199,13 @@ class AutoGenWorkerInProcess(mp.Process):
FfsCmd = self.data_pipe.Get("FfsCommand")
if FfsCmd is None:
FfsCmd = {}
+ GlobalData.FfsCmd = FfsCmd
PlatformMetaFile = self.GetPlatformMetaFile(self.data_pipe.Get("P_Info").get("ActivePlatform"),
self.data_pipe.Get("P_Info").get("WorkspaceDir"))
libConstPcd = self.data_pipe.Get("LibConstPcd")
Refes = self.data_pipe.Get("REFS")
+ GlobalData.libConstPcd = libConstPcd
+ GlobalData.Refes = Refes
while True:
if self.module_queue.empty():
break
@@ -223,8 +232,20 @@ class AutoGenWorkerInProcess(mp.Process):
Ma.ConstPcd = libConstPcd[(Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)]
if (Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path) in Refes:
Ma.ReferenceModules = Refes[(Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)]
+ if GlobalData.gBinCacheSource and CommandTarget in [None, "", "all"]:
+ Ma.GenModuleFilesHash(GlobalData.gCacheIR)
+ Ma.GenPreMakefileHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheIR):
+ continue
+
Ma.CreateCodeFile(False)
Ma.CreateMakeFile(False,GenFfsList=FfsCmd.get((Ma.MetaFile.File, Ma.Arch),[]))
+
+ if GlobalData.gBinCacheSource and CommandTarget in [None, "", "all"]:
+ Ma.GenMakeHeaderFilesHash(GlobalData.gCacheIR)
+ Ma.GenMakeHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyMakeCache(GlobalData.gCacheIR):
+ continue
except Empty:
pass
except:
diff --git a/BaseTools/Source/Python/AutoGen/CacheIR.py b/BaseTools/Source/Python/AutoGen/CacheIR.py
new file mode 100755
index 0000000000..2d9ffe3f0b
--- /dev/null
+++ b/BaseTools/Source/Python/AutoGen/CacheIR.py
@@ -0,0 +1,28 @@
+## @file
+# Build cache intermediate result and state
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+class ModuleBuildCacheIR():
+ def __init__(self, Path, Arch):
+ self.ModulePath = Path
+ self.ModuleArch = Arch
+ self.ModuleFilesHashDigest = None
+ self.ModuleFilesHashHexDigest = None
+ self.ModuleFilesChain = []
+ self.PreMakefileHashHexDigest = None
+ self.CreateCodeFileDone = False
+ self.CreateMakeFileDone = False
+ self.MakefilePath = None
+ self.AutoGenFileList = None
+ self.DependencyHeaderFileSet = None
+ self.MakeHeaderFilesHashChain = None
+ self.MakeHeaderFilesHashDigest = None
+ self.MakeHeaderFilesHashChain = []
+ self.MakeHashDigest = None
+ self.MakeHashHexDigest = None
+ self.MakeHashChain = []
+ self.PreMakeCacheHit = False
+ self.MakeCacheHit = False
diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source/Python/AutoGen/DataPipe.py
old mode 100644
new mode 100755
index 2052084bdb..84e77c301a
--- a/BaseTools/Source/Python/AutoGen/DataPipe.py
+++ b/BaseTools/Source/Python/AutoGen/DataPipe.py
@@ -158,3 +158,11 @@ class MemoryDataPipe(DataPipe):
self.DataContainer = {"FdfParser": True if GlobalData.gFdfParser else False}
self.DataContainer = {"LogLevel": EdkLogger.GetLevel()}
+
+ self.DataContainer = {"BinCacheSource":GlobalData.gBinCacheSource}
+
+ self.DataContainer = {"BinCacheDest":GlobalData.gBinCacheDest}
+
+ self.DataContainer = {"CacheIR":GlobalData.gCacheIR}
+
+ self.DataContainer = {"EnableGenfdsMultiThread":GlobalData.gEnableGenfdsMultiThread}
\ No newline at end of file
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
old mode 100644
new mode 100755
index 5802ae5ae4..79387856bd
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -906,6 +906,11 @@ cleanlib:
self._AutoGenObject.IncludePathList + self._AutoGenObject.BuildOptionIncPathList
)
+ self.DependencyHeaderFileSet = set()
+ if FileDependencyDict:
+ for Dependency in FileDependencyDict.values():
+ self.DependencyHeaderFileSet.update(set(Dependency))
+
# Check if header files are listed in metafile
# Get a list of unique module header source files from MetaFile
headerFilesInMetaFileSet = set()
@@ -1096,7 +1101,7 @@ cleanlib:
## For creating makefile targets for dependent libraries
def ProcessDependentLibrary(self):
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
+ if not LibraryAutoGen.IsBinaryModule:
self.LibraryBuildDirectoryList.append(self.PlaceMacro(LibraryAutoGen.BuildDir, self.Macros))
## Return a list containing source file's dependencies
@@ -1110,114 +1115,9 @@ cleanlib:
def GetFileDependency(self, FileList, ForceInculeList, SearchPathList):
Dependency = {}
for F in FileList:
- Dependency[F] = self.GetDependencyList(F, ForceInculeList, SearchPathList)
+ Dependency[F] = GetDependencyList(self._AutoGenObject, self.FileCache, F, ForceInculeList, SearchPathList)
return Dependency
- ## Find dependencies for one source file
- #
- # By searching recursively "#include" directive in file, find out all the
- # files needed by given source file. The dependencies will be only searched
- # in given search path list.
- #
- # @param File The source file
- # @param ForceInculeList The list of files which will be included forcely
- # @param SearchPathList The list of search path
- #
- # @retval list The list of files the given source file depends on
- #
- def GetDependencyList(self, File, ForceList, SearchPathList):
- EdkLogger.debug(EdkLogger.DEBUG_1, "Try to get dependency files for %s" % File)
- FileStack = [File] + ForceList
- DependencySet = set()
-
- if self._AutoGenObject.Arch not in gDependencyDatabase:
- gDependencyDatabase[self._AutoGenObject.Arch] = {}
- DepDb = gDependencyDatabase[self._AutoGenObject.Arch]
-
- while len(FileStack) > 0:
- F = FileStack.pop()
-
- FullPathDependList = []
- if F in self.FileCache:
- for CacheFile in self.FileCache[F]:
- FullPathDependList.append(CacheFile)
- if CacheFile not in DependencySet:
- FileStack.append(CacheFile)
- DependencySet.update(FullPathDependList)
- continue
-
- CurrentFileDependencyList = []
- if F in DepDb:
- CurrentFileDependencyList = DepDb[F]
- else:
- try:
- Fd = open(F.Path, 'rb')
- FileContent = Fd.read()
- Fd.close()
- except BaseException as X:
- EdkLogger.error("build", FILE_OPEN_FAILURE, ExtraData=F.Path + "\n\t" + str(X))
- if len(FileContent) == 0:
- continue
- try:
- if FileContent[0] == 0xff or FileContent[0] == 0xfe:
- FileContent = FileContent.decode('utf-16')
- else:
- FileContent = FileContent.decode()
- except:
- # The file is not txt file. for example .mcb file
- continue
- IncludedFileList = gIncludePattern.findall(FileContent)
-
- for Inc in IncludedFileList:
- Inc = Inc.strip()
- # if there's macro used to reference header file, expand it
- HeaderList = gMacroPattern.findall(Inc)
- if len(HeaderList) == 1 and len(HeaderList[0]) == 2:
- HeaderType = HeaderList[0][0]
- HeaderKey = HeaderList[0][1]
- if HeaderType in gIncludeMacroConversion:
- Inc = gIncludeMacroConversion[HeaderType] % {"HeaderKey" : HeaderKey}
- else:
- # not known macro used in #include, always build the file by
- # returning a empty dependency
- self.FileCache[File] = []
- return []
- Inc = os.path.normpath(Inc)
- CurrentFileDependencyList.append(Inc)
- DepDb[F] = CurrentFileDependencyList
-
- CurrentFilePath = F.Dir
- PathList = [CurrentFilePath] + SearchPathList
- for Inc in CurrentFileDependencyList:
- for SearchPath in PathList:
- FilePath = os.path.join(SearchPath, Inc)
- if FilePath in gIsFileMap:
- if not gIsFileMap[FilePath]:
- continue
- # If isfile is called too many times, the performance is slow down.
- elif not os.path.isfile(FilePath):
- gIsFileMap[FilePath] = False
- continue
- else:
- gIsFileMap[FilePath] = True
- FilePath = PathClass(FilePath)
- FullPathDependList.append(FilePath)
- if FilePath not in DependencySet:
- FileStack.append(FilePath)
- break
- else:
- EdkLogger.debug(EdkLogger.DEBUG_9, "%s included by %s was not found "\
- "in any given path:\n\t%s" % (Inc, F, "\n\t".join(SearchPathList)))
-
- self.FileCache[F] = FullPathDependList
- DependencySet.update(FullPathDependList)
-
- DependencySet.update(ForceList)
- if File in DependencySet:
- DependencySet.remove(File)
- DependencyList = list(DependencySet) # remove duplicate ones
-
- return DependencyList
## CustomMakefile class
#
@@ -1599,7 +1499,7 @@ cleanlib:
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
+ if not LibraryAutoGen.IsBinaryModule:
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
@@ -1735,7 +1635,7 @@ class TopLevelMakefile(BuildFile):
def GetLibraryBuildDirectoryList(self):
DirList = []
for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
- if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
+ if not LibraryAutoGen.IsBinaryModule:
DirList.append(os.path.join(self._AutoGenObject.BuildDir, LibraryAutoGen.BuildDir))
return DirList
@@ -1743,3 +1643,108 @@ class TopLevelMakefile(BuildFile):
if __name__ == '__main__':
pass
+## Find dependencies for one source file
+#
+# By searching recursively "#include" directive in file, find out all the
+# files needed by given source file. The dependencies will be only searched
+# in given search path list.
+#
+# @param File The source file
+# @param ForceInculeList The list of files which will be included forcely
+# @param SearchPathList The list of search path
+#
+# @retval list The list of files the given source file depends on
+#
+def GetDependencyList(AutoGenObject, FileCache, File, ForceList, SearchPathList):
+ EdkLogger.debug(EdkLogger.DEBUG_1, "Try to get dependency files for %s" % File)
+ FileStack = [File] + ForceList
+ DependencySet = set()
+
+ if AutoGenObject.Arch not in gDependencyDatabase:
+ gDependencyDatabase[AutoGenObject.Arch] = {}
+ DepDb = gDependencyDatabase[AutoGenObject.Arch]
+
+ while len(FileStack) > 0:
+ F = FileStack.pop()
+
+ FullPathDependList = []
+ if F in FileCache:
+ for CacheFile in FileCache[F]:
+ FullPathDependList.append(CacheFile)
+ if CacheFile not in DependencySet:
+ FileStack.append(CacheFile)
+ DependencySet.update(FullPathDependList)
+ continue
+
+ CurrentFileDependencyList = []
+ if F in DepDb:
+ CurrentFileDependencyList = DepDb[F]
+ else:
+ try:
+ Fd = open(F.Path, 'rb')
+ FileContent = Fd.read()
+ Fd.close()
+ except BaseException as X:
+ EdkLogger.error("build", FILE_OPEN_FAILURE, ExtraData=F.Path + "\n\t" + str(X))
+ if len(FileContent) == 0:
+ continue
+ try:
+ if FileContent[0] == 0xff or FileContent[0] == 0xfe:
+ FileContent = FileContent.decode('utf-16')
+ else:
+ FileContent = FileContent.decode()
+ except:
+ # The file is not txt file. for example .mcb file
+ continue
+ IncludedFileList = gIncludePattern.findall(FileContent)
+
+ for Inc in IncludedFileList:
+ Inc = Inc.strip()
+ # if there's macro used to reference header file, expand it
+ HeaderList = gMacroPattern.findall(Inc)
+ if len(HeaderList) == 1 and len(HeaderList[0]) == 2:
+ HeaderType = HeaderList[0][0]
+ HeaderKey = HeaderList[0][1]
+ if HeaderType in gIncludeMacroConversion:
+ Inc = gIncludeMacroConversion[HeaderType] % {"HeaderKey" : HeaderKey}
+ else:
+ # not known macro used in #include, always build the file by
+ # returning a empty dependency
+ FileCache[File] = []
+ return []
+ Inc = os.path.normpath(Inc)
+ CurrentFileDependencyList.append(Inc)
+ DepDb[F] = CurrentFileDependencyList
+
+ CurrentFilePath = F.Dir
+ PathList = [CurrentFilePath] + SearchPathList
+ for Inc in CurrentFileDependencyList:
+ for SearchPath in PathList:
+ FilePath = os.path.join(SearchPath, Inc)
+ if FilePath in gIsFileMap:
+ if not gIsFileMap[FilePath]:
+ continue
+ # If isfile is called too many times, the performance is slow down.
+ elif not os.path.isfile(FilePath):
+ gIsFileMap[FilePath] = False
+ continue
+ else:
+ gIsFileMap[FilePath] = True
+ FilePath = PathClass(FilePath)
+ FullPathDependList.append(FilePath)
+ if FilePath not in DependencySet:
+ FileStack.append(FilePath)
+ break
+ else:
+ EdkLogger.debug(EdkLogger.DEBUG_9, "%s included by %s was not found "\
+ "in any given path:\n\t%s" % (Inc, F, "\n\t".join(SearchPathList)))
+
+ FileCache[F] = FullPathDependList
+ DependencySet.update(FullPathDependList)
+
+ DependencySet.update(ForceList)
+ if File in DependencySet:
+ DependencySet.remove(File)
+ DependencyList = list(DependencySet) # remove duplicate ones
+
+ return DependencyList
\ No newline at end of file
diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
old mode 100644
new mode 100755
index ed6822334e..5749b8a9fa
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -26,6 +26,8 @@ from Workspace.MetaFileCommentParser import UsageList
from .GenPcdDb import CreatePcdDatabaseCode
from Common.caching import cached_class_function
from AutoGen.ModuleAutoGenHelper import PlatformInfo,WorkSpaceInfo
+from AutoGen.CacheIR import ModuleBuildCacheIR
+import json
## Mapping Makefile type
gMakeTypeMap = {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"}
@@ -252,6 +254,8 @@ class ModuleAutoGen(AutoGen):
self.AutoGenDepSet = set()
self.ReferenceModules = []
self.ConstPcd = {}
+ self.Makefile = None
+ self.FileDependCache = {}
def __init_platform_info__(self):
pinfo = self.DataPipe.Get("P_Info")
@@ -1594,12 +1598,37 @@ class ModuleAutoGen(AutoGen):
self.IsAsBuiltInfCreated = True
+ def CacheCopyFile(self, OriginDir, CopyDir, File):
+ sub_dir = os.path.relpath(File, CopyDir)
+ destination_file = os.path.join(OriginDir, sub_dir)
+ destination_dir = os.path.dirname(destination_file)
+ CreateDirectory(destination_dir)
+ try:
+ CopyFileOnChange(File, destination_dir)
+ except:
+ EdkLogger.quiet("[cache warning]: fail to copy file:%s to folder:%s" % (File, destination_dir))
+ return
+
def CopyModuleToCache(self):
- FileDir = path.join(GlobalData.gBinCacheDest, self.PlatformInfo.Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
+ self.GenPreMakefileHash(GlobalData.gCacheIR)
+ if not (self.MetaFile.Path, self.Arch) in GlobalData.gCacheIR or \
+ not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate PreMakefileHash for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return False
+
+ self.GenMakeHash(GlobalData.gCacheIR)
+ if not (self.MetaFile.Path, self.Arch) in GlobalData.gCacheIR or \
+ not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHashChain or \
+ not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate MakeHashChain for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return False
+
+ MakeHashStr = str(GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest)
+ FileDir = path.join(GlobalData.gBinCacheDest, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName, MakeHashStr)
+ FfsDir = path.join(GlobalData.gBinCacheDest, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs", self.Guid + self.Name, MakeHashStr)
+
CreateDirectory (FileDir)
- HashFile = path.join(self.BuildDir, self.Name + '.hash')
- if os.path.exists(HashFile):
- CopyFileOnChange(HashFile, FileDir)
+ self.SaveHashChainFileToCache(GlobalData.gCacheIR)
ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
if os.path.exists(ModuleFile):
CopyFileOnChange(ModuleFile, FileDir)
@@ -1617,38 +1646,76 @@ class ModuleAutoGen(AutoGen):
CreateDirectory(destination_dir)
CopyFileOnChange(File, destination_dir)
- def AttemptModuleCacheCopy(self):
- # If library or Module is binary do not skip by hash
- if self.IsBinaryModule:
+ def SaveHashChainFileToCache(self, gDict):
+ if not GlobalData.gBinCacheDest:
+ return False
+
+ self.GenPreMakefileHash(gDict)
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate PreMakefileHash for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return False
+
+ self.GenMakeHash(gDict)
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain or \
+ not gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate MakeHashChain for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return False
+
+ # save the hash chain list as cache file
+ MakeHashStr = str(GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest)
+ CacheDestDir = path.join(GlobalData.gBinCacheDest, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
+ CacheHashDestDir = path.join(CacheDestDir, MakeHashStr)
+ ModuleHashPair = path.join(CacheDestDir, self.Name + ".ModuleHashPair")
+ MakeHashChain = path.join(CacheHashDestDir, self.Name + ".MakeHashChain")
+ ModuleFilesChain = path.join(CacheHashDestDir, self.Name + ".ModuleFilesChain")
+
+ # save the HashChainDict as json file
+ CreateDirectory (CacheDestDir)
+ CreateDirectory (CacheHashDestDir)
+ try:
+ ModuleHashPairList = [] # tuple list: [tuple(PreMakefileHash, MakeHash)]
+ if os.path.exists(ModuleHashPair):
+ f = open(ModuleHashPair, 'r')
+ ModuleHashPairList = json.load(f)
+ f.close()
+ PreMakeHash = gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest
+ MakeHash = gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest
+ ModuleHashPairList.append((PreMakeHash, MakeHash))
+ ModuleHashPairList = list(set(map(tuple, ModuleHashPairList)))
+ with open(ModuleHashPair, 'w') as f:
+ json.dump(ModuleHashPairList, f, indent=2)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache warning]: fail to save ModuleHashPair file in cache: %s" % ModuleHashPair)
+ return False
+
+ try:
+ with open(MakeHashChain, 'w') as f:
+ json.dump(gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain, f, indent=2)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache warning]: fail to save MakeHashChain file in cache: %s" % MakeHashChain)
+ return False
+
+ try:
+ with open(ModuleFilesChain, 'w') as f:
+ json.dump(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain, f, indent=2)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache warning]: fail to save ModuleFilesChain file in cache: %s" % ModuleFilesChain)
return False
- # .inc is contains binary information so do not skip by hash as well
- for f_ext in self.SourceFileList:
- if '.inc' in str(f_ext):
- return False
- FileDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
- HashFile = path.join(FileDir, self.Name + '.hash')
- if os.path.exists(HashFile):
- f = open(HashFile, 'r')
- CacheHash = f.read()
- f.close()
- self.GenModuleHash()
- if GlobalData.gModuleHash[self.Arch][self.Name]:
- if CacheHash == GlobalData.gModuleHash[self.Arch][self.Name]:
- for root, dir, files in os.walk(FileDir):
- for f in files:
- if self.Name + '.hash' in f:
- CopyFileOnChange(HashFile, self.BuildDir)
- else:
- File = path.join(root, f)
- sub_dir = os.path.relpath(File, FileDir)
- destination_file = os.path.join(self.OutputDir, sub_dir)
- destination_dir = os.path.dirname(destination_file)
- CreateDirectory(destination_dir)
- CopyFileOnChange(File, destination_dir)
- if self.Name == "PcdPeim" or self.Name == "PcdDxe":
- CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
- return True
- return False
+
+ # save the autogenfile and makefile for debug usage
+ CacheDebugDir = path.join(CacheHashDestDir, "CacheDebug")
+ CreateDirectory (CacheDebugDir)
+ CopyFileOnChange(gDict[(self.MetaFile.Path, self.Arch)].MakefilePath, CacheDebugDir)
+ if gDict[(self.MetaFile.Path, self.Arch)].AutoGenFileList:
+ for File in gDict[(self.MetaFile.Path, self.Arch)].AutoGenFileList:
+ CopyFileOnChange(str(File), CacheDebugDir)
+
+ return True
## Create makefile for the module and its dependent libraries
#
@@ -1657,6 +1724,11 @@ class ModuleAutoGen(AutoGen):
#
@cached_class_function
def CreateMakeFile(self, CreateLibraryMakeFile=True, GenFfsList = []):
+ gDict = GlobalData.gCacheIR
+ if (self.MetaFile.Path, self.Arch) in gDict and \
+ gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone:
+ return
+
# nest this function inside it's only caller.
def CreateTimeStamp():
FileSet = {self.MetaFile.Path}
@@ -1687,8 +1759,8 @@ class ModuleAutoGen(AutoGen):
for LibraryAutoGen in self.LibraryAutoGenList:
LibraryAutoGen.CreateMakeFile()
- # Don't enable if hash feature enabled, CanSkip uses timestamps to determine build skipping
- if not GlobalData.gUseHashCache and self.CanSkip():
+ # CanSkip uses timestamps to determine build skipping
+ if self.CanSkip():
return
if len(self.CustomMakefile) == 0:
@@ -1704,6 +1776,24 @@ class ModuleAutoGen(AutoGen):
CreateTimeStamp()
+ MakefileType = Makefile._FileType
+ MakefileName = Makefile._FILE_NAME_[MakefileType]
+ MakefilePath = os.path.join(self.MakeFileDir, MakefileName)
+
+ MewIR = ModuleBuildCacheIR(self.MetaFile.Path, self.Arch)
+ MewIR.MakefilePath = MakefilePath
+ MewIR.DependencyHeaderFileSet = Makefile.DependencyHeaderFileSet
+ MewIR.CreateMakeFileDone = True
+ with GlobalData.file_lock:
+ try:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.MakefilePath = MakefilePath
+ IR.DependencyHeaderFileSet = Makefile.DependencyHeaderFileSet
+ IR.CreateMakeFileDone = True
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ except:
+ gDict[(self.MetaFile.Path, self.Arch)] = MewIR
+
def CopyBinaryFiles(self):
for File in self.Module.Binaries:
SrcPath = File.Path
@@ -1715,6 +1805,11 @@ class ModuleAutoGen(AutoGen):
# dependent libraries will be created
#
def CreateCodeFile(self, CreateLibraryCodeFile=True):
+ gDict = GlobalData.gCacheIR
+ if (self.MetaFile.Path, self.Arch) in gDict and \
+ gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone:
+ return
+
if self.IsCodeFileCreated:
return
@@ -1730,8 +1825,9 @@ class ModuleAutoGen(AutoGen):
if not self.IsLibrary and CreateLibraryCodeFile:
for LibraryAutoGen in self.LibraryAutoGenList:
LibraryAutoGen.CreateCodeFile()
- # Don't enable if hash feature enabled, CanSkip uses timestamps to determine build skipping
- if not GlobalData.gUseHashCache and self.CanSkip():
+
+ # CanSkip uses timestamps to determine build skipping
+ if self.CanSkip():
return
AutoGenList = []
@@ -1771,6 +1867,16 @@ class ModuleAutoGen(AutoGen):
(" ".join(AutoGenList), " ".join(IgoredAutoGenList), self.Name, self.Arch))
self.IsCodeFileCreated = True
+ MewIR = ModuleBuildCacheIR(self.MetaFile.Path, self.Arch)
+ MewIR.CreateCodeFileDone = True
+ with GlobalData.file_lock:
+ try:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.CreateCodeFileDone = True
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ except:
+ gDict[(self.MetaFile.Path, self.Arch)] = MewIR
+
return AutoGenList
## Summarize the ModuleAutoGen objects of all libraries used by this module
@@ -1840,46 +1946,469 @@ class ModuleAutoGen(AutoGen):
return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8')
+ def GenModuleFilesHash(self, gDict):
+ # Early exit if module or library has been hashed and is in memory
+ if (self.MetaFile.Path, self.Arch) in gDict:
+ if gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain:
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ DependencyFileSet = set()
+ # Add Module Meta file
+ DependencyFileSet.add(self.MetaFile)
+
+ # Add Module's source files
+ if self.SourceFileList:
+ for File in set(self.SourceFileList):
+ DependencyFileSet.add(File)
+
+ # Add modules's include header files
+ # Search dependency file list for each source file
+ SourceFileList = []
+ OutPutFileList = []
+ for Target in self.IntroTargetList:
+ SourceFileList.extend(Target.Inputs)
+ OutPutFileList.extend(Target.Outputs)
+ if OutPutFileList:
+ for Item in OutPutFileList:
+ if Item in SourceFileList:
+ SourceFileList.remove(Item)
+ SearchList = []
+ for file_path in self.IncludePathList + self.BuildOptionIncPathList:
+ # skip the folders in platform BuildDir which are not been generated yet
+ if file_path.startswith(os.path.abspath(self.PlatformInfo.BuildDir)+os.sep):
+ continue
+ SearchList.append(file_path)
+ FileDependencyDict = {}
+ ForceIncludedFile = []
+ for F in SourceFileList:
+ # skip the files which are not been generated yet, because
+ # the SourceFileList usually contains intermediate build files, e.g. AutoGen.c
+ if not os.path.exists(F.Path):
+ continue
+ FileDependencyDict[F] = GenMake.GetDependencyList(self, self.FileDependCache, F, ForceIncludedFile, SearchList)
+
+ if FileDependencyDict:
+ for Dependency in FileDependencyDict.values():
+ DependencyFileSet.update(set(Dependency))
+
+ # Caculate all above dependency files hash
+ # Initialze hash object
+ FileList = []
+ m = hashlib.md5()
+ for File in sorted(DependencyFileSet, key=lambda x: str(x)):
+ if not os.path.exists(str(File)):
+ EdkLogger.quiet("[cache warning]: header file %s is missing for module: %s[%s]" % (File, self.MetaFile.Path, self.Arch))
+ continue
+ f = open(str(File), 'rb')
+ Content = f.read()
+ f.close()
+ m.update(Content)
+ FileList.append((str(File), hashlib.md5(Content).hexdigest()))
+
+
+ MewIR = ModuleBuildCacheIR(self.MetaFile.Path, self.Arch)
+ MewIR.ModuleFilesHashDigest = m.digest()
+ MewIR.ModuleFilesHashHexDigest = m.hexdigest()
+ MewIR.ModuleFilesChain = FileList
+ with GlobalData.file_lock:
+ try:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.ModuleFilesHashDigest = m.digest()
+ IR.ModuleFilesHashHexDigest = m.hexdigest()
+ IR.ModuleFilesChain = FileList
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ except:
+ gDict[(self.MetaFile.Path, self.Arch)] = MewIR
+
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ def GenPreMakefileHash(self, gDict):
+ # Early exit if module or library has been hashed and is in memory
+ if (self.MetaFile.Path, self.Arch) in gDict and \
+ gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest:
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ # skip binary module
+ if self.IsBinaryModule:
+ return
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest:
+ self.GenModuleFilesHash(gDict)
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest:
+ EdkLogger.quiet("[cache warning]: Cannot generate ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return
+
+ # Initialze hash object
+ m = hashlib.md5()
+
+ # Add Platform level hash
+ if ('PlatformHash') in gDict:
+ m.update(gDict[('PlatformHash')].encode('utf-8'))
+ else:
+ EdkLogger.quiet("[cache warning]: PlatformHash is missing")
+
+ # Add Package level hash
+ if self.DependentPackageList:
+ for Pkg in sorted(self.DependentPackageList, key=lambda x: x.PackageName):
+ if (Pkg.PackageName, 'PackageHash') in gDict:
+ m.update(gDict[(Pkg.PackageName, 'PackageHash')].encode('utf-8'))
+ else:
+ EdkLogger.quiet("[cache warning]: %s PackageHash needed by %s[%s] is missing" %(Pkg.PackageName, self.MetaFile.Name, self.Arch))
+
+ # Add Library hash
+ if self.LibraryAutoGenList:
+ for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name):
+ if not (Lib.MetaFile.Path, Lib.Arch) in gDict or \
+ not gDict[(Lib.MetaFile.Path, Lib.Arch)].ModuleFilesHashDigest:
+ Lib.GenPreMakefileHash(gDict)
+ m.update(gDict[(Lib.MetaFile.Path, Lib.Arch)].ModuleFilesHashDigest)
+
+ # Add Module self
+ m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest)
+
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.PreMakefileHashHexDigest = m.hexdigest()
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ def GenMakeHeaderFilesHash(self, gDict):
+ # Early exit if module or library has been hashed and is in memory
+ if (self.MetaFile.Path, self.Arch) in gDict and \
+ gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest:
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ # skip binary module
+ if self.IsBinaryModule:
+ return
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone:
+ if self.IsLibrary:
+ if (self.MetaFile.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path) in GlobalData.libConstPcd:
+ self.ConstPcd = GlobalData.libConstPcd[(self.MetaFile.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path)]
+ if (self.MetaFile.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path) in GlobalData.Refes:
+ self.ReferenceModules = GlobalData.Refes[(self.MetaFile.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path)]
+ self.CreateCodeFile()
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone:
+ self.CreateMakeFile(GenFfsList=GlobalData.FfsCmd.get((self.MetaFile.File, self.Arch),[]))
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone or \
+ not gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone:
+ EdkLogger.quiet("[cache warning]: Cannot create CodeFile or Makefile for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return
+
+ DependencyFileSet = set()
+ # Add Makefile
+ if gDict[(self.MetaFile.Path, self.Arch)].MakefilePath:
+ DependencyFileSet.add(gDict[(self.MetaFile.Path, self.Arch)].MakefilePath)
+ else:
+ EdkLogger.quiet("[cache warning]: makefile is missing for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+
+ # Add header files
+ if gDict[(self.MetaFile.Path, self.Arch)].DependencyHeaderFileSet:
+ for File in gDict[(self.MetaFile.Path, self.Arch)].DependencyHeaderFileSet:
+ DependencyFileSet.add(File)
+ else:
+ EdkLogger.quiet("[cache warning]: No dependency header found for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+
+ # Add AutoGen files
+ if self.AutoGenFileList:
+ for File in set(self.AutoGenFileList):
+ DependencyFileSet.add(File)
+
+ # Caculate all above dependency files hash
+ # Initialze hash object
+ FileList = []
+ m = hashlib.md5()
+ for File in sorted(DependencyFileSet, key=lambda x: str(x)):
+ if not os.path.exists(str(File)):
+ EdkLogger.quiet("[cache warning]: header file: %s doesn't exist for module: %s[%s]" % (File, self.MetaFile.Path, self.Arch))
+ continue
+ f = open(str(File), 'rb')
+ Content = f.read()
+ f.close()
+ m.update(Content)
+ FileList.append((str(File), hashlib.md5(Content).hexdigest()))
+
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.AutoGenFileList = self.AutoGenFileList.keys()
+ IR.MakeHeaderFilesHashChain = FileList
+ IR.MakeHeaderFilesHashDigest = m.digest()
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ def GenMakeHash(self, gDict):
+ # Early exit if module or library has been hashed and is in memory
+ if (self.MetaFile.Path, self.Arch) in gDict and \
+ gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain:
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ # skip binary module
+ if self.IsBinaryModule:
+ return
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest:
+ self.GenModuleFilesHash(gDict)
+ if not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest:
+ self.GenMakeHeaderFilesHash(gDict)
+
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest or \
+ not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain or \
+ not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest or \
+ not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashChain:
+ EdkLogger.quiet("[cache warning]: Cannot generate ModuleFilesHash or MakeHeaderFilesHash for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return
+
+ # Initialze hash object
+ m = hashlib.md5()
+ MakeHashChain = []
+
+ # Add hash of makefile and dependency header files
+ m.update(gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest)
+ New = list(set(gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashChain) - set(MakeHashChain))
+ New.sort(key=lambda x: str(x))
+ MakeHashChain += New
+
+ # Add Library hash
+ if self.LibraryAutoGenList:
+ for Lib in sorted(self.LibraryAutoGenList, key=lambda x: x.Name):
+ if not (Lib.MetaFile.Path, Lib.Arch) in gDict or \
+ not gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashChain:
+ Lib.GenMakeHash(gDict)
+ if not gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashDigest:
+ print("Cannot generate MakeHash for lib module:", Lib.MetaFile.Path, Lib.Arch)
+ continue
+ m.update(gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashDigest)
+ New = list(set(gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashChain) - set(MakeHashChain))
+ New.sort(key=lambda x: str(x))
+ MakeHashChain += New
+
+ # Add Module self
+ m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDigest)
+ New = list(set(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain) - set(MakeHashChain))
+ New.sort(key=lambda x: str(x))
+ MakeHashChain += New
+
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.MakeHashDigest = m.digest()
+ IR.MakeHashHexDigest = m.hexdigest()
+ IR.MakeHashChain = MakeHashChain
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+
+ return gDict[(self.MetaFile.Path, self.Arch)]
+
+ ## Decide whether we can skip the left autogen and make process
+ def CanSkipbyPreMakefileCache(self, gDict):
+ if not GlobalData.gBinCacheSource:
+ return False
+
+ # If Module is binary, do not skip by cache
+ if self.IsBinaryModule:
+ return False
+
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ return False
+
+ # Get the module hash values from stored cache and currrent build
+ # then check whether cache hit based on the hash values
+ # if cache hit, restore all the files from cache
+ FileDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
+ FfsDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs", self.Guid + self.Name)
+
+ ModuleHashPairList = [] # tuple list: [tuple(PreMakefileHash, MakeHash)]
+ ModuleHashPair = path.join(FileDir, self.Name + ".ModuleHashPair")
+ if not os.path.exists(ModuleHashPair):
+ EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair file: %s" % ModuleHashPair)
+ return False
+
+ try:
+ f = open(ModuleHashPair, 'r')
+ ModuleHashPairList = json.load(f)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair file: %s" % ModuleHashPair)
+ return False
+
+ self.GenPreMakefileHash(gDict)
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest:
+ EdkLogger.quiet("[cache warning]: PreMakefileHashHexDigest is missing for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return False
+
+ MakeHashStr = None
+ CurrentPreMakeHash = gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest
+ for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairList):
+ if PreMakefileHash == CurrentPreMakeHash:
+ MakeHashStr = str(MakeHash)
+
+ if not MakeHashStr:
+ return False
+
+ TargetHashDir = path.join(FileDir, MakeHashStr)
+ TargetFfsHashDir = path.join(FfsDir, MakeHashStr)
+
+ if not os.path.exists(TargetHashDir):
+ EdkLogger.quiet("[cache warning]: Cache folder is missing: %s" % TargetHashDir)
+ return False
+
+ for root, dir, files in os.walk(TargetHashDir):
+ for f in files:
+ File = path.join(root, f)
+ self.CacheCopyFile(self.OutputDir, TargetHashDir, File)
+ if os.path.exists(TargetFfsHashDir):
+ for root, dir, files in os.walk(TargetFfsHashDir):
+ for f in files:
+ File = path.join(root, f)
+ self.CacheCopyFile(self.FfsOutputDir, TargetFfsHashDir, File)
+
+ if self.Name == "PcdPeim" or self.Name == "PcdDxe":
+ CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
+
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.PreMakeCacheHit = True
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ print("[cache hit]: checkpoint_PreMakefile:", self.MetaFile.Path, self.Arch)
+ #EdkLogger.quiet("cache hit: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return True
+
+ ## Decide whether we can skip the make process
+ def CanSkipbyMakeCache(self, gDict):
+ if not GlobalData.gBinCacheSource:
+ return False
+
+ # If Module is binary, do not skip by cache
+ if self.IsBinaryModule:
+ print("[cache miss]: checkpoint_Makefile: binary module:", self.MetaFile.Path, self.Arch)
+ return False
+
+ # .inc is contains binary information so do not skip by hash as well
+ for f_ext in self.SourceFileList:
+ if '.inc' in str(f_ext):
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.MakeCacheHit = False
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ print("[cache miss]: checkpoint_Makefile: .inc module:", self.MetaFile.Path, self.Arch)
+ return False
+
+ # Get the module hash values from stored cache and currrent build
+ # then check whether cache hit based on the hash values
+ # if cache hit, restore all the files from cache
+ FileDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
+ FfsDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs", self.Guid + self.Name)
+
+ ModuleHashPairList = [] # tuple list: [tuple(PreMakefileHash, MakeHash)]
+ ModuleHashPair = path.join(FileDir, self.Name + ".ModuleHashPair")
+ if not os.path.exists(ModuleHashPair):
+ EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair file: %s" % ModuleHashPair)
+ return False
+
+ try:
+ f = open(ModuleHashPair, 'r')
+ ModuleHashPairList = json.load(f)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair file: %s" % ModuleHashPair)
+ return False
+
+ self.GenMakeHash(gDict)
+ if not (self.MetaFile.Path, self.Arch) in gDict or \
+ not gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest:
+ EdkLogger.quiet("[cache warning]: MakeHashHexDigest is missing for module %s[%s]" %(self.MetaFile.Path, self.Arch))
+ return False
+
+ MakeHashStr = None
+ CurrentMakeHash = gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest
+ for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairList):
+ if MakeHash == CurrentMakeHash:
+ MakeHashStr = str(MakeHash)
+
+ if not MakeHashStr:
+ print("[cache miss]: checkpoint_Makefile:", self.MetaFile.Path, self.Arch)
+ return False
+
+ TargetHashDir = path.join(FileDir, MakeHashStr)
+ TargetFfsHashDir = path.join(FfsDir, MakeHashStr)
+ if not os.path.exists(TargetHashDir):
+ EdkLogger.quiet("[cache warning]: Cache folder is missing: %s" % TargetHashDir)
+ return False
+
+ for root, dir, files in os.walk(TargetHashDir):
+ for f in files:
+ File = path.join(root, f)
+ self.CacheCopyFile(self.OutputDir, TargetHashDir, File)
+
+ if os.path.exists(TargetFfsHashDir):
+ for root, dir, files in os.walk(TargetFfsHashDir):
+ for f in files:
+ File = path.join(root, f)
+ self.CacheCopyFile(self.FfsOutputDir, TargetFfsHashDir, File)
+
+ if self.Name == "PcdPeim" or self.Name == "PcdDxe":
+ CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
+ with GlobalData.file_lock:
+ IR = gDict[(self.MetaFile.Path, self.Arch)]
+ IR.MakeCacheHit = True
+ gDict[(self.MetaFile.Path, self.Arch)] = IR
+ print("[cache hit]: checkpoint_Makefile:", self.MetaFile.Path, self.Arch)
+ return True
+
## Decide whether we can skip the ModuleAutoGen process
- def CanSkipbyHash(self):
+ def CanSkipbyCache(self, gDict):
# Hashing feature is off
- if not GlobalData.gUseHashCache:
+ if not GlobalData.gBinCacheSource:
return False
- # Initialize a dictionary for each arch type
- if self.Arch not in GlobalData.gBuildHashSkipTracking:
- GlobalData.gBuildHashSkipTracking[self.Arch] = dict()
+ if self in GlobalData.gBuildHashSkipTracking:
+ return GlobalData.gBuildHashSkipTracking[self]
# If library or Module is binary do not skip by hash
if self.IsBinaryModule:
+ GlobalData.gBuildHashSkipTracking[self] = False
return False
# .inc is contains binary information so do not skip by hash as well
for f_ext in self.SourceFileList:
if '.inc' in str(f_ext):
+ GlobalData.gBuildHashSkipTracking[self] = False
return False
- # Use Cache, if exists and if Module has a copy in cache
- if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
+ if not (self.MetaFile.Path, self.Arch) in gDict:
+ return False
+
+ if gDict[(self.MetaFile.Path, self.Arch)].PreMakeCacheHit:
+ GlobalData.gBuildHashSkipTracking[self] = True
return True
- # Early exit for libraries that haven't yet finished building
- HashFile = path.join(self.BuildDir, self.Name + ".hash")
- if self.IsLibrary and not os.path.exists(HashFile):
- return False
+ if gDict[(self.MetaFile.Path, self.Arch)].MakeCacheHit:
+ GlobalData.gBuildHashSkipTracking[self] = True
+ return True
- # Return a Boolean based on if can skip by hash, either from memory or from IO.
- if self.Name not in GlobalData.gBuildHashSkipTracking[self.Arch]:
- # If hashes are the same, SaveFileOnChange() will return False.
- GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] = not SaveFileOnChange(HashFile, self.GenModuleHash(), True)
- return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name]
- else:
- return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name]
+ return False
## Decide whether we can skip the ModuleAutoGen process
# If any source file is newer than the module than we cannot skip
#
def CanSkip(self):
+ # Don't skip if cache feature enabled
+ if GlobalData.gUseHashCache or GlobalData.gBinCacheDest or GlobalData.gBinCacheSource:
+ return False
+
if self.MakeFileDir in GlobalData.gSikpAutoGenCache:
return True
if not os.path.exists(self.TimeStampPath):
diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Source/Python/Common/GlobalData.py
old mode 100644
new mode 100755
index bd45a43728..df10814f04
--- a/BaseTools/Source/Python/Common/GlobalData.py
+++ b/BaseTools/Source/Python/Common/GlobalData.py
@@ -119,3 +119,12 @@ gModuleBuildTracking = dict()
# Top Dict: Key: Arch Type Value: Dictionary
# Second Dict: Key: Module\Library Name Value: True\False
gBuildHashSkipTracking = dict()
+
+# Common dictionary to share module cache intermediate result and state
+gCacheIR = None
+# Common lock for the multiple process AutoGens
+file_lock = None
+# Common dictionary to share platform libraries' constant Pcd
+libConstPcd = None
+# Common dictionary to share platform libraries' reference info
+Refes = None
\ No newline at end of file
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
old mode 100644
new mode 100755
index 4de3f43c27..84540d61f5
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -595,7 +595,7 @@ class BuildTask:
#
def AddDependency(self, Dependency):
for Dep in Dependency:
- if not Dep.BuildObject.IsBinaryModule and not Dep.BuildObject.CanSkipbyHash():
+ if not Dep.BuildObject.IsBinaryModule and not Dep.BuildObject.CanSkipbyCache(GlobalData.gCacheIR):
self.DependencyList.append(BuildTask.New(Dep)) # BuildTask list
## The thread wrapper of LaunchCommand function
@@ -811,7 +811,7 @@ class Build():
self.AutoGenMgr = None
EdkLogger.info("")
os.chdir(self.WorkspaceDir)
- self.share_data = Manager().dict()
+ GlobalData.gCacheIR = Manager().dict()
self.log_q = log_q
def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_data):
try:
@@ -820,6 +820,13 @@ class Build():
feedback_q = mp.Queue()
file_lock = mp.Lock()
error_event = mp.Event()
+ GlobalData.file_lock = file_lock
+ FfsCmd = DataPipe.Get("FfsCommand")
+ if FfsCmd is None:
+ FfsCmd = {}
+ GlobalData.FfsCmd = FfsCmd
+ GlobalData.libConstPcd = DataPipe.Get("LibConstPcd")
+ GlobalData.Refes = DataPipe.Get("REFS")
auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,file_lock,share_data,self.log_q,error_event) for _ in range(self.ThreadNumber)]
self.AutoGenMgr = AutoGenManager(auto_workers,feedback_q,error_event)
self.AutoGenMgr.start()
@@ -827,9 +834,21 @@ class Build():
w.start()
if PcdMaList is not None:
for PcdMa in PcdMaList:
+ if GlobalData.gBinCacheSource and self.Target in [None, "", "all"]:
+ PcdMa.GenModuleFilesHash(share_data)
+ PcdMa.GenPreMakefileHash(share_data)
+ if PcdMa.CanSkipbyPreMakefileCache(share_data):
+ continue
+
PcdMa.CreateCodeFile(False)
PcdMa.CreateMakeFile(False,GenFfsList = DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.File, PcdMa.Arch),[]))
+ if GlobalData.gBinCacheSource and self.Target in [None, "", "all"]:
+ PcdMa.GenMakeHeaderFilesHash(share_data)
+ PcdMa.GenMakeHash(share_data)
+ if PcdMa.CanSkipbyMakeCache(share_data):
+ continue
+
self.AutoGenMgr.join()
rt = self.AutoGenMgr.Status
return rt, 0
@@ -1199,10 +1218,11 @@ class Build():
mqueue.put(m)
AutoGenObject.DataPipe.DataContainer = {"FfsCommand":FfsCommand}
+ AutoGenObject.DataPipe.DataContainer = {"CommandTarget": self.Target}
self.Progress.Start("Generating makefile and code")
data_pipe_file = os.path.join(AutoGenObject.BuildDir, "GlobalVar_%s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch))
AutoGenObject.DataPipe.dump(data_pipe_file)
- autogen_rt, errorcode = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
+ autogen_rt,errorcode = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList, GlobalData.gCacheIR)
self.Progress.Stop("done!")
if not autogen_rt:
self.AutoGenMgr.TerminateWorkers()
@@ -1799,6 +1819,15 @@ class Build():
CmdListDict = None
if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
CmdListDict = self._GenFfsCmd(Wa.ArchList)
+
+ # Add Platform and Package level hash in share_data for module hash calculation later
+ if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest:
+ GlobalData.gCacheIR[('PlatformHash')] = GlobalData.gPlatformHash
+ for PkgName in GlobalData.gPackageHash.keys():
+ GlobalData.gCacheIR[(PkgName, 'PackageHash')] = GlobalData.gPackageHash[PkgName]
+ GlobalData.file_lock = mp.Lock()
+ GlobalData.FfsCmd = CmdListDict
+
self.Progress.Stop("done!")
MaList = []
ExitFlag = threading.Event()
@@ -1808,20 +1837,23 @@ class Build():
AutoGenStart = time.time()
GlobalData.gGlobalDefines['ARCH'] = Arch
Pa = PlatformAutoGen(Wa, self.PlatformFile, BuildTarget, ToolChain, Arch)
+ GlobalData.libConstPcd = Pa.DataPipe.Get("LibConstPcd")
+ GlobalData.Refes = Pa.DataPipe.Get("REFS")
for Module in Pa.Platform.Modules:
if self.ModuleFile.Dir == Module.Dir and self.ModuleFile.Name == Module.Name:
Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile,Pa.DataPipe)
if Ma is None:
continue
MaList.append(Ma)
- if Ma.CanSkipbyHash():
- self.HashSkipModules.append(Ma)
- if GlobalData.gBinCacheSource:
- EdkLogger.quiet("cache hit: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
- continue
- else:
- if GlobalData.gBinCacheSource:
- EdkLogger.quiet("cache miss: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
+
+ if GlobalData.gBinCacheSource and self.Target in [None, "", "all"]:
+ Ma.GenModuleFilesHash(GlobalData.gCacheIR)
+ Ma.GenPreMakefileHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheIR):
+ self.HashSkipModules.append(Ma)
+ EdkLogger.quiet("cache hit: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
+ continue
+
# Not to auto-gen for targets 'clean', 'cleanlib', 'cleanall', 'run', 'fds'
if self.Target not in ['clean', 'cleanlib', 'cleanall', 'run', 'fds']:
# for target which must generate AutoGen code and makefile
@@ -1841,6 +1873,18 @@ class Build():
self.Progress.Stop("done!")
if self.Target == "genmake":
return True
+
+ if GlobalData.gBinCacheSource and self.Target in [None, "", "all"]:
+ Ma.GenMakeHeaderFilesHash(GlobalData.gCacheIR)
+ Ma.GenMakeHash(GlobalData.gCacheIR)
+ if Ma.CanSkipbyMakeCache(GlobalData.gCacheIR):
+ self.HashSkipModules.append(Ma)
+ EdkLogger.quiet("cache hit: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
+ continue
+ else:
+ EdkLogger.quiet("cache miss: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
+ Ma.PrintFirstMakeCacheMissFile(GlobalData.gCacheIR)
+
self.BuildModules.append(Ma)
# Initialize all modules in tracking to 'FAIL'
if Ma.Arch not in GlobalData.gModuleBuildTracking:
@@ -1985,11 +2029,18 @@ class Build():
if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
CmdListDict = self._GenFfsCmd(Wa.ArchList)
+ # Add Platform and Package level hash in share_data for module hash calculation later
+ if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest:
+ GlobalData.gCacheIR[('PlatformHash')] = GlobalData.gPlatformHash
+ for PkgName in GlobalData.gPackageHash.keys():
+ GlobalData.gCacheIR[(PkgName, 'PackageHash')] = GlobalData.gPackageHash[PkgName]
+
# multi-thread exit flag
ExitFlag = threading.Event()
ExitFlag.clear()
self.AutoGenTime += int(round((time.time() - WorkspaceAutoGenTime)))
self.BuildModules = []
+ TotalModules = []
for Arch in Wa.ArchList:
PcdMaList = []
AutoGenStart = time.time()
@@ -2009,6 +2060,7 @@ class Build():
ModuleList.append(Inf)
Pa.DataPipe.DataContainer = {"FfsCommand":CmdListDict}
Pa.DataPipe.DataContainer = {"Workspace_timestamp": Wa._SrcTimeStamp}
+ Pa.DataPipe.DataContainer = {"CommandTarget": self.Target}
for Module in ModuleList:
# Get ModuleAutoGen object to generate C code file and makefile
Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile,Pa.DataPipe)
@@ -2019,19 +2071,7 @@ class Build():
Ma.PlatformInfo = Pa
Ma.Workspace = Wa
PcdMaList.append(Ma)
- if Ma.CanSkipbyHash():
- self.HashSkipModules.append(Ma)
- if GlobalData.gBinCacheSource:
- EdkLogger.quiet("cache hit: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
- continue
- else:
- if GlobalData.gBinCacheSource:
- EdkLogger.quiet("cache miss: %s[%s]" % (Ma.MetaFile.Path, Ma.Arch))
-
- # Not to auto-gen for targets 'clean', 'cleanlib', 'cleanall', 'run', 'fds'
- # for target which must generate AutoGen code and makefile
-
- self.BuildModules.append(Ma)
+ TotalModules.append(Ma)
# Initialize all modules in tracking to 'FAIL'
if Ma.Arch not in GlobalData.gModuleBuildTracking:
GlobalData.gModuleBuildTracking[Ma.Arch] = dict()
@@ -2042,7 +2082,22 @@ class Build():
mqueue.put(m)
data_pipe_file = os.path.join(Pa.BuildDir, "GlobalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch))
Pa.DataPipe.dump(data_pipe_file)
- autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
+ autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList, GlobalData.gCacheIR)
+
+ # Skip cache hit modules
+ if GlobalData.gBinCacheSource:
+ for Ma in TotalModules:
+ if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCacheIR and \
+ GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)].PreMakeCacheHit:
+ self.HashSkipModules.append(Ma)
+ continue
+ if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCacheIR and \
+ GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)].MakeCacheHit:
+ self.HashSkipModules.append(Ma)
+ continue
+ self.BuildModules.append(Ma)
+ else:
+ self.BuildModules.extend(TotalModules)
if not autogen_rt:
self.AutoGenMgr.TerminateWorkers()
@@ -2050,9 +2105,24 @@ class Build():
raise FatalError(errorcode)
self.AutoGenTime += int(round((time.time() - AutoGenStart)))
self.Progress.Stop("done!")
+
+ if GlobalData.gBinCacheSource:
+ EdkLogger.quiet("Total cache hit driver num: %s, cache miss driver num: %s" % (len(set(self.HashSkipModules)), len(set(self.BuildModules))))
+ CacheHitMa = set()
+ CacheNotHitMa = set()
+ for IR in GlobalData.gCacheIR.keys():
+ if 'PlatformHash' in IR or 'PackageHash' in IR:
+ continue
+ if GlobalData.gCacheIR[IR].PreMakeCacheHit or GlobalData.gCacheIR[IR].MakeCacheHit:
+ CacheHitMa.add(IR)
+ else:
+ # There might be binary module or module which has .inc files, not count for cache miss
+ CacheNotHitMa.add(IR)
+ EdkLogger.quiet("Total module num: %s, cache hit module num: %s" % (len(CacheHitMa)+len(CacheNotHitMa), len(CacheHitMa)))
+
for Arch in Wa.ArchList:
MakeStart = time.time()
- for Ma in self.BuildModules:
+ for Ma in set(self.BuildModules):
# Generate build task for the module
if not Ma.IsBinaryModule:
Bt = BuildTask.New(ModuleMakeUnit(Ma, Pa.BuildCommand,self.Target))
--
2.17.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/4] BaseTools: Print first cache missing file for build cachle
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
2019-08-08 12:25 ` [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache Steven Shi
@ 2019-08-08 12:25 ` Steven Shi
2019-08-08 12:25 ` [PATCH 3/4] BaseTools: Change the [Arch][Name] module key in Build cache Steven Shi
` (2 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Steven Shi @ 2019-08-08 12:25 UTC (permalink / raw)
To: devel
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson,
Shi, Steven
From: "Shi, Steven" <steven.shi@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1925
When a module build cache miss, add support to print the first
cache missing file path and name.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 ++
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 78 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
index a84ed46f2e..30d2f96fc7 100755
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -246,6 +246,8 @@ class AutoGenWorkerInProcess(mp.Process):
Ma.GenMakeHash(GlobalData.gCacheIR)
if Ma.CanSkipbyMakeCache(GlobalData.gCacheIR):
continue
+ else:
+ Ma.PrintFirstMakeCacheMissFile(GlobalData.gCacheIR)
except Empty:
pass
except:
diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
index 5749b8a9fa..67875f7532 100755
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -2368,6 +2368,82 @@ class ModuleAutoGen(AutoGen):
print("[cache hit]: checkpoint_Makefile:", self.MetaFile.Path, self.Arch)
return True
+ ## Show the first file name which causes cache miss
+ def PrintFirstMakeCacheMissFile(self, gDict):
+ if not GlobalData.gBinCacheSource:
+ return
+
+ # skip binary module
+ if self.IsBinaryModule:
+ return
+
+ if not (self.MetaFile.Path, self.Arch) in gDict:
+ return
+
+ # Only print cache miss file for the MakeCache not hit module
+ if gDict[(self.MetaFile.Path, self.Arch)].MakeCacheHit:
+ return
+
+ if not gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain:
+ EdkLogger.quiet("[cache insight]: MakeHashChain is missing for: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return
+
+ # Find the cache dir name through the .ModuleHashPair file info
+ FileDir = path.join(GlobalData.gBinCacheSource, self.PlatformInfo.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, self.MetaFile.BaseName)
+
+ ModuleHashPairList = [] # tuple list: [tuple(PreMakefileHash, MakeHash)]
+ ModuleHashPair = path.join(FileDir, self.Name + ".ModuleHashPair")
+ if not os.path.exists(ModuleHashPair):
+ EdkLogger.quiet("[cache insight]: Cannot find ModuleHashPair file for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return
+
+ try:
+ f = open(ModuleHashPair, 'r')
+ ModuleHashPairList = json.load(f)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache insight]: Cannot load ModuleHashPair file for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return
+
+ MakeHashSet = set()
+ for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairList):
+ TargetHashDir = path.join(FileDir, str(MakeHash))
+ if os.path.exists(TargetHashDir):
+ MakeHashSet.add(MakeHash)
+ if not MakeHashSet:
+ EdkLogger.quiet("[cache insight]: Cannot find valid cache dir for module: %s[%s]" % (self.MetaFile.Path, self.Arch))
+ return
+
+ TargetHash = list(MakeHashSet)[0]
+ TargetHashDir = path.join(FileDir, str(TargetHash))
+ if len(MakeHashSet) > 1 :
+ EdkLogger.quiet("[cache insight]: found multiple cache dirs for this module, random select dir '%s' to search the first cache miss file: %s[%s]" % (TargetHash, self.MetaFile.Path, self.Arch))
+
+ ListFile = path.join(TargetHashDir, self.Name + '.MakeHashChain')
+ if os.path.exists(ListFile):
+ try:
+ f = open(ListFile, 'r')
+ CachedList = json.load(f)
+ f.close()
+ except:
+ EdkLogger.quiet("[cache insight]: Cannot load MakeHashChain file: %s" % ListFile)
+ return
+ else:
+ EdkLogger.quiet("[cache insight]: Cannot find MakeHashChain file: %s" % ListFile)
+ return
+
+ CurrentList = gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain
+ for idx, (file, hash) in enumerate (CurrentList):
+ (filecached, hashcached) = CachedList[idx]
+ if file != filecached:
+ EdkLogger.quiet("[cache insight]: first different file in %s[%s] is %s, the cached one is %s" % (self.MetaFile.Path, self.Arch, file, filecached))
+ break
+ if hash != hashcached:
+ EdkLogger.quiet("[cache insight]: first cache miss file in %s[%s] is %s" % (self.MetaFile.Path, self.Arch, file))
+ break
+
+ return True
+
## Decide whether we can skip the ModuleAutoGen process
def CanSkipbyCache(self, gDict):
# Hashing feature is off
--
2.17.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/4] BaseTools: Change the [Arch][Name] module key in Build cache
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
2019-08-08 12:25 ` [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache Steven Shi
2019-08-08 12:25 ` [PATCH 2/4] BaseTools: Print first cache missing file for build cachle Steven Shi
@ 2019-08-08 12:25 ` Steven Shi
2019-08-08 12:25 ` [PATCH 4/4] BaseTools: Add GenFds multi-thread support in build cache Steven Shi
2019-08-08 20:43 ` [edk2-devel] [PATCH 0/4] Build cache enhancement Laszlo Ersek
4 siblings, 0 replies; 12+ messages in thread
From: Steven Shi @ 2019-08-08 12:25 UTC (permalink / raw)
To: devel
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson,
Shi, Steven
From: "Shi, Steven" <steven.shi@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1951
Current build cache use the module's [self.Arch][self.Name]
info as the ModuleAutoGen object key in hash list and dictionary.
The [self.Arch][self.Name] is not safe as the module key because
there could be two modules with same module name and arch name in
one platform. E.g. A platform can override a module or library
instance in another different path, the overriding module can has
the same module name and arch name as the original one.
Directly use the ModuleAutoGen obj self as the key, because
the obj __hash__ and __repr__ attributes already contain the
full path and arch name.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Source/Python/AutoGen/GenMake.py | 6 +-----
BaseTools/Source/Python/build/build.py | 49 +++++++++++++++++++++----------------------------
2 files changed, 22 insertions(+), 33 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 79387856bd..de820eeb2f 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -940,16 +940,12 @@ cleanlib:
continue
headerFileDependencySet.add(aFileName)
- # Ensure that gModuleBuildTracking has been initialized per architecture
- if self._AutoGenObject.Arch not in GlobalData.gModuleBuildTracking:
- GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch] = dict()
-
# Check if a module dependency header file is missing from the module's MetaFile
for aFile in headerFileDependencySet:
if aFile in headerFilesInMetaFileSet:
continue
if GlobalData.gUseHashCache:
- GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch][self._AutoGenObject] = 'FAIL_METAFILE'
+ GlobalData.gModuleBuildTracking[self._AutoGenObject] = 'FAIL_METAFILE'
EdkLogger.warn("build","Module MetaFile [Sources] is missing local header!",
ExtraData = "Local Header: " + aFile + " not found in " + self._AutoGenObject.MetaFile.Path
)
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index 84540d61f5..81f0bbb467 100755
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -630,12 +630,11 @@ class BuildTask:
# Set the value used by hash invalidation flow in GlobalData.gModuleBuildTracking to 'SUCCESS'
# If Module or Lib is being tracked, it did not fail header check test, and built successfully
- if (self.BuildItem.BuildObject.Arch in GlobalData.gModuleBuildTracking and
- self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arch] and
- GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arch][self.BuildItem.BuildObject] != 'FAIL_METAFILE' and
+ if (self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking and
+ GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] != 'FAIL_METAFILE' and
not BuildTask._ErrorFlag.isSet()
):
- GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arch][self.BuildItem.BuildObject] = 'SUCCESS'
+ GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] = 'SUCCESS'
# indicate there's a thread is available for another build task
BuildTask._RunningQueueLock.acquire()
@@ -1169,25 +1168,24 @@ class Build():
return
# GlobalData.gModuleBuildTracking contains only modules or libs that cannot be skipped by hash
- for moduleAutoGenObjArch in GlobalData.gModuleBuildTracking.keys():
- for moduleAutoGenObj in GlobalData.gModuleBuildTracking[moduleAutoGenObjArch].keys():
- # Skip invalidating for Successful Module/Lib builds
- if GlobalData.gModuleBuildTracking[moduleAutoGenObjArch][moduleAutoGenObj] == 'SUCCESS':
- continue
+ for Ma in GlobalData.gModuleBuildTracking:
+ # Skip invalidating for Successful Module/Lib builds
+ if GlobalData.gModuleBuildTracking[Ma] == 'SUCCESS':
+ continue
- # The module failed to build, failed to start building, or failed the header check test from this point on
+ # The module failed to build, failed to start building, or failed the header check test from this point on
- # Remove .hash from build
- ModuleHashFile = os.path.join(moduleAutoGenObj.BuildDir, moduleAutoGenObj.Name + ".hash")
- if os.path.exists(ModuleHashFile):
- os.remove(ModuleHashFile)
+ # Remove .hash from build
+ ModuleHashFile = os.path.join(Ma.BuildDir, Ma.Name + ".hash")
+ if os.path.exists(ModuleHashFile):
+ os.remove(ModuleHashFile)
- # Remove .hash file from cache
- if GlobalData.gBinCacheDest:
- FileDir = os.path.join(GlobalData.gBinCacheDest, moduleAutoGenObj.Arch, moduleAutoGenObj.SourceDir, moduleAutoGenObj.MetaFile.BaseName)
- HashFile = os.path.join(FileDir, moduleAutoGenObj.Name + '.hash')
- if os.path.exists(HashFile):
- os.remove(HashFile)
+ # Remove .hash file from cache
+ if GlobalData.gBinCacheDest:
+ FileDir = os.path.join(GlobalData.gBinCacheDest, Ma.PlatformInfo.OutputDir, Ma.BuildTarget + "_" + Ma.ToolChain, Ma.Arch, Ma.SourceDir, Ma.MetaFile.BaseName)
+ HashFile = os.path.join(FileDir, Ma.Name + '.hash')
+ if os.path.exists(HashFile):
+ os.remove(HashFile)
## Build a module or platform
#
@@ -1887,10 +1885,7 @@ class Build():
self.BuildModules.append(Ma)
# Initialize all modules in tracking to 'FAIL'
- if Ma.Arch not in GlobalData.gModuleBuildTracking:
- GlobalData.gModuleBuildTracking[Ma.Arch] = dict()
- if Ma not in GlobalData.gModuleBuildTracking[Ma.Arch]:
- GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = 'FAIL'
+ GlobalData.gModuleBuildTracking[Ma] = 'FAIL'
self.AutoGenTime += int(round((time.time() - AutoGenStart)))
MakeStart = time.time()
for Ma in self.BuildModules:
@@ -2073,10 +2068,8 @@ class Build():
PcdMaList.append(Ma)
TotalModules.append(Ma)
# Initialize all modules in tracking to 'FAIL'
- if Ma.Arch not in GlobalData.gModuleBuildTracking:
- GlobalData.gModuleBuildTracking[Ma.Arch] = dict()
- if Ma not in GlobalData.gModuleBuildTracking[Ma.Arch]:
- GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = 'FAIL'
+ GlobalData.gModuleBuildTracking[Ma] = 'FAIL'
+
mqueue = mp.Queue()
for m in Pa.GetAllModuleInfo:
mqueue.put(m)
--
2.17.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 4/4] BaseTools: Add GenFds multi-thread support in build cache
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
` (2 preceding siblings ...)
2019-08-08 12:25 ` [PATCH 3/4] BaseTools: Change the [Arch][Name] module key in Build cache Steven Shi
@ 2019-08-08 12:25 ` Steven Shi
2019-08-08 20:43 ` [edk2-devel] [PATCH 0/4] Build cache enhancement Laszlo Ersek
4 siblings, 0 replies; 12+ messages in thread
From: Steven Shi @ 2019-08-08 12:25 UTC (permalink / raw)
To: devel
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson,
Shi, Steven
From: "Shi, Steven" <steven.shi@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1923
Fix the issue that the GenFds multi-thread will build fail
if enable the build cache together.
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
index 67875f7532..e73664f3b0 100755
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -1248,11 +1248,13 @@ class ModuleAutoGen(AutoGen):
fStringIO.close ()
fInputfile.close ()
return OutputName
+
@cached_property
def OutputFile(self):
retVal = set()
OutputDir = self.OutputDir.replace('\\', '/').strip('/')
DebugDir = self.DebugDir.replace('\\', '/').strip('/')
+ FfsOutputDir = self.FfsOutputDir.replace('\\', '/').rstrip('/')
for Item in self.CodaTargetList:
File = Item.Target.Path.replace('\\', '/').strip('/').replace(DebugDir, '').replace(OutputDir, '').strip('/')
retVal.add(File)
@@ -1268,6 +1270,12 @@ class ModuleAutoGen(AutoGen):
if File.lower().endswith('.pdb'):
retVal.add(File)
+ for Root, Dirs, Files in os.walk(FfsOutputDir):
+ for File in Files:
+ if File.lower().endswith('.ffs') or File.lower().endswith('.offset') or File.lower().endswith('.raw') \
+ or File.lower().endswith('.raw.txt'):
+ retVal.add(File)
+
return retVal
## Create AsBuilt INF file the module
@@ -1638,13 +1646,16 @@ class ModuleAutoGen(AutoGen):
for File in self.OutputFile:
File = str(File)
if not os.path.isabs(File):
- File = os.path.join(self.OutputDir, File)
+ NewFile = os.path.join(self.OutputDir, File)
+ if not os.path.exists(NewFile):
+ NewFile = os.path.join(self.FfsOutputDir, File)
+ File = NewFile
if os.path.exists(File):
- sub_dir = os.path.relpath(File, self.OutputDir)
- destination_file = os.path.join(FileDir, sub_dir)
- destination_dir = os.path.dirname(destination_file)
- CreateDirectory(destination_dir)
- CopyFileOnChange(File, destination_dir)
+ if File.lower().endswith('.ffs') or File.lower().endswith('.offset') or File.lower().endswith('.raw') \
+ or File.lower().endswith('.raw.txt'):
+ self.CacheCopyFile(FfsDir, self.FfsOutputDir, File)
+ else:
+ self.CacheCopyFile(FileDir, self.OutputDir, File)
def SaveHashChainFileToCache(self, gDict):
if not GlobalData.gBinCacheDest:
--
2.17.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [PATCH 0/4] Build cache enhancement
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
` (3 preceding siblings ...)
2019-08-08 12:25 ` [PATCH 4/4] BaseTools: Add GenFds multi-thread support in build cache Steven Shi
@ 2019-08-08 20:43 ` Laszlo Ersek
4 siblings, 0 replies; 12+ messages in thread
From: Laszlo Ersek @ 2019-08-08 20:43 UTC (permalink / raw)
To: devel, steven.shi
Cc: liming.gao, bob.c.feng, christian.rodriguez, michael.johnson
On 08/08/19 14:25, Steven Shi wrote:
> From: "Shi, Steven" <steven.shi@intel.com>
>
> Enhance the edk2 build cache with below patches:
> Patch 01/04: Improve the cache hit rate through new cache checkpoint and hash algorithm
> Patch 02/04: Print more info to explain why a module build cache miss
> Patch 03/04: Fix the unsafe [self.Arch][self.Name] key usage in build cache
> Patch 04/04 Add the GenFds multi-thread support in build cache
>
> This patch set is based on patch set of [Patch 00/10 V8] Enable multiple process AutoGen
> https://edk2.groups.io/g/devel/topic/patch_00_10_v8_enable/32779325?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,140,32779325
I've now updated BZ#1927 to depend on BZ#1875.
> You can directly try this patch set in the branch:
> https://github.com/shijunjing/edk2/tree/build_cache_improve_v1
>
>
> Shi, Steven (4):
> BaseTools: Improve the cache hit in the edk2 build cache
> BaseTools: Print first cache missing file for build cachle
> BaseTools: Change the [Arch][Name] module key in Build cache
> BaseTools: Add GenFds multi-thread support in build cache
>
> .../Source/Python/AutoGen/AutoGenWorker.py | 23 +
> BaseTools/Source/Python/AutoGen/CacheIR.py | 28 +
> BaseTools/Source/Python/AutoGen/DataPipe.py | 8 +
> BaseTools/Source/Python/AutoGen/GenMake.py | 229 +++---
> .../Source/Python/AutoGen/ModuleAutoGen.py | 742 ++++++++++++++++--
> BaseTools/Source/Python/Common/GlobalData.py | 9 +
> BaseTools/Source/Python/build/build.py | 171 ++--
> 7 files changed, 979 insertions(+), 231 deletions(-)
> mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
> create mode 100755 BaseTools/Source/Python/AutoGen/CacheIR.py
> mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/DataPipe.py
> mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/GenMake.py
> mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> mode change 100644 => 100755 BaseTools/Source/Python/Common/GlobalData.py
> mode change 100644 => 100755 BaseTools/Source/Python/build/build.py
>
Please change the status of BZ#1927 to IN_PROGRESS, and link this thread
(from the mailing list archive) into a comment in BZ#1927, at the same time.
Thanks
Laszlo
^ permalink raw reply [flat|nested] 12+ messages in thread
* commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-08-08 12:25 ` [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache Steven Shi
@ 2019-09-06 20:45 ` vladimir.olovyannikov
2019-09-07 2:16 ` Steven Shi
0 siblings, 1 reply; 12+ messages in thread
From: vladimir.olovyannikov @ 2019-09-06 20:45 UTC (permalink / raw)
To: devel, steven.shi
Cc: Gao, Liming, Bob Feng, christian.rodriguez, michael.johnson
[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7
(ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first
cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing
[/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io for help, attaching following
call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2493, in Main
MyBuild.Launch()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2287, in Launch
self._BuildPlatform()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa,
FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py",
line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py",
line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py",
line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py",
line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 3288 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-09-06 20:45 ` commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error vladimir.olovyannikov
@ 2019-09-07 2:16 ` Steven Shi
2019-09-09 1:22 ` Bob Feng
0 siblings, 1 reply; 12+ messages in thread
From: Steven Shi @ 2019-09-07 2:16 UTC (permalink / raw)
To: Vladimir Olovyannikov, devel@edk2.groups.io
Cc: Gao, Liming, Feng, Bob C, Rodriguez, Christian, Johnson, Michael
[-- Attachment #1: Type: text/plain, Size: 3417 bytes --]
Hi Vladimir,
Did you enable the build cache? Could you let me know your build steps and commands?
Thanks
Steven
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Saturday, September 7, 2019 4:45 AM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@intel.com>
Cc: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <michael.johnson@intel.com>
Subject: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7 (ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing [/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io<mailto:devel@edk2.groups.io> for help, attaching following call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2493, in Main
MyBuild.Launch()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2287, in Launch
self._BuildPlatform()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa, FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py", line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py", line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 7972 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-09-07 2:16 ` Steven Shi
@ 2019-09-09 1:22 ` Bob Feng
2019-09-09 20:14 ` Vladimir Olovyannikov
0 siblings, 1 reply; 12+ messages in thread
From: Bob Feng @ 2019-09-09 1:22 UTC (permalink / raw)
To: Shi, Steven, Vladimir Olovyannikov, devel@edk2.groups.io
Cc: Gao, Liming, Rodriguez, Christian, Johnson, Michael
[-- Attachment #1: Type: text/plain, Size: 4385 bytes --]
Hi Vladimir and Steven,
I checked code and think when the build target are 'clean', 'cleanlib', 'cleanall', 'run' and 'fds', build will fail with the error like you mentioned.
I filed a Bugzilla https://bugzilla.tianocore.org/show_bug.cgi?id=2166
And provide a fix in https://edk2.groups.io/g/devel/message/47014
Thanks,
Bob
From: Shi, Steven
Sent: Saturday, September 7, 2019 10:16 AM
To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>; devel@edk2.groups.io
Cc: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <michael.johnson@intel.com>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Vladimir,
Did you enable the build cache? Could you let me know your build steps and commands?
Thanks
Steven
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Saturday, September 7, 2019 4:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7 (ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing [/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io<mailto:devel@edk2.groups.io> for help, attaching following call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2493, in Main
MyBuild.Launch()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2287, in Launch
self._BuildPlatform()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa, FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py", line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py", line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 11017 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-09-09 1:22 ` Bob Feng
@ 2019-09-09 20:14 ` Vladimir Olovyannikov
2019-09-10 1:25 ` Steven Shi
2019-09-10 1:57 ` Bob Feng
0 siblings, 2 replies; 12+ messages in thread
From: Vladimir Olovyannikov @ 2019-09-09 20:14 UTC (permalink / raw)
To: Feng, Bob C, Shi, Steven, devel
Cc: Gao, Liming, Rodriguez, Christian, Johnson, Michael
[-- Attachment #1: Type: text/plain, Size: 5210 bytes --]
Hi Bob, Steven,
Thank you for update.
Yes, it fails with “build clean”.
Example:
. ./edksetup.sh
build -a AARCH64 -p BroadcomPlatformPkg/StingrayPkg/StingrayPkg.dsc -t GCC5
-b RELEASE -D PLAT=Stingray -D TARGET=RELEASE -D EDK2_STAMP="dev/bcm958742k
59b754c9f697" cleanall
Bob, with your patch it builds fine.
Steven, in case I am missing something, how do you enable the build cache?
Thank you,
Vladimir
*From:* Feng, Bob C [mailto:bob.c.feng@intel.com]
*Sent:* Sunday, September 8, 2019 6:23 PM
*To:* Shi, Steven <steven.shi@intel.com>; Vladimir Olovyannikov <
vladimir.olovyannikov@broadcom.com>; devel@edk2.groups.io
*Cc:* Gao, Liming <liming.gao@intel.com>; Rodriguez, Christian <
christian.rodriguez@intel.com>; Johnson, Michael <michael.johnson@intel.com>
*Subject:* RE: commit "BaseTools: Improve the cache hit in the edk2 build
cache" causes build.py unknown fatal error
Hi Vladimir and Steven,
I checked code and think when the build target are 'clean', 'cleanlib',
'cleanall', 'run' and 'fds', build will fail with the error like you
mentioned.
I filed a Bugzilla https://bugzilla.tianocore.org/show_bug.cgi?id=2166
And provide a fix in https://edk2.groups.io/g/devel/message/47014
Thanks,
Bob
*From:* Shi, Steven
*Sent:* Saturday, September 7, 2019 10:16 AM
*To:* Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com>;
devel@edk2.groups.io
*Cc:* Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>;
Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <
michael.johnson@intel.com>
*Subject:* RE: commit "BaseTools: Improve the cache hit in the edk2 build
cache" causes build.py unknown fatal error
Hi Vladimir,
Did you enable the build cache? Could you let me know your build steps and
commands?
Thanks
Steven
*From:* Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com
<vladimir.olovyannikov@broadcom.com>]
*Sent:* Saturday, September 7, 2019 4:45 AM
*To:* devel@edk2.groups.io; Shi, Steven <steven.shi@intel.com>
*Cc:* Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>;
Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <
michael.johnson@intel.com>
*Subject:* commit "BaseTools: Improve the cache hit in the edk2 build
cache" causes build.py unknown fatal error
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7
(ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first
cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing
[/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io for help, attaching following
call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2493, in Main
MyBuild.Launch()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2287, in Launch
self._BuildPlatform()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa,
FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py",
line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py",
line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py",
line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File
"/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py",
line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 13041 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-09-09 20:14 ` Vladimir Olovyannikov
@ 2019-09-10 1:25 ` Steven Shi
2019-09-10 1:57 ` Bob Feng
1 sibling, 0 replies; 12+ messages in thread
From: Steven Shi @ 2019-09-10 1:25 UTC (permalink / raw)
To: Vladimir Olovyannikov, Feng, Bob C, devel@edk2.groups.io
Cc: Gao, Liming, Rodriguez, Christian, Johnson, Michael
[-- Attachment #1: Type: text/plain, Size: 6768 bytes --]
Hi Vladimir,
Below is an example to do clean build with the cache. The build cache is enabled through --hash --binary-destination and --binary-source options.
· Produce the build cache to a folder (e.g. BinCache):
$ build -p OvmfPkg/OvmfPkgIa32.dsc -a IA32 -t GCC5 -b DEBUG --hash --binary-destination=BinCache
· Remove the Build folder
$ rm Build/ -r –f
· Consume the build cache:
$ build -p OvmfPkg/OvmfPkgIa32.dsc -a IA32 -t GCC5 -b DEBUG --hash --binary-source=BinCache
Thanks
Steven Shi
Intel\SSG\FID\Firmware Infrastructure
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Tuesday, September 10, 2019 4:15 AM
To: Feng, Bob C <bob.c.feng@intel.com>; Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io
Cc: Gao, Liming <liming.gao@intel.com>; Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <michael.johnson@intel.com>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Bob, Steven,
Thank you for update.
Yes, it fails with “build clean”.
Example:
. ./edksetup.sh
build -a AARCH64 -p BroadcomPlatformPkg/StingrayPkg/StingrayPkg.dsc -t GCC5 -b RELEASE -D PLAT=Stingray -D TARGET=RELEASE -D EDK2_STAMP="dev/bcm958742k 59b754c9f697" cleanall
Bob, with your patch it builds fine.
Steven, in case I am missing something, how do you enable the build cache?
Thank you,
Vladimir
From: Feng, Bob C [mailto:bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>]
Sent: Sunday, September 8, 2019 6:23 PM
To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com<mailto:vladimir.olovyannikov@broadcom.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Vladimir and Steven,
I checked code and think when the build target are 'clean', 'cleanlib', 'cleanall', 'run' and 'fds', build will fail with the error like you mentioned.
I filed a Bugzilla https://bugzilla.tianocore.org/show_bug.cgi?id=2166
And provide a fix in https://edk2.groups.io/g/devel/message/47014
Thanks,
Bob
From: Shi, Steven
Sent: Saturday, September 7, 2019 10:16 AM
To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com<mailto:vladimir.olovyannikov@broadcom.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Vladimir,
Did you enable the build cache? Could you let me know your build steps and commands?
Thanks
Steven
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Saturday, September 7, 2019 4:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7 (ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing [/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io<mailto:devel@edk2.groups.io> for help, attaching following call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2493, in Main
MyBuild.Launch()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2287, in Launch
self._BuildPlatform()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa, FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py", line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py", line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 21837 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
2019-09-09 20:14 ` Vladimir Olovyannikov
2019-09-10 1:25 ` Steven Shi
@ 2019-09-10 1:57 ` Bob Feng
1 sibling, 0 replies; 12+ messages in thread
From: Bob Feng @ 2019-09-10 1:57 UTC (permalink / raw)
To: Vladimir Olovyannikov, Shi, Steven, devel@edk2.groups.io
Cc: Gao, Liming, Rodriguez, Christian, Johnson, Michael
[-- Attachment #1: Type: text/plain, Size: 6252 bytes --]
Pushed at 53e2eaba7a19cda3183b1bb2a2619a2494538e52
Thanks,
Bob
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Tuesday, September 10, 2019 4:15 AM
To: Feng, Bob C <bob.c.feng@intel.com>; Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io
Cc: Gao, Liming <liming.gao@intel.com>; Rodriguez, Christian <christian.rodriguez@intel.com>; Johnson, Michael <michael.johnson@intel.com>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Bob, Steven,
Thank you for update.
Yes, it fails with “build clean”.
Example:
. ./edksetup.sh
build -a AARCH64 -p BroadcomPlatformPkg/StingrayPkg/StingrayPkg.dsc -t GCC5 -b RELEASE -D PLAT=Stingray -D TARGET=RELEASE -D EDK2_STAMP="dev/bcm958742k 59b754c9f697" cleanall
Bob, with your patch it builds fine.
Steven, in case I am missing something, how do you enable the build cache?
Thank you,
Vladimir
From: Feng, Bob C [mailto:bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>]
Sent: Sunday, September 8, 2019 6:23 PM
To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com<mailto:vladimir.olovyannikov@broadcom.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Vladimir and Steven,
I checked code and think when the build target are 'clean', 'cleanlib', 'cleanall', 'run' and 'fds', build will fail with the error like you mentioned.
I filed a Bugzilla https://bugzilla.tianocore.org/show_bug.cgi?id=2166
And provide a fix in https://edk2.groups.io/g/devel/message/47014
Thanks,
Bob
From: Shi, Steven
Sent: Saturday, September 7, 2019 10:16 AM
To: Vladimir Olovyannikov <vladimir.olovyannikov@broadcom.com<mailto:vladimir.olovyannikov@broadcom.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: RE: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Vladimir,
Did you enable the build cache? Could you let me know your build steps and commands?
Thanks
Steven
From: Vladimir Olovyannikov [mailto:vladimir.olovyannikov@broadcom.com]
Sent: Saturday, September 7, 2019 4:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Cc: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Rodriguez, Christian <christian.rodriguez@intel.com<mailto:christian.rodriguez@intel.com>>; Johnson, Michael <michael.johnson@intel.com<mailto:michael.johnson@intel.com>>
Subject: commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error
Hi Steven,
The commit
0e7e7a264cd80ab71ea0f9e9da2d0617d4b539c4
BaseTools: Improve the cache hit in the edk2 build cache
Author: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> 2019-08-15 07:26:17
Committer: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> 2019-08-19 23:06:05
Parent: 811328172841ce3f7ec1fe7033c130726ad189b7 (ShellPkg/UefiShellDriver1CommandsLib: Make array big enough)
Child: 56c786b0a9fd4f08b1f0bc10b7718a7577cb0146 (BaseTools: Print first cache missing file for build cachle)
Branches: master, remotes/origin/master
Follows: edk2-stable201905
Precedes: edk2-stable201908
causes "unknown fatal error" in build.py when building our platform.
Here is the log:
...
build.py...
: error C0DE: Unknown fatal error when processing [/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/ArmPkg/Library/ArmGicArchSecLib/ArmGicArchSecLib.inf]
(Please send email to devel@edk2.groups.io<mailto:devel@edk2.groups.io> for help, attaching following call stack trace!)
(Python 3.6.8 on linux) Traceback (most recent call last):
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2493, in Main
MyBuild.Launch()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2287, in Launch
self._BuildPlatform()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1742, in _BuildPlatform
self._BuildPa(self.Target, Pa, FfsCommand=CmdListDict,PcdMaList=PcdMaList)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1236, in _BuildPa
AutoGenObject.CreateCodeFile(True)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/Common/caching.py", line 18, in CallMeHere
Value = self._function(obj, *args,**kwargs)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py", line 171, in CreateCodeFile
Ma.CreateCodeFile(CreateModuleCodeFile)
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1841, in CreateCodeFile
LibraryAutoGen.CreateCodeFile()
File "/home/volovyan/otherwork/sandbox/uefi-related/uefi-orig/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py", line 1886, in CreateCodeFile
with GlobalData.cache_lock:
AttributeError: __enter__
- Failed -
Build end time: 13:31:30, Sep.06 2019
Build total time: 00:00:04
Builds before the above mentioned commit was pushed, were fine.
Thank you,
Vladimir
[-- Attachment #2: Type: text/html, Size: 16391 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-09-10 1:57 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-08-08 12:25 [PATCH 0/4] Build cache enhancement Steven Shi
2019-08-08 12:25 ` [PATCH 1/4] BaseTools: Improve the cache hit in the edk2 build cache Steven Shi
2019-09-06 20:45 ` commit "BaseTools: Improve the cache hit in the edk2 build cache" causes build.py unknown fatal error vladimir.olovyannikov
2019-09-07 2:16 ` Steven Shi
2019-09-09 1:22 ` Bob Feng
2019-09-09 20:14 ` Vladimir Olovyannikov
2019-09-10 1:25 ` Steven Shi
2019-09-10 1:57 ` Bob Feng
2019-08-08 12:25 ` [PATCH 2/4] BaseTools: Print first cache missing file for build cachle Steven Shi
2019-08-08 12:25 ` [PATCH 3/4] BaseTools: Change the [Arch][Name] module key in Build cache Steven Shi
2019-08-08 12:25 ` [PATCH 4/4] BaseTools: Add GenFds multi-thread support in build cache Steven Shi
2019-08-08 20:43 ` [edk2-devel] [PATCH 0/4] Build cache enhancement Laszlo Ersek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox