* [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
@ 2016-11-18 2:29 Derek Lin
2016-11-18 5:17 ` Zhu, Yonghong
0 siblings, 1 reply; 4+ messages in thread
From: Derek Lin @ 2016-11-18 2:29 UTC (permalink / raw)
To: edk2-devel; +Cc: derek.lin2, yonghong.zhu, liming.gao
[Introduction]
The BaseTool Build.py AutoGen parse INF meta-file and generate
AutoGen.c/AutoGen.h/makefile. When we only change .c .h code, the
AutoGen might be not necessary, but Build.py spend a lot of time on it.
There's a -u flag to skip all module's AutoGen. In my environment, it save
35%~50% of time in rebuild a ROM.
However, if user change one .INF meta-file, then -u flag is not available.
[Idea]
AutoGen can compare meta-file's timestamp and decide if the module's AutoGen
can be skipped. With this, when a module's INF is changed, we only run this
module's AutoGen, we don't need to run other module's.
[Implementation]
In the end of a module's AutoGen, we create a AutoGenTimeStamp.
The file save a file list that related to this module's AutoGen.
In other word, the file list in AutoGenTimeStamp is INPUT files of
module AutoGen, AutoGenTimeStamp file is OUTPUT.
During rebuild, we compare time stamp between INPUT and OUTPUT, and
decide if we can skip it.
Below is the Input/Output of a module's AutoGen.
[Input]
1. All the DSC/DEC/FDF used by the platform.
2. INF file of a module.
3. Source files of a module, list in [Sources] section of INF.
4. All the library link by the module.
5. All the .h files included by the module's sources.
[Output]
AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp
[Testing]
This patch save my build time. When I make a change without touching
DSC/DEC/FDF, it is absolutely much faster than original rebuild,
35%~50% time saving in my environment
(compare to original tool rebuild time).
If I change any DSC/DEC/FDF, there's no performance improve, because it
can't skip any module's AutoGen.
Please note that if your environment will generate DSC/FDF during prebuild,
it will not skip any AutoGen because of DSC timestamp is changed. This will
require prebuild script not to update meta file when content is not changed.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++++++
BaseTools/Source/Python/AutoGen/GenMake.py | 3 +
BaseTools/Source/Python/GenFds/FdfParser.py | 4 +
.../Source/Python/Workspace/MetaFileParser.py | 4 +
4 files changed, 121 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index f35ae25..daa5b35 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode
from Workspace.MetaFileCommentParser import UsageList
from Common.MultipleWorkspace import MultipleWorkspace as mws
import InfSectionParser
+import datetime
## Regular expression for splitting Dependency Expression string into tokens
gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)")
@@ -640,6 +641,31 @@ class WorkspaceAutoGen(AutoGen):
self._MakeFileDir = None
self._BuildCommand = None
+ #
+ # Create list of metafiles of the workspace
+ #
+ AllWorkSpaceMetaFiles = self._CreateMetaFileList()
+
+ #
+ # Retrieve latest modified time of all metafiles
+ #
+ SrcTimeStamp = 0
+ for f in AllWorkSpaceMetaFiles:
+ if os.stat(f)[8] > SrcTimeStamp:
+ SrcTimeStamp = os.stat(f)[8]
+ self._SrcTimeStamp = SrcTimeStamp
+
+ #
+ # Write metafile list to build directory
+ #
+ AutoGenFilePath = os.path.join(self.BuildDir, 'AutoGen')
+ if os.path.exists (AutoGenFilePath):
+ os.remove(AutoGenFilePath)
+ if not os.path.exists(self.BuildDir):
+ os.makedirs(self.BuildDir)
+ with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file:
+ for f in AllWorkSpaceMetaFiles:
+ print >> file, f
return True
def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, PcdDatumType, Value):
@@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen):
Value = '0'
return Value
+ def _CreateMetaFileList(self):
+ AllWorkSpaceMetaFiles = []
+ #
+ # add fdf
+ #
+ AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
+ if self.FdfFile:
+ FdfFiles = GlobalData.gFdfParser.GetAllIncludedFile()
+ for f in FdfFiles:
+ AllWorkSpaceMetaFiles.append (f.FileName)
+ #
+ # add dsc & dec
+ #
+ for BuildData in self.BuildDatabase._CACHE_.values():
+ if 'dsc' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+ for filePath in BuildData._RawData.IncludedFiles:
+ AllWorkSpaceMetaFiles.append(filePath.Path)
+ if 'dec' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+
+ return AllWorkSpaceMetaFiles
+
## _CheckDuplicateInFV() method
#
# Check whether there is duplicate modules/files exist in FV section.
@@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen):
# to the [depex] section in module's inf file.
#
class ModuleAutoGen(AutoGen):
+ ## Cache the timestamps of metafiles of every module in a class variable
+ #
+ TimeDict = {}
+
## The real constructor of ModuleAutoGen
#
# This method is not supposed to be called by users of ModuleAutoGen. It's
@@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen):
self._FinalBuildTargetList = None
self._FileTypes = None
self._BuildRules = None
+
+ self._TimeStampPath = None
+
+ self.AutoGenDepSet = set()
+
## The Modules referenced to this Library
# Only Library has this attribute
@@ -3946,6 +4004,8 @@ class ModuleAutoGen(AutoGen):
if self.IsMakeFileCreated:
return
+ if self.CanSkip():
+ return
if not self.IsLibrary and CreateLibraryMakeFile:
for LibraryAutoGen in self.LibraryAutoGenList:
@@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen):
EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of makefile for module %s [%s]" %
(self.Name, self.Arch))
+ self.CreateTimeStamp(Makefile)
self.IsMakeFileCreated = True
def CopyBinaryFiles(self):
@@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen):
def CreateCodeFile(self, CreateLibraryCodeFile=True):
if self.IsCodeFileCreated:
return
+ if self.CanSkip():
+ return
# Need to generate PcdDatabase even PcdDriver is binarymodule
if self.IsBinaryModule and self.PcdIsDriver != '':
@@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen):
self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)
return self._LibraryAutoGenList
+ ## Decide whether we can skip the ModuleAutoGen process
+ # If any source file is newer than the modeule than we cannot skip
+ #
+ def CanSkip(self):
+ if not os.path.exists(self.GetTimeStampPath()):
+ return False
+ #last creation time of the module
+ DstTimeStamp = os.stat(self.GetTimeStampPath())[8]
+
+ SrcTimeStamp = self.Workspace._SrcTimeStamp
+ if SrcTimeStamp > DstTimeStamp:
+ return False
+
+ with open(self.GetTimeStampPath(),'r') as f:
+ for source in f:
+ source = source.rstrip('\n')
+ if source not in ModuleAutoGen.TimeDict :
+ ModuleAutoGen.TimeDict[source] = os.stat(source)[8]
+ if ModuleAutoGen.TimeDict[source] > DstTimeStamp:
+ return False
+ return True
+
+ def GetTimeStampPath(self):
+ if self._TimeStampPath == None:
+ self._TimeStampPath = os.path.join(self.MakeFileDir, 'AutoGenTimeStamp')
+ return self._TimeStampPath
+ def CreateTimeStamp(self, Makefile):
+
+ FileSet = set()
+
+ FileSet.add (self.MetaFile.Path)
+
+ for SourceFile in self.Module.Sources:
+ FileSet.add (SourceFile.Path)
+
+ for Lib in self.DependentLibraryList:
+ FileSet.add (Lib.MetaFile.Path)
+
+ for f in self.AutoGenDepSet:
+ FileSet.add (f.Path)
+
+ if os.path.exists (self.GetTimeStampPath()):
+ os.remove (self.GetTimeStampPath())
+ with open(self.GetTimeStampPath(), 'w+') as file:
+ for f in FileSet:
+ print >> file, f
+
Module = property(_GetModule)
Name = property(_GetBaseName)
Guid = property(_GetGuid)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 51c5238..ea07b97 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -801,6 +801,9 @@ cleanlib:
if not self.FileDependency[File]:
self.FileDependency[File] = ['$(FORCE_REBUILD)']
continue
+
+ self._AutoGenObject.AutoGenDepSet |= set(self.FileDependency[File])
+
# skip non-C files
if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":
continue
diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 2900283..1093ebb 100644
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -4781,6 +4781,10 @@ class FdfParser:
return False
+ def GetAllIncludedFile (self):
+ global AllIncludeFileList
+ return AllIncludeFileList
+
if __name__ == "__main__":
import sys
try:
diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index 1a5fdf5..d0ab5e9 100644
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -852,6 +852,8 @@ class DscParser(MetaFileParser):
SymbolPattern = ValueExpression.SymbolPattern
+ IncludedFiles = set()
+
## Constructor of DscParser
#
# Initialize object of DscParser
@@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser):
Parser = DscParser(IncludedFile1, self._FileType, self._Arch, IncludedFileTable,
Owner=Owner, From=Owner)
+ self.IncludedFiles.add (IncludedFile1)
+
# Does not allow lower level included file to include upper level included file
if Parser._From != Owner and int(Owner) > int (Parser._From):
EdkLogger.error('parser', FILE_ALREADY_EXIST, File=self._FileWithError,
--
2.7.4.windows.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
2016-11-18 2:29 [PATCH] BaseTools: Skip module AutoGen by comparing timestamp Derek Lin
@ 2016-11-18 5:17 ` Zhu, Yonghong
2016-11-19 13:43 ` Zhu, Yonghong
0 siblings, 1 reply; 4+ messages in thread
From: Zhu, Yonghong @ 2016-11-18 5:17 UTC (permalink / raw)
To: Derek Lin, edk2-devel@lists.01.org; +Cc: Gao, Liming, Zhu, Yonghong
Thanks for your contribution. I will review it and do some verification. May give comment later.
Best Regards,
Zhu Yonghong
-----Original Message-----
From: Derek Lin [mailto:derek.lin2@hpe.com]
Sent: Friday, November 18, 2016 10:30 AM
To: edk2-devel@lists.01.org
Cc: derek.lin2@hpe.com; Zhu, Yonghong <yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>
Subject: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
[Introduction]
The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/AutoGen.h/makefile. When we only change .c .h code, the AutoGen might be not necessary, but Build.py spend a lot of time on it.
There's a -u flag to skip all module's AutoGen. In my environment, it save 35%~50% of time in rebuild a ROM.
However, if user change one .INF meta-file, then -u flag is not available.
[Idea]
AutoGen can compare meta-file's timestamp and decide if the module's AutoGen can be skipped. With this, when a module's INF is changed, we only run this module's AutoGen, we don't need to run other module's.
[Implementation]
In the end of a module's AutoGen, we create a AutoGenTimeStamp.
The file save a file list that related to this module's AutoGen.
In other word, the file list in AutoGenTimeStamp is INPUT files of module AutoGen, AutoGenTimeStamp file is OUTPUT.
During rebuild, we compare time stamp between INPUT and OUTPUT, and decide if we can skip it.
Below is the Input/Output of a module's AutoGen.
[Input]
1. All the DSC/DEC/FDF used by the platform.
2. INF file of a module.
3. Source files of a module, list in [Sources] section of INF.
4. All the library link by the module.
5. All the .h files included by the module's sources.
[Output]
AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp
[Testing]
This patch save my build time. When I make a change without touching DSC/DEC/FDF, it is absolutely much faster than original rebuild, 35%~50% time saving in my environment (compare to original tool rebuild time).
If I change any DSC/DEC/FDF, there's no performance improve, because it can't skip any module's AutoGen.
Please note that if your environment will generate DSC/FDF during prebuild, it will not skip any AutoGen because of DSC timestamp is changed. This will require prebuild script not to update meta file when content is not changed.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++++++
BaseTools/Source/Python/AutoGen/GenMake.py | 3 +
BaseTools/Source/Python/GenFds/FdfParser.py | 4 +
.../Source/Python/Workspace/MetaFileParser.py | 4 +
4 files changed, 121 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index f35ae25..daa5b35 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspace.MetaFileCommentParser import UsageList from Common.MultipleWorkspace import MultipleWorkspace as mws import InfSectionParser
+import datetime
## Regular expression for splitting Dependency Expression string into tokens gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +641,31 @@ class WorkspaceAutoGen(AutoGen):
self._MakeFileDir = None
self._BuildCommand = None
+ #
+ # Create list of metafiles of the workspace
+ #
+ AllWorkSpaceMetaFiles = self._CreateMetaFileList()
+
+ #
+ # Retrieve latest modified time of all metafiles
+ #
+ SrcTimeStamp = 0
+ for f in AllWorkSpaceMetaFiles:
+ if os.stat(f)[8] > SrcTimeStamp:
+ SrcTimeStamp = os.stat(f)[8]
+ self._SrcTimeStamp = SrcTimeStamp
+
+ #
+ # Write metafile list to build directory
+ #
+ AutoGenFilePath = os.path.join(self.BuildDir, 'AutoGen')
+ if os.path.exists (AutoGenFilePath):
+ os.remove(AutoGenFilePath)
+ if not os.path.exists(self.BuildDir):
+ os.makedirs(self.BuildDir)
+ with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file:
+ for f in AllWorkSpaceMetaFiles:
+ print >> file, f
return True
def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, PcdDatumType, Value):
@@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen):
Value = '0'
return Value
+ def _CreateMetaFileList(self):
+ AllWorkSpaceMetaFiles = []
+ #
+ # add fdf
+ #
+ AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
+ if self.FdfFile:
+ FdfFiles = GlobalData.gFdfParser.GetAllIncludedFile()
+ for f in FdfFiles:
+ AllWorkSpaceMetaFiles.append (f.FileName)
+ #
+ # add dsc & dec
+ #
+ for BuildData in self.BuildDatabase._CACHE_.values():
+ if 'dsc' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+ for filePath in BuildData._RawData.IncludedFiles:
+ AllWorkSpaceMetaFiles.append(filePath.Path)
+ if 'dec' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+
+ return AllWorkSpaceMetaFiles
+
## _CheckDuplicateInFV() method
#
# Check whether there is duplicate modules/files exist in FV section.
@@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen):
# to the [depex] section in module's inf file.
#
class ModuleAutoGen(AutoGen):
+ ## Cache the timestamps of metafiles of every module in a class variable
+ #
+ TimeDict = {}
+
## The real constructor of ModuleAutoGen
#
# This method is not supposed to be called by users of ModuleAutoGen. It's @@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen):
self._FinalBuildTargetList = None
self._FileTypes = None
self._BuildRules = None
+
+ self._TimeStampPath = None
+
+ self.AutoGenDepSet = set()
+
## The Modules referenced to this Library
# Only Library has this attribute @@ -3946,6 +4004,8 @@ class ModuleAutoGen(AutoGen):
if self.IsMakeFileCreated:
return
+ if self.CanSkip():
+ return
if not self.IsLibrary and CreateLibraryMakeFile:
for LibraryAutoGen in self.LibraryAutoGenList:
@@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen):
EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of makefile for module %s [%s]" %
(self.Name, self.Arch))
+ self.CreateTimeStamp(Makefile)
self.IsMakeFileCreated = True
def CopyBinaryFiles(self):
@@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen):
def CreateCodeFile(self, CreateLibraryCodeFile=True):
if self.IsCodeFileCreated:
return
+ if self.CanSkip():
+ return
# Need to generate PcdDatabase even PcdDriver is binarymodule
if self.IsBinaryModule and self.PcdIsDriver != '':
@@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen):
self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)
return self._LibraryAutoGenList
+ ## Decide whether we can skip the ModuleAutoGen process
+ # If any source file is newer than the modeule than we cannot skip
+ #
+ def CanSkip(self):
+ if not os.path.exists(self.GetTimeStampPath()):
+ return False
+ #last creation time of the module
+ DstTimeStamp = os.stat(self.GetTimeStampPath())[8]
+
+ SrcTimeStamp = self.Workspace._SrcTimeStamp
+ if SrcTimeStamp > DstTimeStamp:
+ return False
+
+ with open(self.GetTimeStampPath(),'r') as f:
+ for source in f:
+ source = source.rstrip('\n')
+ if source not in ModuleAutoGen.TimeDict :
+ ModuleAutoGen.TimeDict[source] = os.stat(source)[8]
+ if ModuleAutoGen.TimeDict[source] > DstTimeStamp:
+ return False
+ return True
+
+ def GetTimeStampPath(self):
+ if self._TimeStampPath == None:
+ self._TimeStampPath = os.path.join(self.MakeFileDir, 'AutoGenTimeStamp')
+ return self._TimeStampPath
+ def CreateTimeStamp(self, Makefile):
+
+ FileSet = set()
+
+ FileSet.add (self.MetaFile.Path)
+
+ for SourceFile in self.Module.Sources:
+ FileSet.add (SourceFile.Path)
+
+ for Lib in self.DependentLibraryList:
+ FileSet.add (Lib.MetaFile.Path)
+
+ for f in self.AutoGenDepSet:
+ FileSet.add (f.Path)
+
+ if os.path.exists (self.GetTimeStampPath()):
+ os.remove (self.GetTimeStampPath())
+ with open(self.GetTimeStampPath(), 'w+') as file:
+ for f in FileSet:
+ print >> file, f
+
Module = property(_GetModule)
Name = property(_GetBaseName)
Guid = property(_GetGuid)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 51c5238..ea07b97 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -801,6 +801,9 @@ cleanlib:
if not self.FileDependency[File]:
self.FileDependency[File] = ['$(FORCE_REBUILD)']
continue
+
+ self._AutoGenObject.AutoGenDepSet |=
+ set(self.FileDependency[File])
+
# skip non-C files
if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":
continue
diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 2900283..1093ebb 100644
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -4781,6 +4781,10 @@ class FdfParser:
return False
+ def GetAllIncludedFile (self):
+ global AllIncludeFileList
+ return AllIncludeFileList
+
if __name__ == "__main__":
import sys
try:
diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index 1a5fdf5..d0ab5e9 100644
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -852,6 +852,8 @@ class DscParser(MetaFileParser):
SymbolPattern = ValueExpression.SymbolPattern
+ IncludedFiles = set()
+
## Constructor of DscParser
#
# Initialize object of DscParser
@@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser):
Parser = DscParser(IncludedFile1, self._FileType, self._Arch, IncludedFileTable,
Owner=Owner, From=Owner)
+ self.IncludedFiles.add (IncludedFile1)
+
# Does not allow lower level included file to include upper level included file
if Parser._From != Owner and int(Owner) > int (Parser._From):
EdkLogger.error('parser', FILE_ALREADY_EXIST, File=self._FileWithError,
--
2.7.4.windows.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
2016-11-18 5:17 ` Zhu, Yonghong
@ 2016-11-19 13:43 ` Zhu, Yonghong
2016-11-21 6:17 ` Lin, Derek (HPS UEFI Dev)
0 siblings, 1 reply; 4+ messages in thread
From: Zhu, Yonghong @ 2016-11-19 13:43 UTC (permalink / raw)
To: 'Derek Lin', 'edk2-devel@lists.01.org'
Cc: Gao, Liming, Zhu, Yonghong
Hi Derek Lin,
Thanks for your contribution. I tried this patch and have few comments.
1. build MdeModulePkg failure. Please help to fix. Thanks.
File "D:\Git_Latest\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line 702, in _CreateMetaFileList
AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
AttributeError: 'str' object has no attribute 'Path'
2. Please help to use the PatchCheck.py under BaseTools\Scripts to fix some coding style issues. Thanks.
Best Regards,
Zhu Yonghong
-----Original Message-----
From: Zhu, Yonghong
Sent: Friday, November 18, 2016 1:18 PM
To: Derek Lin <derek.lin2@hpe.com>; edk2-devel@lists.01.org
Cc: Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
Thanks for your contribution. I will review it and do some verification. May give comment later.
Best Regards,
Zhu Yonghong
-----Original Message-----
From: Derek Lin [mailto:derek.lin2@hpe.com]
Sent: Friday, November 18, 2016 10:30 AM
To: edk2-devel@lists.01.org
Cc: derek.lin2@hpe.com; Zhu, Yonghong <yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>
Subject: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
[Introduction]
The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/AutoGen.h/makefile. When we only change .c .h code, the AutoGen might be not necessary, but Build.py spend a lot of time on it.
There's a -u flag to skip all module's AutoGen. In my environment, it save 35%~50% of time in rebuild a ROM.
However, if user change one .INF meta-file, then -u flag is not available.
[Idea]
AutoGen can compare meta-file's timestamp and decide if the module's AutoGen can be skipped. With this, when a module's INF is changed, we only run this module's AutoGen, we don't need to run other module's.
[Implementation]
In the end of a module's AutoGen, we create a AutoGenTimeStamp.
The file save a file list that related to this module's AutoGen.
In other word, the file list in AutoGenTimeStamp is INPUT files of module AutoGen, AutoGenTimeStamp file is OUTPUT.
During rebuild, we compare time stamp between INPUT and OUTPUT, and decide if we can skip it.
Below is the Input/Output of a module's AutoGen.
[Input]
1. All the DSC/DEC/FDF used by the platform.
2. INF file of a module.
3. Source files of a module, list in [Sources] section of INF.
4. All the library link by the module.
5. All the .h files included by the module's sources.
[Output]
AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp
[Testing]
This patch save my build time. When I make a change without touching DSC/DEC/FDF, it is absolutely much faster than original rebuild, 35%~50% time saving in my environment (compare to original tool rebuild time).
If I change any DSC/DEC/FDF, there's no performance improve, because it can't skip any module's AutoGen.
Please note that if your environment will generate DSC/FDF during prebuild, it will not skip any AutoGen because of DSC timestamp is changed. This will require prebuild script not to update meta file when content is not changed.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++++++
BaseTools/Source/Python/AutoGen/GenMake.py | 3 +
BaseTools/Source/Python/GenFds/FdfParser.py | 4 +
.../Source/Python/Workspace/MetaFileParser.py | 4 +
4 files changed, 121 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index f35ae25..daa5b35 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspace.MetaFileCommentParser import UsageList from Common.MultipleWorkspace import MultipleWorkspace as mws import InfSectionParser
+import datetime
## Regular expression for splitting Dependency Expression string into tokens gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +641,31 @@ class WorkspaceAutoGen(AutoGen):
self._MakeFileDir = None
self._BuildCommand = None
+ #
+ # Create list of metafiles of the workspace
+ #
+ AllWorkSpaceMetaFiles = self._CreateMetaFileList()
+
+ #
+ # Retrieve latest modified time of all metafiles
+ #
+ SrcTimeStamp = 0
+ for f in AllWorkSpaceMetaFiles:
+ if os.stat(f)[8] > SrcTimeStamp:
+ SrcTimeStamp = os.stat(f)[8]
+ self._SrcTimeStamp = SrcTimeStamp
+
+ #
+ # Write metafile list to build directory
+ #
+ AutoGenFilePath = os.path.join(self.BuildDir, 'AutoGen')
+ if os.path.exists (AutoGenFilePath):
+ os.remove(AutoGenFilePath)
+ if not os.path.exists(self.BuildDir):
+ os.makedirs(self.BuildDir)
+ with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file:
+ for f in AllWorkSpaceMetaFiles:
+ print >> file, f
return True
def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, PcdDatumType, Value):
@@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen):
Value = '0'
return Value
+ def _CreateMetaFileList(self):
+ AllWorkSpaceMetaFiles = []
+ #
+ # add fdf
+ #
+ AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
+ if self.FdfFile:
+ FdfFiles = GlobalData.gFdfParser.GetAllIncludedFile()
+ for f in FdfFiles:
+ AllWorkSpaceMetaFiles.append (f.FileName)
+ #
+ # add dsc & dec
+ #
+ for BuildData in self.BuildDatabase._CACHE_.values():
+ if 'dsc' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+ for filePath in BuildData._RawData.IncludedFiles:
+ AllWorkSpaceMetaFiles.append(filePath.Path)
+ if 'dec' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+
+ return AllWorkSpaceMetaFiles
+
## _CheckDuplicateInFV() method
#
# Check whether there is duplicate modules/files exist in FV section.
@@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen):
# to the [depex] section in module's inf file.
#
class ModuleAutoGen(AutoGen):
+ ## Cache the timestamps of metafiles of every module in a class variable
+ #
+ TimeDict = {}
+
## The real constructor of ModuleAutoGen
#
# This method is not supposed to be called by users of ModuleAutoGen. It's @@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen):
self._FinalBuildTargetList = None
self._FileTypes = None
self._BuildRules = None
+
+ self._TimeStampPath = None
+
+ self.AutoGenDepSet = set()
+
## The Modules referenced to this Library
# Only Library has this attribute @@ -3946,6 +4004,8 @@ class ModuleAutoGen(AutoGen):
if self.IsMakeFileCreated:
return
+ if self.CanSkip():
+ return
if not self.IsLibrary and CreateLibraryMakeFile:
for LibraryAutoGen in self.LibraryAutoGenList:
@@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen):
EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of makefile for module %s [%s]" %
(self.Name, self.Arch))
+ self.CreateTimeStamp(Makefile)
self.IsMakeFileCreated = True
def CopyBinaryFiles(self):
@@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen):
def CreateCodeFile(self, CreateLibraryCodeFile=True):
if self.IsCodeFileCreated:
return
+ if self.CanSkip():
+ return
# Need to generate PcdDatabase even PcdDriver is binarymodule
if self.IsBinaryModule and self.PcdIsDriver != '':
@@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen):
self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)
return self._LibraryAutoGenList
+ ## Decide whether we can skip the ModuleAutoGen process
+ # If any source file is newer than the modeule than we cannot skip
+ #
+ def CanSkip(self):
+ if not os.path.exists(self.GetTimeStampPath()):
+ return False
+ #last creation time of the module
+ DstTimeStamp = os.stat(self.GetTimeStampPath())[8]
+
+ SrcTimeStamp = self.Workspace._SrcTimeStamp
+ if SrcTimeStamp > DstTimeStamp:
+ return False
+
+ with open(self.GetTimeStampPath(),'r') as f:
+ for source in f:
+ source = source.rstrip('\n')
+ if source not in ModuleAutoGen.TimeDict :
+ ModuleAutoGen.TimeDict[source] = os.stat(source)[8]
+ if ModuleAutoGen.TimeDict[source] > DstTimeStamp:
+ return False
+ return True
+
+ def GetTimeStampPath(self):
+ if self._TimeStampPath == None:
+ self._TimeStampPath = os.path.join(self.MakeFileDir, 'AutoGenTimeStamp')
+ return self._TimeStampPath
+ def CreateTimeStamp(self, Makefile):
+
+ FileSet = set()
+
+ FileSet.add (self.MetaFile.Path)
+
+ for SourceFile in self.Module.Sources:
+ FileSet.add (SourceFile.Path)
+
+ for Lib in self.DependentLibraryList:
+ FileSet.add (Lib.MetaFile.Path)
+
+ for f in self.AutoGenDepSet:
+ FileSet.add (f.Path)
+
+ if os.path.exists (self.GetTimeStampPath()):
+ os.remove (self.GetTimeStampPath())
+ with open(self.GetTimeStampPath(), 'w+') as file:
+ for f in FileSet:
+ print >> file, f
+
Module = property(_GetModule)
Name = property(_GetBaseName)
Guid = property(_GetGuid)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 51c5238..ea07b97 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -801,6 +801,9 @@ cleanlib:
if not self.FileDependency[File]:
self.FileDependency[File] = ['$(FORCE_REBUILD)']
continue
+
+ self._AutoGenObject.AutoGenDepSet |=
+ set(self.FileDependency[File])
+
# skip non-C files
if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":
continue
diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 2900283..1093ebb 100644
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -4781,6 +4781,10 @@ class FdfParser:
return False
+ def GetAllIncludedFile (self):
+ global AllIncludeFileList
+ return AllIncludeFileList
+
if __name__ == "__main__":
import sys
try:
diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index 1a5fdf5..d0ab5e9 100644
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -852,6 +852,8 @@ class DscParser(MetaFileParser):
SymbolPattern = ValueExpression.SymbolPattern
+ IncludedFiles = set()
+
## Constructor of DscParser
#
# Initialize object of DscParser
@@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser):
Parser = DscParser(IncludedFile1, self._FileType, self._Arch, IncludedFileTable,
Owner=Owner, From=Owner)
+ self.IncludedFiles.add (IncludedFile1)
+
# Does not allow lower level included file to include upper level included file
if Parser._From != Owner and int(Owner) > int (Parser._From):
EdkLogger.error('parser', FILE_ALREADY_EXIST, File=self._FileWithError,
--
2.7.4.windows.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
2016-11-19 13:43 ` Zhu, Yonghong
@ 2016-11-21 6:17 ` Lin, Derek (HPS UEFI Dev)
0 siblings, 0 replies; 4+ messages in thread
From: Lin, Derek (HPS UEFI Dev) @ 2016-11-21 6:17 UTC (permalink / raw)
To: Zhu, Yonghong, 'edk2-devel@lists.01.org'; +Cc: Gao, Liming
Hi Yonghong,
Thanks for review.
I just sent a patch v2 to fix them.
Thanks,
Derek
-----Original Message-----
From: Zhu, Yonghong [mailto:yonghong.zhu@intel.com]
Sent: Saturday, November 19, 2016 9:43 PM
To: Lin, Derek (HPS UEFI Dev) <derek.lin2@hpe.com>; 'edk2-devel@lists.01.org' <edk2-devel@lists.01.org>
Cc: Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
Hi Derek Lin,
Thanks for your contribution. I tried this patch and have few comments.
1. build MdeModulePkg failure. Please help to fix. Thanks.
File "D:\Git_Latest\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line 702, in _CreateMetaFileList
AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
AttributeError: 'str' object has no attribute 'Path'
2. Please help to use the PatchCheck.py under BaseTools\Scripts to fix some coding style issues. Thanks.
Best Regards,
Zhu Yonghong
-----Original Message-----
From: Zhu, Yonghong
Sent: Friday, November 18, 2016 1:18 PM
To: Derek Lin <derek.lin2@hpe.com>; edk2-devel@lists.01.org
Cc: Gao, Liming <liming.gao@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
Thanks for your contribution. I will review it and do some verification. May give comment later.
Best Regards,
Zhu Yonghong
-----Original Message-----
From: Derek Lin [mailto:derek.lin2@hpe.com]
Sent: Friday, November 18, 2016 10:30 AM
To: edk2-devel@lists.01.org
Cc: derek.lin2@hpe.com; Zhu, Yonghong <yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>
Subject: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp.
[Introduction]
The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/AutoGen.h/makefile. When we only change .c .h code, the AutoGen might be not necessary, but Build.py spend a lot of time on it.
There's a -u flag to skip all module's AutoGen. In my environment, it save 35%~50% of time in rebuild a ROM.
However, if user change one .INF meta-file, then -u flag is not available.
[Idea]
AutoGen can compare meta-file's timestamp and decide if the module's AutoGen can be skipped. With this, when a module's INF is changed, we only run this module's AutoGen, we don't need to run other module's.
[Implementation]
In the end of a module's AutoGen, we create a AutoGenTimeStamp.
The file save a file list that related to this module's AutoGen.
In other word, the file list in AutoGenTimeStamp is INPUT files of module AutoGen, AutoGenTimeStamp file is OUTPUT.
During rebuild, we compare time stamp between INPUT and OUTPUT, and decide if we can skip it.
Below is the Input/Output of a module's AutoGen.
[Input]
1. All the DSC/DEC/FDF used by the platform.
2. INF file of a module.
3. Source files of a module, list in [Sources] section of INF.
4. All the library link by the module.
5. All the .h files included by the module's sources.
[Output]
AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp
[Testing]
This patch save my build time. When I make a change without touching DSC/DEC/FDF, it is absolutely much faster than original rebuild, 35%~50% time saving in my environment (compare to original tool rebuild time).
If I change any DSC/DEC/FDF, there's no performance improve, because it can't skip any module's AutoGen.
Please note that if your environment will generate DSC/FDF during prebuild, it will not skip any AutoGen because of DSC timestamp is changed. This will require prebuild script not to update meta file when content is not changed.
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Derek Lin <derek.lin2@hpe.com>
---
BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++++++
BaseTools/Source/Python/AutoGen/GenMake.py | 3 +
BaseTools/Source/Python/GenFds/FdfParser.py | 4 +
.../Source/Python/Workspace/MetaFileParser.py | 4 +
4 files changed, 121 insertions(+)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index f35ae25..daa5b35 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspace.MetaFileCommentParser import UsageList from Common.MultipleWorkspace import MultipleWorkspace as mws import InfSectionParser
+import datetime
## Regular expression for splitting Dependency Expression string into tokens gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +641,31 @@ class WorkspaceAutoGen(AutoGen):
self._MakeFileDir = None
self._BuildCommand = None
+ #
+ # Create list of metafiles of the workspace
+ #
+ AllWorkSpaceMetaFiles = self._CreateMetaFileList()
+
+ #
+ # Retrieve latest modified time of all metafiles
+ #
+ SrcTimeStamp = 0
+ for f in AllWorkSpaceMetaFiles:
+ if os.stat(f)[8] > SrcTimeStamp:
+ SrcTimeStamp = os.stat(f)[8]
+ self._SrcTimeStamp = SrcTimeStamp
+
+ #
+ # Write metafile list to build directory
+ #
+ AutoGenFilePath = os.path.join(self.BuildDir, 'AutoGen')
+ if os.path.exists (AutoGenFilePath):
+ os.remove(AutoGenFilePath)
+ if not os.path.exists(self.BuildDir):
+ os.makedirs(self.BuildDir)
+ with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file:
+ for f in AllWorkSpaceMetaFiles:
+ print >> file, f
return True
def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, PcdDatumType, Value):
@@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen):
Value = '0'
return Value
+ def _CreateMetaFileList(self):
+ AllWorkSpaceMetaFiles = []
+ #
+ # add fdf
+ #
+ AllWorkSpaceMetaFiles.append (self.FdfFile.Path)
+ if self.FdfFile:
+ FdfFiles = GlobalData.gFdfParser.GetAllIncludedFile()
+ for f in FdfFiles:
+ AllWorkSpaceMetaFiles.append (f.FileName)
+ #
+ # add dsc & dec
+ #
+ for BuildData in self.BuildDatabase._CACHE_.values():
+ if 'dsc' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+ for filePath in BuildData._RawData.IncludedFiles:
+ AllWorkSpaceMetaFiles.append(filePath.Path)
+ if 'dec' is BuildData.MetaFile.Ext:
+ AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path)
+
+ return AllWorkSpaceMetaFiles
+
## _CheckDuplicateInFV() method
#
# Check whether there is duplicate modules/files exist in FV section.
@@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen):
# to the [depex] section in module's inf file.
#
class ModuleAutoGen(AutoGen):
+ ## Cache the timestamps of metafiles of every module in a class variable
+ #
+ TimeDict = {}
+
## The real constructor of ModuleAutoGen
#
# This method is not supposed to be called by users of ModuleAutoGen. It's @@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen):
self._FinalBuildTargetList = None
self._FileTypes = None
self._BuildRules = None
+
+ self._TimeStampPath = None
+
+ self.AutoGenDepSet = set()
+
## The Modules referenced to this Library
# Only Library has this attribute @@ -3946,6 +4004,8 @@ class ModuleAutoGen(AutoGen):
if self.IsMakeFileCreated:
return
+ if self.CanSkip():
+ return
if not self.IsLibrary and CreateLibraryMakeFile:
for LibraryAutoGen in self.LibraryAutoGenList:
@@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen):
EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of makefile for module %s [%s]" %
(self.Name, self.Arch))
+ self.CreateTimeStamp(Makefile)
self.IsMakeFileCreated = True
def CopyBinaryFiles(self):
@@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen):
def CreateCodeFile(self, CreateLibraryCodeFile=True):
if self.IsCodeFileCreated:
return
+ if self.CanSkip():
+ return
# Need to generate PcdDatabase even PcdDriver is binarymodule
if self.IsBinaryModule and self.PcdIsDriver != '':
@@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen):
self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE)
return self._LibraryAutoGenList
+ ## Decide whether we can skip the ModuleAutoGen process
+ # If any source file is newer than the modeule than we cannot skip
+ #
+ def CanSkip(self):
+ if not os.path.exists(self.GetTimeStampPath()):
+ return False
+ #last creation time of the module
+ DstTimeStamp = os.stat(self.GetTimeStampPath())[8]
+
+ SrcTimeStamp = self.Workspace._SrcTimeStamp
+ if SrcTimeStamp > DstTimeStamp:
+ return False
+
+ with open(self.GetTimeStampPath(),'r') as f:
+ for source in f:
+ source = source.rstrip('\n')
+ if source not in ModuleAutoGen.TimeDict :
+ ModuleAutoGen.TimeDict[source] = os.stat(source)[8]
+ if ModuleAutoGen.TimeDict[source] > DstTimeStamp:
+ return False
+ return True
+
+ def GetTimeStampPath(self):
+ if self._TimeStampPath == None:
+ self._TimeStampPath = os.path.join(self.MakeFileDir, 'AutoGenTimeStamp')
+ return self._TimeStampPath
+ def CreateTimeStamp(self, Makefile):
+
+ FileSet = set()
+
+ FileSet.add (self.MetaFile.Path)
+
+ for SourceFile in self.Module.Sources:
+ FileSet.add (SourceFile.Path)
+
+ for Lib in self.DependentLibraryList:
+ FileSet.add (Lib.MetaFile.Path)
+
+ for f in self.AutoGenDepSet:
+ FileSet.add (f.Path)
+
+ if os.path.exists (self.GetTimeStampPath()):
+ os.remove (self.GetTimeStampPath())
+ with open(self.GetTimeStampPath(), 'w+') as file:
+ for f in FileSet:
+ print >> file, f
+
Module = property(_GetModule)
Name = property(_GetBaseName)
Guid = property(_GetGuid)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 51c5238..ea07b97 100644
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -801,6 +801,9 @@ cleanlib:
if not self.FileDependency[File]:
self.FileDependency[File] = ['$(FORCE_REBUILD)']
continue
+
+ self._AutoGenObject.AutoGenDepSet |=
+ set(self.FileDependency[File])
+
# skip non-C files
if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":
continue
diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 2900283..1093ebb 100644
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -4781,6 +4781,10 @@ class FdfParser:
return False
+ def GetAllIncludedFile (self):
+ global AllIncludeFileList
+ return AllIncludeFileList
+
if __name__ == "__main__":
import sys
try:
diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index 1a5fdf5..d0ab5e9 100644
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -852,6 +852,8 @@ class DscParser(MetaFileParser):
SymbolPattern = ValueExpression.SymbolPattern
+ IncludedFiles = set()
+
## Constructor of DscParser
#
# Initialize object of DscParser
@@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser):
Parser = DscParser(IncludedFile1, self._FileType, self._Arch, IncludedFileTable,
Owner=Owner, From=Owner)
+ self.IncludedFiles.add (IncludedFile1)
+
# Does not allow lower level included file to include upper level included file
if Parser._From != Owner and int(Owner) > int (Parser._From):
EdkLogger.error('parser', FILE_ALREADY_EXIST, File=self._FileWithError,
--
2.7.4.windows.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-11-21 6:17 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-11-18 2:29 [PATCH] BaseTools: Skip module AutoGen by comparing timestamp Derek Lin
2016-11-18 5:17 ` Zhu, Yonghong
2016-11-19 13:43 ` Zhu, Yonghong
2016-11-21 6:17 ` Lin, Derek (HPS UEFI Dev)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox