From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0765181E5E for ; Sat, 19 Nov 2016 05:43:10 -0800 (PST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP; 19 Nov 2016 05:43:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.31,515,1473145200"; d="scan'208";a="1087513027" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga002.fm.intel.com with ESMTP; 19 Nov 2016 05:43:15 -0800 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.248.2; Sat, 19 Nov 2016 05:43:14 -0800 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by FMSMSX112.amr.corp.intel.com (10.18.116.6) with Microsoft SMTP Server (TLS) id 14.3.248.2; Sat, 19 Nov 2016 05:43:14 -0800 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.96]) by shsmsx102.ccr.corp.intel.com ([169.254.2.239]) with mapi id 14.03.0248.002; Sat, 19 Nov 2016 21:43:10 +0800 From: "Zhu, Yonghong" To: 'Derek Lin' , "'edk2-devel@lists.01.org'" CC: "Gao, Liming" , "Zhu, Yonghong" Thread-Topic: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. Thread-Index: AQHSQUO8/mcySUThnky8ygXxXaOiB6DeM2nQgAIaPMA= Date: Sat, 19 Nov 2016 13:43:10 +0000 Message-ID: References: <1479436171-8656-1-git-send-email-derek.lin2@hpe.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2016 13:43:10 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Derek Lin, Thanks for your contribution. I tried this patch and have few comments. 1. build MdeModulePkg failure. Please help to fix. Thanks. File "D:\Git_Latest\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line = 702, in _CreateMetaFileList AllWorkSpaceMetaFiles.append (self.FdfFile.Path) AttributeError: 'str' object has no attribute 'Path' 2. Please help to use the PatchCheck.py under BaseTools\Scripts to fix some= coding style issues. Thanks. Best Regards, Zhu Yonghong -----Original Message----- From: Zhu, Yonghong=20 Sent: Friday, November 18, 2016 1:18 PM To: Derek Lin ; edk2-devel@lists.01.org Cc: Gao, Liming ; Zhu, Yonghong Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. Thanks for your contribution. I will review it and do some verification. Ma= y give comment later. Best Regards, Zhu Yonghong -----Original Message----- From: Derek Lin [mailto:derek.lin2@hpe.com]=20 Sent: Friday, November 18, 2016 10:30 AM To: edk2-devel@lists.01.org Cc: derek.lin2@hpe.com; Zhu, Yonghong ; Gao, Liming= Subject: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. [Introduction] The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/Au= toGen.h/makefile. When we only change .c .h code, the AutoGen might be not = necessary, but Build.py spend a lot of time on it. There's a -u flag to skip all module's AutoGen. In my environment, it save = 35%~50% of time in rebuild a ROM. However, if user change one .INF meta-file, then -u flag is not available. [Idea] AutoGen can compare meta-file's timestamp and decide if the module's AutoGe= n can be skipped. With this, when a module's INF is changed, we only run th= is module's AutoGen, we don't need to run other module's. [Implementation] In the end of a module's AutoGen, we create a AutoGenTimeStamp. The file save a file list that related to this module's AutoGen. In other word, the file list in AutoGenTimeStamp is INPUT files of module A= utoGen, AutoGenTimeStamp file is OUTPUT. During rebuild, we compare time stamp between INPUT and OUTPUT, and decide = if we can skip it. Below is the Input/Output of a module's AutoGen. [Input] 1. All the DSC/DEC/FDF used by the platform. 2. INF file of a module. 3. Source files of a module, list in [Sources] section of INF. 4. All the library link by the module. 5. All the .h files included by the module's sources. [Output] AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp [Testing] This patch save my build time. When I make a change without touching DSC/DE= C/FDF, it is absolutely much faster than original rebuild, 35%~50% time sav= ing in my environment (compare to original tool rebuild time). If I change any DSC/DEC/FDF, there's no performance improve, because it can= 't skip any module's AutoGen. Please note that if your environment will generate DSC/FDF during prebuild,= it will not skip any AutoGen because of DSC timestamp is changed. This wil= l require prebuild script not to update meta file when content is not chang= ed. Contributed-under: TianoCore Contribution Agreement 1.0 Signed-off-by: Derek Lin --- BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++= ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 3 + BaseTools/Source/Python/GenFds/FdfParser.py | 4 + .../Source/Python/Workspace/MetaFileParser.py | 4 + 4 files changed, 121 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index f35ae25..daa5b35 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspac= e.MetaFileCommentParser import UsageList from Common.MultipleWorkspace imp= ort MultipleWorkspace as mws import InfSectionParser +import datetime =20 ## Regular expression for splitting Dependency Expression string into toke= ns gDepexTokenPattern =3D re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +6= 41,31 @@ class WorkspaceAutoGen(AutoGen): self._MakeFileDir =3D None self._BuildCommand =3D None =20 + # + # Create list of metafiles of the workspace + # + AllWorkSpaceMetaFiles =3D self._CreateMetaFileList() + =20 + # + # Retrieve latest modified time of all metafiles + # + SrcTimeStamp =3D 0 + for f in AllWorkSpaceMetaFiles: + if os.stat(f)[8] > SrcTimeStamp: + SrcTimeStamp =3D os.stat(f)[8] =20 + self._SrcTimeStamp =3D SrcTimeStamp + + # + # Write metafile list to build directory + # + AutoGenFilePath =3D os.path.join(self.BuildDir, 'AutoGen') + if os.path.exists (AutoGenFilePath): + os.remove(AutoGenFilePath) + if not os.path.exists(self.BuildDir): + os.makedirs(self.BuildDir) + with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file: + for f in AllWorkSpaceMetaFiles: + print >> file, f return True =20 def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, = PcdDatumType, Value): @@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen): Value =3D '0' return Value =20 + def _CreateMetaFileList(self): + AllWorkSpaceMetaFiles =3D [] + # + # add fdf + # + AllWorkSpaceMetaFiles.append (self.FdfFile.Path) + if self.FdfFile: + FdfFiles =3D GlobalData.gFdfParser.GetAllIncludedFile() + for f in FdfFiles: + AllWorkSpaceMetaFiles.append (f.FileName) =20 + # + # add dsc & dec + # + for BuildData in self.BuildDatabase._CACHE_.values(): + if 'dsc' is BuildData.MetaFile.Ext: + AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path) + for filePath in BuildData._RawData.IncludedFiles: + AllWorkSpaceMetaFiles.append(filePath.Path) + if 'dec' is BuildData.MetaFile.Ext: + AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path) + + return AllWorkSpaceMetaFiles + =20 ## _CheckDuplicateInFV() method # # Check whether there is duplicate modules/files exist in FV section.= =20 @@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen): # to the [depex] section in module's inf file. # class ModuleAutoGen(AutoGen): + ## Cache the timestamps of metafiles of every module in a class variab= le + # + TimeDict =3D {} + =20 ## The real constructor of ModuleAutoGen # # This method is not supposed to be called by users of ModuleAutoGen.= It's @@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen): self._FinalBuildTargetList =3D None self._FileTypes =3D None self._BuildRules =3D None + + self._TimeStampPath =3D None + + self.AutoGenDepSet =3D set() + =20 =20 ## The Modules referenced to this Library # Only Library has this attribute @@ -3946,6 +4004,8 @@ class Mod= uleAutoGen(AutoGen): =20 if self.IsMakeFileCreated: return + if self.CanSkip(): + return =20 if not self.IsLibrary and CreateLibraryMakeFile: for LibraryAutoGen in self.LibraryAutoGenList: @@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen): EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = makefile for module %s [%s]" % (self.Name, self.Arch)) =20 + self.CreateTimeStamp(Makefile) self.IsMakeFileCreated =3D True =20 def CopyBinaryFiles(self): @@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen): def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): if self.IsCodeFileCreated: return + if self.CanSkip(): + return =20 # Need to generate PcdDatabase even PcdDriver is binarymodule if self.IsBinaryModule and self.PcdIsDriver !=3D '': @@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen): self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE) return self._LibraryAutoGenList =20 + ## Decide whether we can skip the ModuleAutoGen process =20 + # If any source file is newer than the modeule than we cannot skip + # + def CanSkip(self): + if not os.path.exists(self.GetTimeStampPath()): + return False + #last creation time of the module + DstTimeStamp =3D os.stat(self.GetTimeStampPath())[8] + =20 + SrcTimeStamp =3D self.Workspace._SrcTimeStamp + if SrcTimeStamp > DstTimeStamp: + return False + + with open(self.GetTimeStampPath(),'r') as f: + for source in f: + source =3D source.rstrip('\n') + if source not in ModuleAutoGen.TimeDict : + ModuleAutoGen.TimeDict[source] =3D os.stat(source)[8]= =20 + if ModuleAutoGen.TimeDict[source] > DstTimeStamp: + return False + return True =20 + + def GetTimeStampPath(self): + if self._TimeStampPath =3D=3D None: + self._TimeStampPath =3D os.path.join(self.MakeFileDir, 'AutoGe= nTimeStamp') + return self._TimeStampPath + def CreateTimeStamp(self, Makefile): + + FileSet =3D set() + + FileSet.add (self.MetaFile.Path) + + for SourceFile in self.Module.Sources: + FileSet.add (SourceFile.Path) + + for Lib in self.DependentLibraryList: + FileSet.add (Lib.MetaFile.Path) + + for f in self.AutoGenDepSet: + FileSet.add (f.Path) + + if os.path.exists (self.GetTimeStampPath()): + os.remove (self.GetTimeStampPath()) + with open(self.GetTimeStampPath(), 'w+') as file: + for f in FileSet: + print >> file, f + Module =3D property(_GetModule) Name =3D property(_GetBaseName) Guid =3D property(_GetGuid) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 51c5238..ea07b97 100644 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -801,6 +801,9 @@ cleanlib: if not self.FileDependency[File]: self.FileDependency[File] =3D ['$(FORCE_REBUILD)'] continue + + self._AutoGenObject.AutoGenDepSet |=3D=20 + set(self.FileDependency[File]) + # skip non-C files if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoGen.c= ": continue diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source= /Python/GenFds/FdfParser.py index 2900283..1093ebb 100644 --- a/BaseTools/Source/Python/GenFds/FdfParser.py +++ b/BaseTools/Source/Python/GenFds/FdfParser.py @@ -4781,6 +4781,10 @@ class FdfParser: =20 return False =20 + def GetAllIncludedFile (self): + global AllIncludeFileList + return AllIncludeFileList + if __name__ =3D=3D "__main__": import sys try: diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTool= s/Source/Python/Workspace/MetaFileParser.py index 1a5fdf5..d0ab5e9 100644 --- a/BaseTools/Source/Python/Workspace/MetaFileParser.py +++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py @@ -852,6 +852,8 @@ class DscParser(MetaFileParser): =20 SymbolPattern =3D ValueExpression.SymbolPattern =20 + IncludedFiles =3D set() + ## Constructor of DscParser # # Initialize object of DscParser @@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser): Parser =3D DscParser(IncludedFile1, self._FileType, self._Arch= , IncludedFileTable, Owner=3DOwner, From=3DOwner) =20 + self.IncludedFiles.add (IncludedFile1) + # Does not allow lower level included file to include upper le= vel included file if Parser._From !=3D Owner and int(Owner) > int (Parser._From)= : EdkLogger.error('parser', FILE_ALREADY_EXIST, File=3Dself.= _FileWithError, -- 2.7.4.windows.1