From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 98A65803B7 for ; Wed, 22 Mar 2017 20:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1490240718; x=1521776718; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=Pxr46eIo5+RXe5OPtk8POgjOE6+5po4ZDJ39431E/gc=; b=Q5EzGy4PTCojNR4KrNORQ9dOhsN7v/V2ImuSCNQ+3ORlAGAW5JOGUNbJ ElfJEOz3iLuLwEbZZkZm5qnz8ulrxQ==; Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Mar 2017 20:45:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,208,1486454400"; d="scan'208";a="947278034" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga003.jf.intel.com with ESMTP; 22 Mar 2017 20:45:17 -0700 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 22 Mar 2017 20:45:16 -0700 Received: from shsmsx104.ccr.corp.intel.com (10.239.4.70) by FMSMSX112.amr.corp.intel.com (10.18.116.6) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 22 Mar 2017 20:45:16 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.20]) by SHSMSX104.ccr.corp.intel.com ([10.239.4.70]) with mapi id 14.03.0248.002; Thu, 23 Mar 2017 11:45:14 +0800 From: "Zhu, Yonghong" To: Derek Lin , "edk2-devel@lists.01.org" CC: "Gao, Liming" , "Zhu, Yonghong" Thread-Topic: [PATCH v3] BaseTools: Skip module AutoGen by comparing timestamp. Thread-Index: AQHSjm9SPh4oQr4WakWTusb7fVWAbKGh8lsw Date: Thu, 23 Mar 2017 03:45:13 +0000 Message-ID: References: <20170224072619.568-1-derek.lin2@hpe.com> In-Reply-To: <20170224072619.568-1-derek.lin2@hpe.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH v3] BaseTools: Skip module AutoGen by comparing timestamp. X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2017 03:45:18 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Yonghong Zhu =20 I will push this patch. Best Regards, Zhu Yonghong -----Original Message----- From: Derek Lin [mailto:derek.lin2@hpe.com]=20 Sent: Friday, February 24, 2017 3:26 PM To: edk2-devel@lists.01.org Cc: Zhu, Yonghong ; Gao, Liming ; Derek Lin Subject: [PATCH v3] BaseTools: Skip module AutoGen by comparing timestamp. [Introduction] The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/Au= toGen.h/makefile. When we only change .c .h code, the AutoGen might be not = necessary, but Build.py spend a lot of time on it. There's a -u flag to skip all module's AutoGen. In my environment, it save = 35%~50% of time in rebuild a ROM. However, if user change one .INF meta-file, then -u flag is not available. [Idea] AutoGen can compare meta-file's timestamp and decide if the module's AutoGe= n can be skipped. With this, when a module's INF is changed, we only run th= is module's AutoGen, we don't need to run other module's. [Implementation] In the end of a module's AutoGen, we create a AutoGenTimeStamp. The file save a file list that related to this module's AutoGen. In other word, the file list in AutoGenTimeStamp is INPUT files of module A= utoGen, AutoGenTimeStamp file is OUTPUT. During rebuild, we compare time stamp between INPUT and OUTPUT, and decide = if we can skip it. Below is the Input/Output of a module's AutoGen. [Input] 1. All the DSC/DEC/FDF used by the platform. 2. Macro and PCD defined by Build Options such as "build -D AAA=3DTRUE --pcd BbbPcd=3D0". 3. INF file of a module. 4. Source files of a module, list in [Sources] section of INF. 5. All the library link by the module. 6. All the .h files included by the module's sources. [Output] AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp [Testing] This patch save my build time. When I make a change without touching DSC/DE= C/FDF, it is absolutely much faster than original rebuild, 35%~50% time sav= ing in my environment (compare to original tool rebuild time). If I change any DSC/DEC/FDF, there's no performance improve, because it can= 't skip any module's AutoGen. Please note that if your environment will generate DSC/FDF during prebuild,= it will not skip any AutoGen because of DSC timestamp is changed. This wil= l require prebuild script not to update metafile when content is not change= d. --- Changes in v3: - Consider BuildOption such as -D AAA=3DTRUE --pcd BbbPcd=3D0, add BuildOpt= ion meta-file. Contributed-under: TianoCore Contribution Agreement 1.0 Signed-off-by: Derek Lin --- BaseTools/Source/Python/AutoGen/AutoGen.py | 136 +++++++++++++++++= ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 3 + BaseTools/Source/Python/GenFds/FdfParser.py | 4 + .../Source/Python/Workspace/MetaFileParser.py | 4 + 4 files changed, 147 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index f35ae252b0..6a2ea7d7ca 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspac= e.MetaFileCommentParser import UsageList from Common.MultipleWorkspace imp= ort MultipleWorkspace as mws import InfSectionParser +import datetime =20 ## Regular expression for splitting Dependency Expression string into toke= ns gDepexTokenPattern =3D re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +6= 41,41 @@ class WorkspaceAutoGen(AutoGen): self._MakeFileDir =3D None self._BuildCommand =3D None =20 + # + # Create BuildOptions Macro & PCD metafile. + # + content =3D 'gCommandLineDefines: ' + content +=3D str(GlobalData.gCommandLineDefines) + content +=3D os.linesep + content +=3D 'BuildOptionPcd: ' + content +=3D str(GlobalData.BuildOptionPcd) + SaveFileOnChange(os.path.join(self.BuildDir, 'BuildOptions'),=20 + content, False) + + # + # Get set of workspace metafiles + # + AllWorkSpaceMetaFiles =3D self._GetMetaFiles(Target, Toolchain,=20 + Arch) + + # + # Retrieve latest modified time of all metafiles + # + SrcTimeStamp =3D 0 + for f in AllWorkSpaceMetaFiles: + if os.stat(f)[8] > SrcTimeStamp: + SrcTimeStamp =3D os.stat(f)[8] + self._SrcTimeStamp =3D SrcTimeStamp + + # + # Write metafile list to build directory + # + AutoGenFilePath =3D os.path.join(self.BuildDir, 'AutoGen') + if os.path.exists (AutoGenFilePath): + os.remove(AutoGenFilePath) + if not os.path.exists(self.BuildDir): + os.makedirs(self.BuildDir) + with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file: + for f in AllWorkSpaceMetaFiles: + print >> file, f return True =20 def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, = PcdDatumType, Value): @@ -668,6 +704,45 @@ class WorkspaceAutoGen(AutoGen): Value =3D '0' return Value =20 + def _GetMetaFiles(self, Target, Toolchain, Arch): + AllWorkSpaceMetaFiles =3D set() + # + # add fdf + # + if self.FdfFile: + AllWorkSpaceMetaFiles.add (self.FdfFile.Path) + if self.FdfFile: + FdfFiles =3D GlobalData.gFdfParser.GetAllIncludedFile() + for f in FdfFiles: + AllWorkSpaceMetaFiles.add (f.FileName) + # + # add dsc + # + AllWorkSpaceMetaFiles.add(self.MetaFile.Path) + + # + # add BuildOption metafile + # + AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir,=20 + 'BuildOptions')) + + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, Target, T= oolchain] + PGen =3D PlatformAutoGen(self, self.MetaFile, Target,=20 + Toolchain, Arch) + + # + # add dec + # + for Package in PGen.PackageList: + AllWorkSpaceMetaFiles.add(Package.MetaFile.Path) + + # + # add included dsc + # + for filePath in Platform._RawData.IncludedFiles: + AllWorkSpaceMetaFiles.add(filePath.Path) + + return AllWorkSpaceMetaFiles + ## _CheckDuplicateInFV() method # # Check whether there is duplicate modules/files exist in FV section.= =20 @@ -2520,6 +2595,10 @@ class PlatformAutoGen(AutoGen): # to the [depex] section in module's inf file. # class ModuleAutoGen(AutoGen): + ## Cache the timestamps of metafiles of every module in a class variab= le + # + TimeDict =3D {} + ## The real constructor of ModuleAutoGen # # This method is not supposed to be called by users of ModuleAutoGen.= It's @@ -2619,6 +2698,11 @@ class ModuleAutoGen(AutoGen): self._FinalBuildTargetList =3D None self._FileTypes =3D None self._BuildRules =3D None + + self._TimeStampPath =3D None + + self.AutoGenDepSet =3D set() + =20 ## The Modules referenced to this Library # Only Library has this attribute @@ -3946,6 +4030,8 @@ class Mod= uleAutoGen(AutoGen): =20 if self.IsMakeFileCreated: return + if self.CanSkip(): + return =20 if not self.IsLibrary and CreateLibraryMakeFile: for LibraryAutoGen in self.LibraryAutoGenList: @@ -3962,6 +4048,7 @@ class ModuleAutoGen(AutoGen): EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = makefile for module %s [%s]" % (self.Name, self.Arch)) =20 + self.CreateTimeStamp(Makefile) self.IsMakeFileCreated =3D True =20 def CopyBinaryFiles(self): @@ -3977,6 +4064,8 @@ class ModuleAutoGen(AutoGen): def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): if self.IsCodeFileCreated: return + if self.CanSkip(): + return =20 # Need to generate PcdDatabase even PcdDriver is binarymodule if self.IsBinaryModule and self.PcdIsDriver !=3D '': @@ -4056,6 +4145,53 @@ class ModuleAutoGen(AutoGen): self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE) return self._LibraryAutoGenList =20 + ## Decide whether we can skip the ModuleAutoGen process + # If any source file is newer than the modeule than we cannot skip + # + def CanSkip(self): + if not os.path.exists(self.GetTimeStampPath()): + return False + #last creation time of the module + DstTimeStamp =3D os.stat(self.GetTimeStampPath())[8] + + SrcTimeStamp =3D self.Workspace._SrcTimeStamp + if SrcTimeStamp > DstTimeStamp: + return False + + with open(self.GetTimeStampPath(),'r') as f: + for source in f: + source =3D source.rstrip('\n') + if source not in ModuleAutoGen.TimeDict : + ModuleAutoGen.TimeDict[source] =3D os.stat(source)[8] + if ModuleAutoGen.TimeDict[source] > DstTimeStamp: + return False + return True + + def GetTimeStampPath(self): + if self._TimeStampPath =3D=3D None: + self._TimeStampPath =3D os.path.join(self.MakeFileDir, 'AutoGe= nTimeStamp') + return self._TimeStampPath + def CreateTimeStamp(self, Makefile): + + FileSet =3D set() + + FileSet.add (self.MetaFile.Path) + + for SourceFile in self.Module.Sources: + FileSet.add (SourceFile.Path) + + for Lib in self.DependentLibraryList: + FileSet.add (Lib.MetaFile.Path) + + for f in self.AutoGenDepSet: + FileSet.add (f.Path) + + if os.path.exists (self.GetTimeStampPath()): + os.remove (self.GetTimeStampPath()) + with open(self.GetTimeStampPath(), 'w+') as file: + for f in FileSet: + print >> file, f + Module =3D property(_GetModule) Name =3D property(_GetBaseName) Guid =3D property(_GetGuid) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 51c5238fd1..ea07b97786 100644 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -801,6 +801,9 @@ cleanlib: if not self.FileDependency[File]: self.FileDependency[File] =3D ['$(FORCE_REBUILD)'] continue + + self._AutoGenObject.AutoGenDepSet |=3D=20 + set(self.FileDependency[File]) + # skip non-C files if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoGen.c= ": continue diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source= /Python/GenFds/FdfParser.py index 27688e2ff8..a1825baac7 100644 --- a/BaseTools/Source/Python/GenFds/FdfParser.py +++ b/BaseTools/Source/Python/GenFds/FdfParser.py @@ -4797,6 +4797,10 @@ class FdfParser: =20 return False =20 + def GetAllIncludedFile (self): + global AllIncludeFileList + return AllIncludeFileList + if __name__ =3D=3D "__main__": import sys try: diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTool= s/Source/Python/Workspace/MetaFileParser.py index 1a5fdf5e62..d0ab5e938b 100644 --- a/BaseTools/Source/Python/Workspace/MetaFileParser.py +++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py @@ -852,6 +852,8 @@ class DscParser(MetaFileParser): =20 SymbolPattern =3D ValueExpression.SymbolPattern =20 + IncludedFiles =3D set() + ## Constructor of DscParser # # Initialize object of DscParser @@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser): Parser =3D DscParser(IncludedFile1, self._FileType, self._Arch= , IncludedFileTable, Owner=3DOwner, From=3DOwner) =20 + self.IncludedFiles.add (IncludedFile1) + # Does not allow lower level included file to include upper le= vel included file if Parser._From !=3D Owner and int(Owner) > int (Parser._From)= : EdkLogger.error('parser', FILE_ALREADY_EXIST, File=3Dself.= _FileWithError, -- 2.11.1.windows.1