From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0729.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe46::729]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 0585081D57 for ; Sun, 20 Nov 2016 22:17:35 -0800 (PST) Received: from DF4PR84MB0025.NAMPRD84.PROD.OUTLOOK.COM (10.162.192.14) by DF4PR84MB0026.NAMPRD84.PROD.OUTLOOK.COM (10.162.192.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.8; Mon, 21 Nov 2016 06:17:33 +0000 Received: from DF4PR84MB0025.NAMPRD84.PROD.OUTLOOK.COM ([10.162.192.14]) by DF4PR84MB0025.NAMPRD84.PROD.OUTLOOK.COM ([10.162.192.14]) with mapi id 15.01.0734.014; Mon, 21 Nov 2016 06:17:32 +0000 From: "Lin, Derek (HPS UEFI Dev)" To: "Zhu, Yonghong" , "'edk2-devel@lists.01.org'" CC: "Gao, Liming" Thread-Topic: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. Thread-Index: AQHSQUOtlzJOfxH+zU2go7VmKDRsxaDeM7EAgAIfggCAAqeCAA== Date: Mon, 21 Nov 2016 06:17:32 +0000 Message-ID: References: <1479436171-8656-1-git-send-email-derek.lin2@hpe.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=derek.lin2@hpe.com; x-originating-ip: [15.211.131.14] x-microsoft-exchange-diagnostics: 1; DF4PR84MB0026; 7:vVe6gOWH+/Bbx4Z9wYHDa8vO6wObML9pIs7Qax+HaVwGXACO4F6Czxm4cpU+mgrGYz4y/98N4zxSt7/gMJq+rSiiKfcvDx+F2EwMKflQ+mQN/Cl+tkmuNtX9s4k/caFgG/mElZeSzglh01+obmteCMVCHl63WTT4C1tnd31EZdVvBOtdPJPwUpja9qNrtFNz2FQroVSMi+utE3qSExCNSEtHY21ncgdzgDaYendoAkyZJqGPNUFW8VKBdPjTzGNB9NZLYlEkj8Ob7+dSTBCIJTu0MZjVppzSN7bjizPxun+8/+CRFPNK446Pcd8MRAeK3rD+Q8NIH1pxszfOmayHr5eksYh56VaeRKm0QUrOVbM= x-ms-office365-filtering-correlation-id: 6199b566-d829-4b71-9f14-08d411d6146a x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DF4PR84MB0026; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(227479698468861)(162533806227266)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040307)(6060326)(6045199)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026)(6041248)(6061324); SRVR:DF4PR84MB0026; BCL:0; PCL:0; RULEID:; SRVR:DF4PR84MB0026; x-forefront-prvs: 01334458E5 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(189002)(13464003)(199003)(377454003)(122556002)(66066001)(5660300001)(229853002)(6506003)(38730400001)(81156014)(81166006)(8676002)(2950100002)(2900100001)(3900700001)(7696004)(87936001)(3660700001)(74316002)(189998001)(3280700002)(33656002)(105586002)(3846002)(6116002)(68736007)(99286002)(102836003)(86362001)(4326007)(575784001)(2906002)(305945005)(101416001)(97736004)(5001770100001)(9686002)(77096005)(92566002)(76176999)(54356999)(50986999)(106116001)(106356001)(7846002)(8936002)(7736002)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:DF4PR84MB0026; H:DF4PR84MB0025.NAMPRD84.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: hpe.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Nov 2016 06:17:32.8026 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR84MB0026 Subject: Re: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 06:17:35 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Yonghong, Thanks for review. I just sent a patch v2 to fix them. Thanks, Derek -----Original Message----- From: Zhu, Yonghong [mailto:yonghong.zhu@intel.com]=20 Sent: Saturday, November 19, 2016 9:43 PM To: Lin, Derek (HPS UEFI Dev) ; 'edk2-devel@lists.01.or= g' Cc: Gao, Liming ; Zhu, Yonghong Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. Hi Derek Lin, Thanks for your contribution. I tried this patch and have few comments. 1. build MdeModulePkg failure. Please help to fix. Thanks. File "D:\Git_Latest\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line = 702, in _CreateMetaFileList AllWorkSpaceMetaFiles.append (self.FdfFile.Path) AttributeError: 'str' object has no attribute 'Path' 2. Please help to use the PatchCheck.py under BaseTools\Scripts to fix some= coding style issues. Thanks. Best Regards, Zhu Yonghong -----Original Message----- From: Zhu, Yonghong=20 Sent: Friday, November 18, 2016 1:18 PM To: Derek Lin ; edk2-devel@lists.01.org Cc: Gao, Liming ; Zhu, Yonghong Subject: RE: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. Thanks for your contribution. I will review it and do some verification. Ma= y give comment later. Best Regards, Zhu Yonghong -----Original Message----- From: Derek Lin [mailto:derek.lin2@hpe.com]=20 Sent: Friday, November 18, 2016 10:30 AM To: edk2-devel@lists.01.org Cc: derek.lin2@hpe.com; Zhu, Yonghong ; Gao, Liming= Subject: [PATCH] BaseTools: Skip module AutoGen by comparing timestamp. [Introduction] The BaseTool Build.py AutoGen parse INF meta-file and generate AutoGen.c/Au= toGen.h/makefile. When we only change .c .h code, the AutoGen might be not = necessary, but Build.py spend a lot of time on it. There's a -u flag to skip all module's AutoGen. In my environment, it save = 35%~50% of time in rebuild a ROM. However, if user change one .INF meta-file, then -u flag is not available. [Idea] AutoGen can compare meta-file's timestamp and decide if the module's AutoGe= n can be skipped. With this, when a module's INF is changed, we only run th= is module's AutoGen, we don't need to run other module's. [Implementation] In the end of a module's AutoGen, we create a AutoGenTimeStamp. The file save a file list that related to this module's AutoGen. In other word, the file list in AutoGenTimeStamp is INPUT files of module A= utoGen, AutoGenTimeStamp file is OUTPUT. During rebuild, we compare time stamp between INPUT and OUTPUT, and decide = if we can skip it. Below is the Input/Output of a module's AutoGen. [Input] 1. All the DSC/DEC/FDF used by the platform. 2. INF file of a module. 3. Source files of a module, list in [Sources] section of INF. 4. All the library link by the module. 5. All the .h files included by the module's sources. [Output] AutoGen.c/AutoGen.h/makefile/AutoGenTimeStamp [Testing] This patch save my build time. When I make a change without touching DSC/DE= C/FDF, it is absolutely much faster than original rebuild, 35%~50% time sav= ing in my environment (compare to original tool rebuild time). If I change any DSC/DEC/FDF, there's no performance improve, because it can= 't skip any module's AutoGen. Please note that if your environment will generate DSC/FDF during prebuild,= it will not skip any AutoGen because of DSC timestamp is changed. This wil= l require prebuild script not to update meta file when content is not chang= ed. Contributed-under: TianoCore Contribution Agreement 1.0 Signed-off-by: Derek Lin --- BaseTools/Source/Python/AutoGen/AutoGen.py | 110 +++++++++++++++++= ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 3 + BaseTools/Source/Python/GenFds/FdfParser.py | 4 + .../Source/Python/Workspace/MetaFileParser.py | 4 + 4 files changed, 121 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index f35ae25..daa5b35 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -42,6 +42,7 @@ from GenPcdDb import CreatePcdDatabaseCode from Workspac= e.MetaFileCommentParser import UsageList from Common.MultipleWorkspace imp= ort MultipleWorkspace as mws import InfSectionParser +import datetime =20 ## Regular expression for splitting Dependency Expression string into toke= ns gDepexTokenPattern =3D re.compile("(\(|\)|\w+| \S+\.inf)") @@ -640,6 +6= 41,31 @@ class WorkspaceAutoGen(AutoGen): self._MakeFileDir =3D None self._BuildCommand =3D None =20 + # + # Create list of metafiles of the workspace + # + AllWorkSpaceMetaFiles =3D self._CreateMetaFileList() + =20 + # + # Retrieve latest modified time of all metafiles + # + SrcTimeStamp =3D 0 + for f in AllWorkSpaceMetaFiles: + if os.stat(f)[8] > SrcTimeStamp: + SrcTimeStamp =3D os.stat(f)[8] =20 + self._SrcTimeStamp =3D SrcTimeStamp + + # + # Write metafile list to build directory + # + AutoGenFilePath =3D os.path.join(self.BuildDir, 'AutoGen') + if os.path.exists (AutoGenFilePath): + os.remove(AutoGenFilePath) + if not os.path.exists(self.BuildDir): + os.makedirs(self.BuildDir) + with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file: + for f in AllWorkSpaceMetaFiles: + print >> file, f return True =20 def _BuildOptionPcdValueFormat(self, TokenSpaceGuidCName, TokenCName, = PcdDatumType, Value): @@ -668,6 +694,29 @@ class WorkspaceAutoGen(AutoGen): Value =3D '0' return Value =20 + def _CreateMetaFileList(self): + AllWorkSpaceMetaFiles =3D [] + # + # add fdf + # + AllWorkSpaceMetaFiles.append (self.FdfFile.Path) + if self.FdfFile: + FdfFiles =3D GlobalData.gFdfParser.GetAllIncludedFile() + for f in FdfFiles: + AllWorkSpaceMetaFiles.append (f.FileName) =20 + # + # add dsc & dec + # + for BuildData in self.BuildDatabase._CACHE_.values(): + if 'dsc' is BuildData.MetaFile.Ext: + AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path) + for filePath in BuildData._RawData.IncludedFiles: + AllWorkSpaceMetaFiles.append(filePath.Path) + if 'dec' is BuildData.MetaFile.Ext: + AllWorkSpaceMetaFiles.append(BuildData.MetaFile.Path) + + return AllWorkSpaceMetaFiles + =20 ## _CheckDuplicateInFV() method # # Check whether there is duplicate modules/files exist in FV section.= =20 @@ -2520,6 +2569,10 @@ class PlatformAutoGen(AutoGen): # to the [depex] section in module's inf file. # class ModuleAutoGen(AutoGen): + ## Cache the timestamps of metafiles of every module in a class variab= le + # + TimeDict =3D {} + =20 ## The real constructor of ModuleAutoGen # # This method is not supposed to be called by users of ModuleAutoGen.= It's @@ -2619,6 +2672,11 @@ class ModuleAutoGen(AutoGen): self._FinalBuildTargetList =3D None self._FileTypes =3D None self._BuildRules =3D None + + self._TimeStampPath =3D None + + self.AutoGenDepSet =3D set() + =20 =20 ## The Modules referenced to this Library # Only Library has this attribute @@ -3946,6 +4004,8 @@ class Mod= uleAutoGen(AutoGen): =20 if self.IsMakeFileCreated: return + if self.CanSkip(): + return =20 if not self.IsLibrary and CreateLibraryMakeFile: for LibraryAutoGen in self.LibraryAutoGenList: @@ -3962,6 +4022,7 @@ class ModuleAutoGen(AutoGen): EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = makefile for module %s [%s]" % (self.Name, self.Arch)) =20 + self.CreateTimeStamp(Makefile) self.IsMakeFileCreated =3D True =20 def CopyBinaryFiles(self): @@ -3977,6 +4038,8 @@ class ModuleAutoGen(AutoGen): def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): if self.IsCodeFileCreated: return + if self.CanSkip(): + return =20 # Need to generate PcdDatabase even PcdDriver is binarymodule if self.IsBinaryModule and self.PcdIsDriver !=3D '': @@ -4056,6 +4119,53 @@ class ModuleAutoGen(AutoGen): self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE) return self._LibraryAutoGenList =20 + ## Decide whether we can skip the ModuleAutoGen process =20 + # If any source file is newer than the modeule than we cannot skip + # + def CanSkip(self): + if not os.path.exists(self.GetTimeStampPath()): + return False + #last creation time of the module + DstTimeStamp =3D os.stat(self.GetTimeStampPath())[8] + =20 + SrcTimeStamp =3D self.Workspace._SrcTimeStamp + if SrcTimeStamp > DstTimeStamp: + return False + + with open(self.GetTimeStampPath(),'r') as f: + for source in f: + source =3D source.rstrip('\n') + if source not in ModuleAutoGen.TimeDict : + ModuleAutoGen.TimeDict[source] =3D os.stat(source)[8]= =20 + if ModuleAutoGen.TimeDict[source] > DstTimeStamp: + return False + return True =20 + + def GetTimeStampPath(self): + if self._TimeStampPath =3D=3D None: + self._TimeStampPath =3D os.path.join(self.MakeFileDir, 'AutoGe= nTimeStamp') + return self._TimeStampPath + def CreateTimeStamp(self, Makefile): + + FileSet =3D set() + + FileSet.add (self.MetaFile.Path) + + for SourceFile in self.Module.Sources: + FileSet.add (SourceFile.Path) + + for Lib in self.DependentLibraryList: + FileSet.add (Lib.MetaFile.Path) + + for f in self.AutoGenDepSet: + FileSet.add (f.Path) + + if os.path.exists (self.GetTimeStampPath()): + os.remove (self.GetTimeStampPath()) + with open(self.GetTimeStampPath(), 'w+') as file: + for f in FileSet: + print >> file, f + Module =3D property(_GetModule) Name =3D property(_GetBaseName) Guid =3D property(_GetGuid) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 51c5238..ea07b97 100644 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -801,6 +801,9 @@ cleanlib: if not self.FileDependency[File]: self.FileDependency[File] =3D ['$(FORCE_REBUILD)'] continue + + self._AutoGenObject.AutoGenDepSet |=3D=20 + set(self.FileDependency[File]) + # skip non-C files if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoGen.c= ": continue diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source= /Python/GenFds/FdfParser.py index 2900283..1093ebb 100644 --- a/BaseTools/Source/Python/GenFds/FdfParser.py +++ b/BaseTools/Source/Python/GenFds/FdfParser.py @@ -4781,6 +4781,10 @@ class FdfParser: =20 return False =20 + def GetAllIncludedFile (self): + global AllIncludeFileList + return AllIncludeFileList + if __name__ =3D=3D "__main__": import sys try: diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTool= s/Source/Python/Workspace/MetaFileParser.py index 1a5fdf5..d0ab5e9 100644 --- a/BaseTools/Source/Python/Workspace/MetaFileParser.py +++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py @@ -852,6 +852,8 @@ class DscParser(MetaFileParser): =20 SymbolPattern =3D ValueExpression.SymbolPattern =20 + IncludedFiles =3D set() + ## Constructor of DscParser # # Initialize object of DscParser @@ -1494,6 +1496,8 @@ class DscParser(MetaFileParser): Parser =3D DscParser(IncludedFile1, self._FileType, self._Arch= , IncludedFileTable, Owner=3DOwner, From=3DOwner) =20 + self.IncludedFiles.add (IncludedFile1) + # Does not allow lower level included file to include upper le= vel included file if Parser._From !=3D Owner and int(Owner) > int (Parser._From)= : EdkLogger.error('parser', FILE_ALREADY_EXIST, File=3Dself.= _FileWithError, -- 2.7.4.windows.1