From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: liming.gao@intel.com) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by groups.io with SMTP; Mon, 09 Sep 2019 00:15:47 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Sep 2019 00:15:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,484,1559545200"; d="scan'208";a="196107852" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by orsmga002.jf.intel.com with ESMTP; 09 Sep 2019 00:15:45 -0700 Received: from fmsmsx162.amr.corp.intel.com (10.18.125.71) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 9 Sep 2019 00:15:44 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by fmsmsx162.amr.corp.intel.com (10.18.125.71) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 9 Sep 2019 00:15:44 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.32]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.140]) with mapi id 14.03.0439.000; Mon, 9 Sep 2019 15:15:42 +0800 From: "Liming Gao" To: "devel@edk2.groups.io" , "Gao, Liming" , "Feng, Bob C" CC: "Shi, Steven" Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Topic: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Index: AQHVY8kIs0aCbMr88kSF/66xSaGVcqcciT6AgACQx4CAAAQscIAF0kgAgAAEsNA= Date: Mon, 9 Sep 2019 07:15:41 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E4F4965@SHSMSX104.ccr.corp.intel.com> References: <20190905090458.22976-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E4F3352@SHSMSX104.ccr.corp.intel.com> <15C18FCC38905317.10639@groups.io> <08650203BA1BD64D8AD9B6D5D74A85D16153133C@SHSMSX104.ccr.corp.intel.com> <15C2B2E9C55F0B32.16248@groups.io> In-Reply-To: <15C2B2E9C55F0B32.16248@groups.io> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: liming.gao@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sorry, should be Acked-by: Liming Gao >-----Original Message----- >From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of >Liming Gao >Sent: Monday, September 09, 2019 2:59 PM >To: Feng, Bob C ; devel@edk2.groups.io >Cc: Shi, Steven >Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >IgnoreAutoGen > >Liming Gao > >>-----Original Message----- >>From: Feng, Bob C >>Sent: Thursday, September 05, 2019 10:06 PM >>To: devel@edk2.groups.io; Feng, Bob C ; Gao, >Liming >> >>Cc: Shi, Steven >>Subject: RE: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >>IgnoreAutoGen >> >>Sorry for the incomplete mail. >> >>I verified this patch on Ovmf, Emulator, MinPlatforms and Intel Server >>Platform. >> >>Thanks, >>Bob >> >>-----Original Message----- >>From: devel@edk2.groups.io On Behalf Of Bob >Feng >>Sent: Thursday, September 5, 2019 10:04 PM >>To: Gao, Liming ; devel@edk2.groups.io >>Cc: Shi, Steven >>Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >>IgnoreAutoGen >> >>Liming, >> >>Yes, this change only impact -u option. >> >>I separate _MultiThreadBuildPlatform function into two functions and cre= ate >2 >>new functions specially for IgnoreAutoGen, so the there looks big chang= e >for >>build.py in this patch. I verified this patch on Ovmf, E >> >>-Bob >> >>-----Original Message----- >>From: Gao, Liming >>Sent: Thursday, September 5, 2019 9:12 PM >>To: devel@edk2.groups.io; Feng, Bob C >>Cc: Shi, Steven >>Subject: RE: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >>IgnoreAutoGen >> >>Bob: >> Does this change only impact -u option behavior? >> >>Thanks >>Liming >>> -----Original Message----- >>> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of >>> Bob Feng >>> Sent: Thursday, September 5, 2019 5:05 PM >>> To: devel@edk2.groups.io >>> Cc: Gao, Liming ; Shi, Steven >>> ; Feng, Bob C >>> Subject: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >>> IgnoreAutoGen >>> >>> https://bugzilla.tianocore.org/show_bug.cgi?id=3D2080 >>> >>> This patch is to improve build -u option to re-use >>> GlobalVar__.bin file which is introduced by >>> multiple-process-autogen feature. >>> >>> Cc: Liming Gao >>> Cc: Steven Shi >>> Signed-off-by: Bob Feng >>> --- >>> .../Source/Python/AutoGen/AutoGenWorker.py | 2 +- >>> .../Source/Python/AutoGen/ModuleAutoGen.py | 4 +- >>> .../Python/AutoGen/ModuleAutoGenHelper.py | 29 ++ >>> .../Source/Python/AutoGen/PlatformAutoGen.py | 2 + >>> .../Source/Python/AutoGen/WorkspaceAutoGen.py | 1 - >>> BaseTools/Source/Python/build/build.py | 355 +++++++++++------= - >>> 6 files changed, 255 insertions(+), 138 deletions(-) >>> >>> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >>> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >>> index 2e68538b1cb4..f488ae9d5f80 100755 >>> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >>> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >>> @@ -236,11 +236,11 @@ class AutoGenWorkerInProcess(mp.Process): >>> Ma.ReferenceModules =3D >>Refes[(Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)] >>> if GlobalData.gBinCacheSource and CommandTarget in [N= one, "", >>"all"]: >>> Ma.GenModuleFilesHash(GlobalData.gCacheIR) >>> Ma.GenPreMakefileHash(GlobalData.gCacheIR) >>> if Ma.CanSkipbyPreMakefileCache(GlobalData.gCache= IR): >>> - continue >>> + continue >>> >>> Ma.CreateCodeFile(False) >>> >>> Ma.CreateMakeFile(False,GenFfsList=3DFfsCmd.get((Ma.MetaFile.File, >>> Ma.Arch),[])) >>> >>> if GlobalData.gBinCacheSource and CommandTarget in [N= one, "", >>"all"]: >>> diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >>> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >>> index 3bb7e91154ac..5f28681e3146 100755 >>> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >>> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >>> @@ -2058,12 +2058,12 @@ class ModuleAutoGen(AutoGen): >>> not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHash= Digest: >>> self.GenModuleFilesHash(gDict) >>> >>> if not (self.MetaFile.Path, self.Arch) in gDict or \ >>> not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHash= Digest: >>> - EdkLogger.quiet("[cache warning]: Cannot generate >>ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, >>> self.Arch)) >>> - return >>> + EdkLogger.quiet("[cache warning]: Cannot generate >>> + ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, >>> self.Arch)) >>> + return >>> >>> # Initialze hash object >>> m =3D hashlib.md5() >>> >>> # Add Platform level hash >>> diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >>> b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >>> index c7591253debd..9dd93b9beb12 100644 >>> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >>> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >>> @@ -234,22 +234,51 @@ class AutoGenInfo(object): >>> # >>> #This class is the pruned WorkSpaceAutoGen for ModuleAutoGen in >>> multiple thread # class WorkSpaceInfo(AutoGenInfo): >>> def __init__(self,Workspace, MetaFile, Target, ToolChain, Arch): >>> + if not hasattr(self, "_Init"): >>> + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch= ) >>> + self._Init =3D True >>> + def do_init(self,Workspace, MetaFile, Target, ToolChain, Arch): >>> self._SrcTimeStamp =3D 0 >>> self.Db =3D BuildDB >>> self.BuildDatabase =3D self.Db.BuildObject >>> self.Target =3D Target >>> self.ToolChain =3D ToolChain >>> self.WorkspaceDir =3D Workspace >>> self.ActivePlatform =3D MetaFile >>> self.ArchList =3D Arch >>> + self.AutoGenObjectList =3D [] >>> + @property >>> + def BuildDir(self): >>> + return self.AutoGenObjectList[0].BuildDir >>> >>> + @property >>> + def Name(self): >>> + return self.AutoGenObjectList[0].Platform.PlatformName >>> + >>> + @property >>> + def FlashDefinition(self): >>> + return self.AutoGenObjectList[0].Platform.FlashDefinition >>> + @property >>> + def GenFdsCommandDict(self): >>> + FdsCommandDict =3D >>self.AutoGenObjectList[0].DataPipe.Get("FdsCommandDict") >>> + if FdsCommandDict: >>> + return FdsCommandDict >>> + return {} >>> + >>> + @cached_property >>> + def FvDir(self): >>> + return os.path.join(self.BuildDir, TAB_FV_DIRECTORY) >>> >>> class PlatformInfo(AutoGenInfo): >>> def __init__(self, Workspace, MetaFile, Target, ToolChain, >Arch,DataPipe): >>> + if not hasattr(self, "_Init"): >>> + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch= ,DataPipe) >>> + self._Init =3D True >>> + def do_init(self,Workspace, MetaFile, Target, ToolChain, >Arch,DataPipe): >>> self.Wa =3D Workspace >>> self.WorkspaceDir =3D self.Wa.WorkspaceDir >>> self.MetaFile =3D MetaFile >>> self.Arch =3D Arch >>> self.Target =3D Target >>> diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >>> b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >>> index 565424a95ead..1e17b6687129 100644 >>> --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >>> +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >>> @@ -186,11 +186,13 @@ class PlatformAutoGen(AutoGen): >>> key =3D (Ma.MetaFile.File, self.Arch) >>> if key in FfsCommand: >>> Ma.CreateMakeFile(CreateModuleMakeFile, FfsComman= d[key]) >>> else: >>> Ma.CreateMakeFile(CreateModuleMakeFile) >>> + self.CreateLibModuelDirs() >>> >>> + def CreateLibModuelDirs(self): >>> # no need to create makefile for the platform more than once >>> if self.IsMakeFileCreated: >>> return >>> >>> # create library/module build dirs for platform diff --git >>> a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >>> b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >>> index 24942674721f..f9d2c216169c 100644 >>> --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >>> +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >>> @@ -126,11 +126,10 @@ class WorkspaceAutoGen(AutoGen): >>> self._CheckPcdDefineAndType() >>> >>> self.CreateBuildOptionsFile() >>> self.CreatePcdTokenNumberFile() >>> self.CreateModuleHashInfo() >>> - GlobalData.gAutoGenPhase =3D False >>> >>> # >>> # Merge Arch >>> # >>> def MergeArch(self): >>> diff --git a/BaseTools/Source/Python/build/build.py >>> b/BaseTools/Source/Python/build/build.py >>> index 0406ac314b65..db57c3e526da 100755 >>> --- a/BaseTools/Source/Python/build/build.py >>> +++ b/BaseTools/Source/Python/build/build.py >>> @@ -52,10 +52,13 @@ from PatchPcdValue.PatchPcdValue import >>> PatchBinaryFile >>> >>> import Common.GlobalData as GlobalData from GenFds.GenFds import >>> GenFds, GenFdsApi import multiprocessing as mp from multiprocessing >>> import Manager >>> +from AutoGen.DataPipe import MemoryDataPipe from >>> +AutoGen.ModuleAutoGenHelper import WorkSpaceInfo, PlatformInfo >>from >>> +GenFds.FdfParser import FdfParser >>> >>> >>> ## standard targets of build command >>> gSupportedTarget =3D ['all', 'genc', 'genmake', 'modules', 'libraries= ', >>> 'fds', 'clean', 'cleanall', 'cleanlib', 'run'] >>> >>> @@ -837,11 +840,11 @@ class Build(): >>> for PcdMa in PcdMaList: >>> if GlobalData.gBinCacheSource and self.Target in = [None, "", "all"]: >>> PcdMa.GenModuleFilesHash(share_data) >>> PcdMa.GenPreMakefileHash(share_data) >>> if PcdMa.CanSkipbyPreMakefileCache(share_data= ): >>> - continue >>> + continue >>> >>> PcdMa.CreateCodeFile(False) >>> PcdMa.CreateMakeFile(False,GenFfsList =3D >>> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.File, >>> PcdMa.Arch),[])) >>> >>> if GlobalData.gBinCacheSource and self.Target in = [None, "", "all"]: >>> @@ -1218,18 +1221,28 @@ class Build(): >>> mqueue =3D mp.Queue() >>> for m in AutoGenObject.GetAllModuleInfo: >>> mqueue.put(m) >>> >>> AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget"= : >>> self.Target} >>> + AutoGenObject.DataPipe.DataContainer =3D >{"Workspace_timestamp": >>AutoGenObject.Workspace._SrcTimeStamp} >>> + AutoGenObject.CreateLibModuelDirs() >>> + AutoGenObject.DataPipe.DataContainer =3D >>{"LibraryBuildDirectoryList":AutoGenObject.LibraryBuildDirectoryList} >>> + AutoGenObject.DataPipe.DataContainer =3D >>{"ModuleBuildDirectoryList":AutoGenObject.ModuleBuildDirectoryList} >>> + AutoGenObject.DataPipe.DataContainer =3D {"FdsCommandDict= ": >>> + AutoGenObject.Workspace.GenFdsCommandDict} >>> self.Progress.Start("Generating makefile and code") >>> data_pipe_file =3D os.path.join(AutoGenObject.BuildDir, >>> "GlobalVar_%s_%s.bin" % >>> (str(AutoGenObject.Guid),AutoGenObject.Arch)) >>> AutoGenObject.DataPipe.dump(data_pipe_file) >>> autogen_rt,errorcode =3D self.StartAutoGen(mqueue, >>> AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList, >>> GlobalData.gCacheIR) >>> + AutoGenIdFile =3D >>os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >>> + with open(AutoGenIdFile,"w") as fw: >>> + fw.write("Arch=3D%s\n" % >>"|".join((AutoGenObject.Workspace.ArchList))) >>> + fw.write("BuildDir=3D%s\n" % AutoGenObject.Workspace.= BuildDir) >>> + fw.write("PlatformGuid=3D%s\n" % >>> + str(AutoGenObject.Guid)) >>> self.Progress.Stop("done!") >>> if not autogen_rt: >>> self.AutoGenMgr.TerminateWorkers() >>> - self.AutoGenMgr.join(0.1) >>> + self.AutoGenMgr.join(1) >>> raise FatalError(errorcode) >>> AutoGenObject.CreateCodeFile(False) >>> AutoGenObject.CreateMakeFile(False) >>> else: >>> # always recreate top/platform makefile when clean, just >>> in case of inconsistency @@ -1778,10 +1791,11 @@ class Build(): >>> self._CollectFvMapBuffer(MapBuffer, Wa, Modul= eList) >>> # >>> # Save MAP buffer into MAP file. >>> # >>> self._SaveMapFile (MapBuffer, Wa) >>> + self.CreateGuidedSectionToolsFile(Wa) >>> >>> ## Build active module for different build targets, different too= l chains >>and different archs >>> # >>> def _BuildModule(self): >>> for BuildTarget in self.BuildTargetList: >>> @@ -1852,13 +1866,13 @@ class Build(): >>> >>> if GlobalData.gBinCacheSource and self.Ta= rget in [None, "", >>"all"]: >>> Ma.GenModuleFilesHash(GlobalData.gCac= heIR) >>> Ma.GenPreMakefileHash(GlobalData.gCac= heIR) >>> if Ma.CanSkipbyPreMakefileCache(Globa= lData.gCacheIR): >>> - self.HashSkipModules.append(Ma) >>> - EdkLogger.quiet("cache hit: %s[%s]= " % >(Ma.MetaFile.Path, >>Ma.Arch)) >>> - continue >>> + self.HashSkipModules.append(Ma) >>> + EdkLogger.quiet("cache hit: %s[%s= ]" % >>(Ma.MetaFile.Path, Ma.Arch)) >>> + continue >>> >>> # Not to auto-gen for targets 'clean', 'c= leanlib', 'cleanall', 'run', >>'fds' >>> if self.Target not in ['clean', 'cleanlib= ', 'cleanall', 'run', 'fds']: >>> # for target which must generate Auto= Gen code and >>makefile >>> if not self.SkipAutoGen or self.Targe= t =3D=3D 'genc': >>> @@ -1987,125 +2001,221 @@ class Build(): >>> GenFfsDict =3D GenFds.GenFfsMakefile('', GlobalData.gFdfParse= r, self, >>ArchList, GlobalData) >>> for Cmd in GenFfsDict: >>> tmpInf, tmpArch =3D GenFfsDict[Cmd] >>> CmdSetDict[tmpInf, tmpArch].add(Cmd) >>> return CmdSetDict >>> + def VerifyAutoGenFiles(self): >>> + AutoGenIdFile =3D >>os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >>> + try: >>> + with open(AutoGenIdFile) as fd: >>> + lines =3D fd.readlines() >>> + except: >>> + return None >>> + for line in lines: >>> + if "Arch" in line: >>> + ArchList =3D line.strip().split("=3D")[1].split("|") >>> + if "BuildDir" in line: >>> + BuildDir =3D line.split("=3D")[1].strip() >>> + if "PlatformGuid" in line: >>> + PlatformGuid =3D line.split("=3D")[1].strip() >>> + GlobalVarList =3D [] >>> + for arch in ArchList: >>> + global_var =3D os.path.join(BuildDir, "GlobalVar_%s_%s.bi= n" % >>(str(PlatformGuid),arch)) >>> + if not os.path.exists(global_var): >>> + return None >>> + GlobalVarList.append(global_var) >>> + for global_var in GlobalVarList: >>> + data_pipe =3D MemoryDataPipe() >>> + data_pipe.load(global_var) >>> + target =3D data_pipe.Get("P_Info").get("Target") >>> + toolchain =3D data_pipe.Get("P_Info").get("ToolChain") >>> + archlist =3D data_pipe.Get("P_Info").get("ArchList") >>> + Arch =3D data_pipe.Get("P_Info").get("Arch") >>> + active_p =3D data_pipe.Get("P_Info").get("ActivePlatform"= ) >>> + workspacedir =3D data_pipe.Get("P_Info").get("WorkspaceDi= r") >>> + PackagesPath =3D os.getenv("PACKAGES_PATH") >>> + mws.setWs(workspacedir, PackagesPath) >>> + LibraryBuildDirectoryList =3D >data_pipe.Get("LibraryBuildDirectoryList") >>> + ModuleBuildDirectoryList =3D >>> + data_pipe.Get("ModuleBuildDirectoryList") >>> + >>> + for m_build_dir in LibraryBuildDirectoryList: >>> + if not >>os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[G >e >>nMake.gMakeType])): >>> + return None >>> + for m_build_dir in ModuleBuildDirectoryList: >>> + if not >>os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[G >e >>nMake.gMakeType])): >>> + return None >>> + Wa =3D WorkSpaceInfo( >>> + workspacedir,active_p,target,toolchain,archlist >>> + ) >>> + Pa =3D PlatformInfo(Wa, active_p, target, toolchain, Arch= ,data_pipe) >>> + Wa.AutoGenObjectList.append(Pa) >>> + return Wa >>> + def SetupMakeSetting(self,Wa): >>> + BuildModules =3D [] >>> + for Pa in Wa.AutoGenObjectList: >>> + for m in Pa._MbList: >>> + ma =3D ModuleAutoGen(Wa,m.MetaFile, Pa.BuildTarget, >>Wa.ToolChain, Pa.Arch, Pa.MetaFile,Pa.DataPipe) >>> + BuildModules.append(ma) >>> + fdf_file =3D Wa.FlashDefinition >>> + if fdf_file: >>> + Fdf =3D FdfParser(fdf_file.Path) >>> + Fdf.ParseFile() >>> + GlobalData.gFdfParser =3D Fdf >>> + if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile= .FdDict: >>> + FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] >>> + for FdRegion in FdDict.RegionList: >>> + if str(FdRegion.RegionType) is 'FILE' and >>self.Platform.VpdToolGuid in str(FdRegion.RegionDataList): >>> + if int(FdRegion.Offset) % 8 !=3D 0: >>> + EdkLogger.error("build", FORMAT_INVALID, >>> + 'The VPD Base Address %s must be 8-byte aligned.' % >>> (FdRegion.Offset)) >>> + Wa.FdfProfile =3D Fdf.Profile >>> + self.Fdf =3D Fdf >>> + else: >>> + self.Fdf =3D None >>> + return BuildModules >>> >>> ## Build a platform in multi-thread mode >>> # >>> - def _MultiThreadBuildPlatform(self): >>> - SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# F= ILE >>auto-generated\n', False) >>> - for BuildTarget in self.BuildTargetList: >>> - GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget >>> - index =3D 0 >>> - for ToolChain in self.ToolChainList: >>> - WorkspaceAutoGenTime =3D time.time() >>> - GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain >>> - GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolC= hain >>> - GlobalData.gGlobalDefines['FAMILY'] =3D self.ToolChai= nFamily[index] >>> - index +=3D 1 >>> - Wa =3D WorkspaceAutoGen( >>> - self.WorkspaceDir, >>> - self.PlatformFile, >>> - BuildTarget, >>> - ToolChain, >>> - self.ArchList, >>> - self.BuildDatabase, >>> - self.TargetTxt, >>> - self.ToolDef, >>> - self.Fdf, >>> - self.FdList, >>> - self.FvList, >>> - self.CapList, >>> - self.SkuId, >>> - self.UniFlag, >>> - self.Progress >>> - ) >>> - self.Fdf =3D Wa.FdfFile >>> - self.LoadFixAddress =3D Wa.Platform.LoadFixAddress >>> - self.BuildReport.AddPlatformReport(Wa) >>> - Wa.CreateMakeFile(False) >>> + def PerformAutoGen(self,BuildTarget,ToolChain): >>> + WorkspaceAutoGenTime =3D time.time() >>> + Wa =3D WorkspaceAutoGen( >>> + self.WorkspaceDir, >>> + self.PlatformFile, >>> + BuildTarget, >>> + ToolChain, >>> + self.ArchList, >>> + self.BuildDatabase, >>> + self.TargetTxt, >>> + self.ToolDef, >>> + self.Fdf, >>> + self.FdList, >>> + self.FvList, >>> + self.CapList, >>> + self.SkuId, >>> + self.UniFlag, >>> + self.Progress >>> + ) >>> + self.Fdf =3D Wa.FdfFile >>> + self.LoadFixAddress =3D Wa.Platform.LoadFixAddress >>> + self.BuildReport.AddPlatformReport(Wa) >>> + Wa.CreateMakeFile(False) >>> >>> # Add ffs build to makefile >>> - CmdListDict =3D {} >>> - if GlobalData.gEnableGenfdsMultiThread and self.Fdf: >>> - CmdListDict =3D self._GenFfsCmd(Wa.ArchList) >>> + CmdListDict =3D {} >>> + if GlobalData.gEnableGenfdsMultiThread and self.Fdf: >>> + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) >>> >>> - # Add Platform and Package level hash in share_data f= or module >>hash calculation later >>> - if GlobalData.gBinCacheSource or GlobalData.gBinCache= Dest: >>> - GlobalData.gCacheIR[('PlatformHash')] =3D >>GlobalData.gPlatformHash >>> - for PkgName in GlobalData.gPackageHash.keys(): >>> - GlobalData.gCacheIR[(PkgName, 'PackageHash')]= =3D >>GlobalData.gPackageHash[PkgName] >>> + # Add Platform and Package level hash in share_data for modul= e hash >>calculation later >>> + if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest: >>> + GlobalData.gCacheIR[('PlatformHash')] =3D GlobalData.gPla= tformHash >>> + for PkgName in GlobalData.gPackageHash.keys(): >>> + GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D >>> + GlobalData.gPackageHash[PkgName] >>> >>> - # multi-thread exit flag >>> - ExitFlag =3D threading.Event() >>> - ExitFlag.clear() >>> - self.AutoGenTime +=3D int(round((time.time() - >>WorkspaceAutoGenTime))) >>> - self.BuildModules =3D [] >>> - TotalModules =3D [] >>> - for Arch in Wa.ArchList: >>> - PcdMaList =3D [] >>> - AutoGenStart =3D time.time() >>> - GlobalData.gGlobalDefines['ARCH'] =3D Arch >>> - Pa =3D PlatformAutoGen(Wa, self.PlatformFile, Bui= ldTarget, >>ToolChain, Arch) >>> - if Pa is None: >>> - continue >>> - ModuleList =3D [] >>> - for Inf in Pa.Platform.Modules: >>> - ModuleList.append(Inf) >>> + self.AutoGenTime +=3D int(round((time.time() - >>WorkspaceAutoGenTime))) >>> + BuildModules =3D [] >>> + TotalModules =3D [] >>> + for Arch in Wa.ArchList: >>> + PcdMaList =3D [] >>> + AutoGenStart =3D time.time() >>> + GlobalData.gGlobalDefines['ARCH'] =3D Arch >>> + Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTarget= , ToolChain, >>Arch) >>> + if Pa is None: >>> + continue >>> + ModuleList =3D [] >>> + for Inf in Pa.Platform.Modules: >>> + ModuleList.append(Inf) >>> # Add the INF only list in FDF >>> - if GlobalData.gFdfParser is not None: >>> - for InfName in GlobalData.gFdfParser.Profile.= InfList: >>> - Inf =3D PathClass(NormPath(InfName), self= .WorkspaceDir, >Arch) >>> - if Inf in Pa.Platform.Modules: >>> - continue >>> - ModuleList.append(Inf) >>> - Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdLi= stDict} >>> - Pa.DataPipe.DataContainer =3D {"Workspace_timesta= mp": >>Wa._SrcTimeStamp} >>> - Pa.DataPipe.DataContainer =3D {"CommandTarget": s= elf.Target} >>> - for Module in ModuleList: >>> + if GlobalData.gFdfParser is not None: >>> + for InfName in GlobalData.gFdfParser.Profile.InfList: >>> + Inf =3D PathClass(NormPath(InfName), self.Workspa= ceDir, Arch) >>> + if Inf in Pa.Platform.Modules: >>> + continue >>> + ModuleList.append(Inf) >>> + Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDict} >>> + Pa.DataPipe.DataContainer =3D {"Workspace_timestamp": >>Wa._SrcTimeStamp} >>> + Pa.DataPipe.DataContainer =3D {"CommandTarget": self.Targ= et} >>> + Pa.CreateLibModuelDirs() >>> + Pa.DataPipe.DataContainer =3D >>{"LibraryBuildDirectoryList":Pa.LibraryBuildDirectoryList} >>> + Pa.DataPipe.DataContainer =3D >>{"ModuleBuildDirectoryList":Pa.ModuleBuildDirectoryList} >>> + Pa.DataPipe.DataContainer =3D {"FdsCommandDict": >>Wa.GenFdsCommandDict} >>> + ModuleCodaFile =3D {} >>> + for ma in Pa.ModuleAutoGenList: >>> + >>> + >ModuleCodaFile[(ma.MetaFile.File,ma.MetaFile.Root,ma.Arch,ma.MetaFil >>> + e.Path)] =3D [item.Target for item in >>> ma.CodaTargetList] >>> + Pa.DataPipe.DataContainer =3D {"ModuleCodaFile":ModuleCod= aFile} >>> + for Module in ModuleList: >>> # Get ModuleAutoGen object to generate C code= file and >>makefile >>> - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget,= ToolChain, >>Arch, self.PlatformFile,Pa.DataPipe) >>> + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, >>> + ToolChain, Arch, self.PlatformFile,Pa.DataPipe) >>> >>> - if Ma is None: >>> + if Ma is None: >>> + continue >>> + if Ma.PcdIsDriver: >>> + Ma.PlatformInfo =3D Pa >>> + Ma.Workspace =3D Wa >>> + PcdMaList.append(Ma) >>> + TotalModules.append(Ma) >>> + # Initialize all modules in tracking to 'FAIL' >>> + GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' >>> + >>> + >>> + mqueue =3D mp.Queue() >>> + for m in Pa.GetAllModuleInfo: >>> + mqueue.put(m) >>> + data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVar_%= s_%s.bin" % >>(str(Pa.Guid),Pa.Arch)) >>> + Pa.DataPipe.dump(data_pipe_file) >>> + >>> + autogen_rt, errorcode =3D self.StartAutoGen(mqueue, >>> + Pa.DataPipe, self.SkipAutoGen, PcdMaList,GlobalData.gCacheIR) >>> + >>> + # Skip cache hit modules >>> + if GlobalData.gBinCacheSource: >>> + for Ma in TotalModules: >>> + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCac= heIR and \ >>> + GlobalData.gCacheIR[(Ma.MetaFile.Path, >>Ma.Arch)].PreMakeCacheHit: >>> + self.HashSkipModules.append(Ma) >>> + continue >>> + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCac= heIR and \ >>> + GlobalData.gCacheIR[(Ma.MetaFile.Path, >>Ma.Arch)].MakeCacheHit: >>> + self.HashSkipModules.append(Ma) >>> continue >>> - if Ma.PcdIsDriver: >>> - Ma.PlatformInfo =3D Pa >>> - Ma.Workspace =3D Wa >>> - PcdMaList.append(Ma) >>> - TotalModules.append(Ma) >>> - # Initialize all modules in tracking to 'FAIL= ' >>> - GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL= ' >>> + BuildModules.append(Ma) >>> + else: >>> + BuildModules.extend(TotalModules) >>> >>> - mqueue =3D mp.Queue() >>> - for m in Pa.GetAllModuleInfo: >>> - mqueue.put(m) >>> - data_pipe_file =3D os.path.join(Pa.BuildDir, >>"GlobalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) >>> - Pa.DataPipe.dump(data_pipe_file) >>> - autogen_rt, errorcode =3D self.StartAutoGen(mqueu= e, >Pa.DataPipe, >>self.SkipAutoGen, PcdMaList, >>> GlobalData.gCacheIR) >>> + if not autogen_rt: >>> + self.AutoGenMgr.TerminateWorkers() >>> + self.AutoGenMgr.join(1) >>> + raise FatalError(errorcode) >>> + self.AutoGenTime +=3D int(round((time.time() - AutoGenSta= rt))) >>> + AutoGenIdFile =3D >>os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >>> + with open(AutoGenIdFile,"w") as fw: >>> + fw.write("Arch=3D%s\n" % "|".join((Wa.ArchList))) >>> + fw.write("BuildDir=3D%s\n" % Wa.BuildDir) >>> + fw.write("PlatformGuid=3D%s\n" % >>str(Wa.AutoGenObjectList[0].Guid)) >>> + self.Progress.Stop("done!") >>> + return Wa, BuildModules >>> >>> - # Skip cache hit modules >>> - if GlobalData.gBinCacheSource: >>> - for Ma in TotalModules: >>> - if (Ma.MetaFile.Path, Ma.Arch) in GlobalD= ata.gCacheIR and >\ >>> - GlobalData.gCacheIR[(Ma.MetaFile.Path= , >>Ma.Arch)].PreMakeCacheHit: >>> - self.HashSkipModules.append(Ma) >>> - continue >>> - if (Ma.MetaFile.Path, Ma.Arch) in GlobalD= ata.gCacheIR and >\ >>> - GlobalData.gCacheIR[(Ma.MetaFile.Path= , >>Ma.Arch)].MakeCacheHit: >>> - self.HashSkipModules.append(Ma) >>> - continue >>> - self.BuildModules.append(Ma) >>> + def _MultiThreadBuildPlatform(self): >>> + SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# F= ILE >>auto-generated\n', False) >>> + for BuildTarget in self.BuildTargetList: >>> + GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget >>> + index =3D 0 >>> + for ToolChain in self.ToolChainList: >>> + GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain >>> + GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolC= hain >>> + GlobalData.gGlobalDefines['FAMILY'] =3D >self.ToolChainFamily[index] >>> + index +=3D 1 >>> + ExitFlag =3D threading.Event() >>> + ExitFlag.clear() >>> + if self.SkipAutoGen: >>> + Wa =3D self.VerifyAutoGenFiles() >>> + if Wa is None: >>> + Wa, self.BuildModules =3D >>> + self.PerformAutoGen(BuildTarget,ToolChain) >>> else: >>> - self.BuildModules.extend(TotalModules) >>> - >>> - if not autogen_rt: >>> - self.AutoGenMgr.TerminateWorkers() >>> - self.AutoGenMgr.join(0.1) >>> - raise FatalError(errorcode) >>> - self.AutoGenTime +=3D int(round((time.time() - AutoGe= nStart))) >>> - self.Progress.Stop("done!") >>> + GlobalData.gAutoGenPhase =3D True >>> + self.BuildModules =3D self.SetupMakeSetting(W= a) >>> + else: >>> + Wa, self.BuildModules =3D >>self.PerformAutoGen(BuildTarget,ToolChain) >>> + Pa =3D Wa.AutoGenObjectList[0] >>> + GlobalData.gAutoGenPhase =3D False >>> >>> if GlobalData.gBinCacheSource: >>> EdkLogger.quiet("Total cache hit driver num: %s, >>> cache miss driver num: %s" % (len(set(self.HashSkipModules)), >>> len(set(self.BuildModules)))) >>> CacheHitMa =3D set() >>> CacheNotHitMa =3D set() @@ -2136,18 +2246,17 @@ >>> class Build(): >>> # Start task scheduler >>> if not BuildTask.IsOnGoing(): >>> >>> BuildTask.StartScheduler(self.ThreadNumber, ExitFlag) >>> >>> # in case there's an interruption. we need a full= version of >>makefile for platform >>> - Pa.CreateMakeFile(False) >>> + >>> if BuildTask.HasError(): >>> self.invalidateHash() >>> EdkLogger.error("build", BUILD_ERROR, "Failed= to build >>module", ExtraData=3DGlobalData.gBuildingModule) >>> self.MakeTime +=3D int(round((time.time() - >>> MakeStart))) >>> >>> MakeContiue =3D time.time() >>> - >>> # >>> # >>> # All modules have been put in build tasks queue. Tel= l task >>scheduler >>> # to exit if all tasks are completed >>> # >>> @@ -2175,17 +2284,12 @@ class Build(): >>> if (Arch =3D=3D 'IA32' or Arch =3D=3D 'ARM') = and >>> self.LoadFixAddress !=3D 0xFFFFFFFFFFFFFFFF and self.LoadFixAddress >= =3D >>0x100000000: >>> EdkLogger.error("build", >>> PARAMETER_INVALID, "FIX_LOAD_TOP_MEMORY_ADDRESS can't be set >to >>larger than or equal to 4G for the platorm with IA32 or ARM arch modules= ") >>> # >>> # Get Module List >>> # >>> - ModuleList =3D {} >>> - for Pa in Wa.AutoGenObjectList: >>> - for Ma in Pa.ModuleAutoGenList: >>> - if Ma is None: >>> - continue >>> - if not Ma.IsLibrary: >>> - ModuleList[Ma.Guid.upper()] =3D Ma >>> + ModuleList =3D {ma.Guid.upper():ma for ma in >>> + self.BuildModules} >>> + >>> # >>> # Rebase module to the preferred memory address b= efore >>GenFds >>> # >>> MapBuffer =3D [] >>> if self.LoadFixAddress !=3D 0: >>> @@ -2206,33 +2310,17 @@ class Build(): >>> self.GenFdsTime +=3D int(round((time.time() -= GenFdsStart))) >>> # >>> # Save MAP buffer into MAP file. >>> # >>> self._SaveMapFile(MapBuffer, Wa) >>> + self.CreateGuidedSectionToolsFile(Wa) >>> self.invalidateHash() >>> - >>> ## Generate GuidedSectionTools.txt in the FV directories. >>> # >>> - def CreateGuidedSectionToolsFile(self): >>> + def CreateGuidedSectionToolsFile(self,Wa): >>> for BuildTarget in self.BuildTargetList: >>> for ToolChain in self.ToolChainList: >>> - Wa =3D WorkspaceAutoGen( >>> - self.WorkspaceDir, >>> - self.PlatformFile, >>> - BuildTarget, >>> - ToolChain, >>> - self.ArchList, >>> - self.BuildDatabase, >>> - self.TargetTxt, >>> - self.ToolDef, >>> - self.Fdf, >>> - self.FdList, >>> - self.FvList, >>> - self.CapList, >>> - self.SkuId, >>> - self.UniFlag >>> - ) >>> FvDir =3D Wa.FvDir >>> if not os.path.exists(FvDir): >>> continue >>> >>> for Arch in self.ArchList: >>> @@ -2285,11 +2373,10 @@ class Build(): >>> if not self.SpawnMode or self.Target not in ["", "all"]: >>> self.SpawnMode =3D False >>> self._BuildPlatform() >>> else: >>> self._MultiThreadBuildPlatform() >>> - self.CreateGuidedSectionToolsFile() >>> else: >>> self.SpawnMode =3D False >>> self._BuildModule() >>> >>> if self.Target =3D=3D 'cleanall': >>> -- >>> 2.20.1.windows.1 >>> >>> >>> >> >> >> > > >