From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.100, mailfrom: liming.gao@intel.com) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by groups.io with SMTP; Sun, 08 Sep 2019 23:58:57 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Sep 2019 23:58:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,484,1559545200"; d="scan'208";a="267981307" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by orsmga001.jf.intel.com with ESMTP; 08 Sep 2019 23:58:55 -0700 Received: from FMSMSX110.amr.corp.intel.com (10.18.116.10) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 8 Sep 2019 23:58:55 -0700 Received: from shsmsx105.ccr.corp.intel.com (10.239.4.158) by fmsmsx110.amr.corp.intel.com (10.18.116.10) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 8 Sep 2019 23:58:55 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.32]) by SHSMSX105.ccr.corp.intel.com ([169.254.11.23]) with mapi id 14.03.0439.000; Mon, 9 Sep 2019 14:58:53 +0800 From: "Liming Gao" To: "Feng, Bob C" , "devel@edk2.groups.io" CC: "Shi, Steven" Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Topic: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Index: AQHVY8kIs0aCbMr88kSF/66xSaGVcqcciT6AgACQx4CAAAQscIAF0kgA Date: Mon, 9 Sep 2019 06:58:53 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E4F4923@SHSMSX104.ccr.corp.intel.com> References: <20190905090458.22976-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E4F3352@SHSMSX104.ccr.corp.intel.com> <15C18FCC38905317.10639@groups.io> <08650203BA1BD64D8AD9B6D5D74A85D16153133C@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D16153133C@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: liming.gao@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Liming Gao >-----Original Message----- >From: Feng, Bob C >Sent: Thursday, September 05, 2019 10:06 PM >To: devel@edk2.groups.io; Feng, Bob C ; Gao, Liming > >Cc: Shi, Steven >Subject: RE: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >IgnoreAutoGen > >Sorry for the incomplete mail. > >I verified this patch on Ovmf, Emulator, MinPlatforms and Intel Server >Platform. > >Thanks, >Bob > >-----Original Message----- >From: devel@edk2.groups.io On Behalf Of Bob Feng >Sent: Thursday, September 5, 2019 10:04 PM >To: Gao, Liming ; devel@edk2.groups.io >Cc: Shi, Steven >Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >IgnoreAutoGen > >Liming, > >Yes, this change only impact -u option. > >I separate _MultiThreadBuildPlatform function into two functions and crea= te 2 >new functions specially for IgnoreAutoGen, so the there looks big change= for >build.py in this patch. I verified this patch on Ovmf, E > >-Bob > >-----Original Message----- >From: Gao, Liming >Sent: Thursday, September 5, 2019 9:12 PM >To: devel@edk2.groups.io; Feng, Bob C >Cc: Shi, Steven >Subject: RE: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >IgnoreAutoGen > >Bob: > Does this change only impact -u option behavior? > >Thanks >Liming >> -----Original Message----- >> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of >> Bob Feng >> Sent: Thursday, September 5, 2019 5:05 PM >> To: devel@edk2.groups.io >> Cc: Gao, Liming ; Shi, Steven >> ; Feng, Bob C >> Subject: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for >> IgnoreAutoGen >> >> https://bugzilla.tianocore.org/show_bug.cgi?id=3D2080 >> >> This patch is to improve build -u option to re-use >> GlobalVar__.bin file which is introduced by >> multiple-process-autogen feature. >> >> Cc: Liming Gao >> Cc: Steven Shi >> Signed-off-by: Bob Feng >> --- >> .../Source/Python/AutoGen/AutoGenWorker.py | 2 +- >> .../Source/Python/AutoGen/ModuleAutoGen.py | 4 +- >> .../Python/AutoGen/ModuleAutoGenHelper.py | 29 ++ >> .../Source/Python/AutoGen/PlatformAutoGen.py | 2 + >> .../Source/Python/AutoGen/WorkspaceAutoGen.py | 1 - >> BaseTools/Source/Python/build/build.py | 355 +++++++++++------- >> 6 files changed, 255 insertions(+), 138 deletions(-) >> >> diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >> b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >> index 2e68538b1cb4..f488ae9d5f80 100755 >> --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >> +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py >> @@ -236,11 +236,11 @@ class AutoGenWorkerInProcess(mp.Process): >> Ma.ReferenceModules =3D >Refes[(Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)] >> if GlobalData.gBinCacheSource and CommandTarget in [No= ne, "", >"all"]: >> Ma.GenModuleFilesHash(GlobalData.gCacheIR) >> Ma.GenPreMakefileHash(GlobalData.gCacheIR) >> if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheI= R): >> - continue >> + continue >> >> Ma.CreateCodeFile(False) >> >> Ma.CreateMakeFile(False,GenFfsList=3DFfsCmd.get((Ma.MetaFile.File, >> Ma.Arch),[])) >> >> if GlobalData.gBinCacheSource and CommandTarget in [No= ne, "", >"all"]: >> diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >> index 3bb7e91154ac..5f28681e3146 100755 >> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py >> @@ -2058,12 +2058,12 @@ class ModuleAutoGen(AutoGen): >> not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashD= igest: >> self.GenModuleFilesHash(gDict) >> >> if not (self.MetaFile.Path, self.Arch) in gDict or \ >> not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashD= igest: >> - EdkLogger.quiet("[cache warning]: Cannot generate >ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, >> self.Arch)) >> - return >> + EdkLogger.quiet("[cache warning]: Cannot generate >> + ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, >> self.Arch)) >> + return >> >> # Initialze hash object >> m =3D hashlib.md5() >> >> # Add Platform level hash >> diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >> b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >> index c7591253debd..9dd93b9beb12 100644 >> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py >> @@ -234,22 +234,51 @@ class AutoGenInfo(object): >> # >> #This class is the pruned WorkSpaceAutoGen for ModuleAutoGen in >> multiple thread # class WorkSpaceInfo(AutoGenInfo): >> def __init__(self,Workspace, MetaFile, Target, ToolChain, Arch): >> + if not hasattr(self, "_Init"): >> + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch) >> + self._Init =3D True >> + def do_init(self,Workspace, MetaFile, Target, ToolChain, Arch): >> self._SrcTimeStamp =3D 0 >> self.Db =3D BuildDB >> self.BuildDatabase =3D self.Db.BuildObject >> self.Target =3D Target >> self.ToolChain =3D ToolChain >> self.WorkspaceDir =3D Workspace >> self.ActivePlatform =3D MetaFile >> self.ArchList =3D Arch >> + self.AutoGenObjectList =3D [] >> + @property >> + def BuildDir(self): >> + return self.AutoGenObjectList[0].BuildDir >> >> + @property >> + def Name(self): >> + return self.AutoGenObjectList[0].Platform.PlatformName >> + >> + @property >> + def FlashDefinition(self): >> + return self.AutoGenObjectList[0].Platform.FlashDefinition >> + @property >> + def GenFdsCommandDict(self): >> + FdsCommandDict =3D >self.AutoGenObjectList[0].DataPipe.Get("FdsCommandDict") >> + if FdsCommandDict: >> + return FdsCommandDict >> + return {} >> + >> + @cached_property >> + def FvDir(self): >> + return os.path.join(self.BuildDir, TAB_FV_DIRECTORY) >> >> class PlatformInfo(AutoGenInfo): >> def __init__(self, Workspace, MetaFile, Target, ToolChain, Arch,Da= taPipe): >> + if not hasattr(self, "_Init"): >> + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch,= DataPipe) >> + self._Init =3D True >> + def do_init(self,Workspace, MetaFile, Target, ToolChain, Arch,Data= Pipe): >> self.Wa =3D Workspace >> self.WorkspaceDir =3D self.Wa.WorkspaceDir >> self.MetaFile =3D MetaFile >> self.Arch =3D Arch >> self.Target =3D Target >> diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >> b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >> index 565424a95ead..1e17b6687129 100644 >> --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >> +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py >> @@ -186,11 +186,13 @@ class PlatformAutoGen(AutoGen): >> key =3D (Ma.MetaFile.File, self.Arch) >> if key in FfsCommand: >> Ma.CreateMakeFile(CreateModuleMakeFile, FfsCommand= [key]) >> else: >> Ma.CreateMakeFile(CreateModuleMakeFile) >> + self.CreateLibModuelDirs() >> >> + def CreateLibModuelDirs(self): >> # no need to create makefile for the platform more than once >> if self.IsMakeFileCreated: >> return >> >> # create library/module build dirs for platform diff --git >> a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >> b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >> index 24942674721f..f9d2c216169c 100644 >> --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >> +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py >> @@ -126,11 +126,10 @@ class WorkspaceAutoGen(AutoGen): >> self._CheckPcdDefineAndType() >> >> self.CreateBuildOptionsFile() >> self.CreatePcdTokenNumberFile() >> self.CreateModuleHashInfo() >> - GlobalData.gAutoGenPhase =3D False >> >> # >> # Merge Arch >> # >> def MergeArch(self): >> diff --git a/BaseTools/Source/Python/build/build.py >> b/BaseTools/Source/Python/build/build.py >> index 0406ac314b65..db57c3e526da 100755 >> --- a/BaseTools/Source/Python/build/build.py >> +++ b/BaseTools/Source/Python/build/build.py >> @@ -52,10 +52,13 @@ from PatchPcdValue.PatchPcdValue import >> PatchBinaryFile >> >> import Common.GlobalData as GlobalData from GenFds.GenFds import >> GenFds, GenFdsApi import multiprocessing as mp from multiprocessing >> import Manager >> +from AutoGen.DataPipe import MemoryDataPipe from >> +AutoGen.ModuleAutoGenHelper import WorkSpaceInfo, PlatformInfo >from >> +GenFds.FdfParser import FdfParser >> >> >> ## standard targets of build command >> gSupportedTarget =3D ['all', 'genc', 'genmake', 'modules', 'libraries'= , >> 'fds', 'clean', 'cleanall', 'cleanlib', 'run'] >> >> @@ -837,11 +840,11 @@ class Build(): >> for PcdMa in PcdMaList: >> if GlobalData.gBinCacheSource and self.Target in [= None, "", "all"]: >> PcdMa.GenModuleFilesHash(share_data) >> PcdMa.GenPreMakefileHash(share_data) >> if PcdMa.CanSkipbyPreMakefileCache(share_data)= : >> - continue >> + continue >> >> PcdMa.CreateCodeFile(False) >> PcdMa.CreateMakeFile(False,GenFfsList =3D >> DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.File, >> PcdMa.Arch),[])) >> >> if GlobalData.gBinCacheSource and self.Target in [= None, "", "all"]: >> @@ -1218,18 +1221,28 @@ class Build(): >> mqueue =3D mp.Queue() >> for m in AutoGenObject.GetAllModuleInfo: >> mqueue.put(m) >> >> AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget": >> self.Target} >> + AutoGenObject.DataPipe.DataContainer =3D {"Workspace_times= tamp": >AutoGenObject.Workspace._SrcTimeStamp} >> + AutoGenObject.CreateLibModuelDirs() >> + AutoGenObject.DataPipe.DataContainer =3D >{"LibraryBuildDirectoryList":AutoGenObject.LibraryBuildDirectoryList} >> + AutoGenObject.DataPipe.DataContainer =3D >{"ModuleBuildDirectoryList":AutoGenObject.ModuleBuildDirectoryList} >> + AutoGenObject.DataPipe.DataContainer =3D {"FdsCommandDict"= : >> + AutoGenObject.Workspace.GenFdsCommandDict} >> self.Progress.Start("Generating makefile and code") >> data_pipe_file =3D os.path.join(AutoGenObject.BuildDir, >> "GlobalVar_%s_%s.bin" % >> (str(AutoGenObject.Guid),AutoGenObject.Arch)) >> AutoGenObject.DataPipe.dump(data_pipe_file) >> autogen_rt,errorcode =3D self.StartAutoGen(mqueue, >> AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList, >> GlobalData.gCacheIR) >> + AutoGenIdFile =3D >os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >> + with open(AutoGenIdFile,"w") as fw: >> + fw.write("Arch=3D%s\n" % >"|".join((AutoGenObject.Workspace.ArchList))) >> + fw.write("BuildDir=3D%s\n" % AutoGenObject.Workspace.B= uildDir) >> + fw.write("PlatformGuid=3D%s\n" % >> + str(AutoGenObject.Guid)) >> self.Progress.Stop("done!") >> if not autogen_rt: >> self.AutoGenMgr.TerminateWorkers() >> - self.AutoGenMgr.join(0.1) >> + self.AutoGenMgr.join(1) >> raise FatalError(errorcode) >> AutoGenObject.CreateCodeFile(False) >> AutoGenObject.CreateMakeFile(False) >> else: >> # always recreate top/platform makefile when clean, just >> in case of inconsistency @@ -1778,10 +1791,11 @@ class Build(): >> self._CollectFvMapBuffer(MapBuffer, Wa, Module= List) >> # >> # Save MAP buffer into MAP file. >> # >> self._SaveMapFile (MapBuffer, Wa) >> + self.CreateGuidedSectionToolsFile(Wa) >> >> ## Build active module for different build targets, different tool= chains >and different archs >> # >> def _BuildModule(self): >> for BuildTarget in self.BuildTargetList: >> @@ -1852,13 +1866,13 @@ class Build(): >> >> if GlobalData.gBinCacheSource and self.Tar= get in [None, "", >"all"]: >> Ma.GenModuleFilesHash(GlobalData.gCach= eIR) >> Ma.GenPreMakefileHash(GlobalData.gCach= eIR) >> if Ma.CanSkipbyPreMakefileCache(Global= Data.gCacheIR): >> - self.HashSkipModules.append(Ma) >> - EdkLogger.quiet("cache hit: %s[%s]"= % (Ma.MetaFile.Path, >Ma.Arch)) >> - continue >> + self.HashSkipModules.append(Ma) >> + EdkLogger.quiet("cache hit: %s[%s]= " % >(Ma.MetaFile.Path, Ma.Arch)) >> + continue >> >> # Not to auto-gen for targets 'clean', 'cl= eanlib', 'cleanall', 'run', >'fds' >> if self.Target not in ['clean', 'cleanlib'= , 'cleanall', 'run', 'fds']: >> # for target which must generate AutoG= en code and >makefile >> if not self.SkipAutoGen or self.Target= =3D=3D 'genc': >> @@ -1987,125 +2001,221 @@ class Build(): >> GenFfsDict =3D GenFds.GenFfsMakefile('', GlobalData.gFdfParser= , self, >ArchList, GlobalData) >> for Cmd in GenFfsDict: >> tmpInf, tmpArch =3D GenFfsDict[Cmd] >> CmdSetDict[tmpInf, tmpArch].add(Cmd) >> return CmdSetDict >> + def VerifyAutoGenFiles(self): >> + AutoGenIdFile =3D >os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >> + try: >> + with open(AutoGenIdFile) as fd: >> + lines =3D fd.readlines() >> + except: >> + return None >> + for line in lines: >> + if "Arch" in line: >> + ArchList =3D line.strip().split("=3D")[1].split("|") >> + if "BuildDir" in line: >> + BuildDir =3D line.split("=3D")[1].strip() >> + if "PlatformGuid" in line: >> + PlatformGuid =3D line.split("=3D")[1].strip() >> + GlobalVarList =3D [] >> + for arch in ArchList: >> + global_var =3D os.path.join(BuildDir, "GlobalVar_%s_%s.bin= " % >(str(PlatformGuid),arch)) >> + if not os.path.exists(global_var): >> + return None >> + GlobalVarList.append(global_var) >> + for global_var in GlobalVarList: >> + data_pipe =3D MemoryDataPipe() >> + data_pipe.load(global_var) >> + target =3D data_pipe.Get("P_Info").get("Target") >> + toolchain =3D data_pipe.Get("P_Info").get("ToolChain") >> + archlist =3D data_pipe.Get("P_Info").get("ArchList") >> + Arch =3D data_pipe.Get("P_Info").get("Arch") >> + active_p =3D data_pipe.Get("P_Info").get("ActivePlatform") >> + workspacedir =3D data_pipe.Get("P_Info").get("WorkspaceDir= ") >> + PackagesPath =3D os.getenv("PACKAGES_PATH") >> + mws.setWs(workspacedir, PackagesPath) >> + LibraryBuildDirectoryList =3D data_pipe.Get("LibraryBuildD= irectoryList") >> + ModuleBuildDirectoryList =3D >> + data_pipe.Get("ModuleBuildDirectoryList") >> + >> + for m_build_dir in LibraryBuildDirectoryList: >> + if not >os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[Ge >nMake.gMakeType])): >> + return None >> + for m_build_dir in ModuleBuildDirectoryList: >> + if not >os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[Ge >nMake.gMakeType])): >> + return None >> + Wa =3D WorkSpaceInfo( >> + workspacedir,active_p,target,toolchain,archlist >> + ) >> + Pa =3D PlatformInfo(Wa, active_p, target, toolchain, Arch,= data_pipe) >> + Wa.AutoGenObjectList.append(Pa) >> + return Wa >> + def SetupMakeSetting(self,Wa): >> + BuildModules =3D [] >> + for Pa in Wa.AutoGenObjectList: >> + for m in Pa._MbList: >> + ma =3D ModuleAutoGen(Wa,m.MetaFile, Pa.BuildTarget, >Wa.ToolChain, Pa.Arch, Pa.MetaFile,Pa.DataPipe) >> + BuildModules.append(ma) >> + fdf_file =3D Wa.FlashDefinition >> + if fdf_file: >> + Fdf =3D FdfParser(fdf_file.Path) >> + Fdf.ParseFile() >> + GlobalData.gFdfParser =3D Fdf >> + if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile.= FdDict: >> + FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] >> + for FdRegion in FdDict.RegionList: >> + if str(FdRegion.RegionType) is 'FILE' and >self.Platform.VpdToolGuid in str(FdRegion.RegionDataList): >> + if int(FdRegion.Offset) % 8 !=3D 0: >> + EdkLogger.error("build", FORMAT_INVALID, >> + 'The VPD Base Address %s must be 8-byte aligned.' % >> (FdRegion.Offset)) >> + Wa.FdfProfile =3D Fdf.Profile >> + self.Fdf =3D Fdf >> + else: >> + self.Fdf =3D None >> + return BuildModules >> >> ## Build a platform in multi-thread mode >> # >> - def _MultiThreadBuildPlatform(self): >> - SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# FI= LE >auto-generated\n', False) >> - for BuildTarget in self.BuildTargetList: >> - GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget >> - index =3D 0 >> - for ToolChain in self.ToolChainList: >> - WorkspaceAutoGenTime =3D time.time() >> - GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain >> - GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolCh= ain >> - GlobalData.gGlobalDefines['FAMILY'] =3D self.ToolChain= Family[index] >> - index +=3D 1 >> - Wa =3D WorkspaceAutoGen( >> - self.WorkspaceDir, >> - self.PlatformFile, >> - BuildTarget, >> - ToolChain, >> - self.ArchList, >> - self.BuildDatabase, >> - self.TargetTxt, >> - self.ToolDef, >> - self.Fdf, >> - self.FdList, >> - self.FvList, >> - self.CapList, >> - self.SkuId, >> - self.UniFlag, >> - self.Progress >> - ) >> - self.Fdf =3D Wa.FdfFile >> - self.LoadFixAddress =3D Wa.Platform.LoadFixAddress >> - self.BuildReport.AddPlatformReport(Wa) >> - Wa.CreateMakeFile(False) >> + def PerformAutoGen(self,BuildTarget,ToolChain): >> + WorkspaceAutoGenTime =3D time.time() >> + Wa =3D WorkspaceAutoGen( >> + self.WorkspaceDir, >> + self.PlatformFile, >> + BuildTarget, >> + ToolChain, >> + self.ArchList, >> + self.BuildDatabase, >> + self.TargetTxt, >> + self.ToolDef, >> + self.Fdf, >> + self.FdList, >> + self.FvList, >> + self.CapList, >> + self.SkuId, >> + self.UniFlag, >> + self.Progress >> + ) >> + self.Fdf =3D Wa.FdfFile >> + self.LoadFixAddress =3D Wa.Platform.LoadFixAddress >> + self.BuildReport.AddPlatformReport(Wa) >> + Wa.CreateMakeFile(False) >> >> # Add ffs build to makefile >> - CmdListDict =3D {} >> - if GlobalData.gEnableGenfdsMultiThread and self.Fdf: >> - CmdListDict =3D self._GenFfsCmd(Wa.ArchList) >> + CmdListDict =3D {} >> + if GlobalData.gEnableGenfdsMultiThread and self.Fdf: >> + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) >> >> - # Add Platform and Package level hash in share_data fo= r module >hash calculation later >> - if GlobalData.gBinCacheSource or GlobalData.gBinCacheD= est: >> - GlobalData.gCacheIR[('PlatformHash')] =3D >GlobalData.gPlatformHash >> - for PkgName in GlobalData.gPackageHash.keys(): >> - GlobalData.gCacheIR[(PkgName, 'PackageHash')] = = =3D >GlobalData.gPackageHash[PkgName] >> + # Add Platform and Package level hash in share_data for module= hash >calculation later >> + if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest: >> + GlobalData.gCacheIR[('PlatformHash')] =3D GlobalData.gPlat= formHash >> + for PkgName in GlobalData.gPackageHash.keys(): >> + GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D >> + GlobalData.gPackageHash[PkgName] >> >> - # multi-thread exit flag >> - ExitFlag =3D threading.Event() >> - ExitFlag.clear() >> - self.AutoGenTime +=3D int(round((time.time() - >WorkspaceAutoGenTime))) >> - self.BuildModules =3D [] >> - TotalModules =3D [] >> - for Arch in Wa.ArchList: >> - PcdMaList =3D [] >> - AutoGenStart =3D time.time() >> - GlobalData.gGlobalDefines['ARCH'] =3D Arch >> - Pa =3D PlatformAutoGen(Wa, self.PlatformFile, Buil= dTarget, >ToolChain, Arch) >> - if Pa is None: >> - continue >> - ModuleList =3D [] >> - for Inf in Pa.Platform.Modules: >> - ModuleList.append(Inf) >> + self.AutoGenTime +=3D int(round((time.time() - >WorkspaceAutoGenTime))) >> + BuildModules =3D [] >> + TotalModules =3D [] >> + for Arch in Wa.ArchList: >> + PcdMaList =3D [] >> + AutoGenStart =3D time.time() >> + GlobalData.gGlobalDefines['ARCH'] =3D Arch >> + Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTarget,= ToolChain, >Arch) >> + if Pa is None: >> + continue >> + ModuleList =3D [] >> + for Inf in Pa.Platform.Modules: >> + ModuleList.append(Inf) >> # Add the INF only list in FDF >> - if GlobalData.gFdfParser is not None: >> - for InfName in GlobalData.gFdfParser.Profile.I= nfList: >> - Inf =3D PathClass(NormPath(InfName), self.= WorkspaceDir, Arch) >> - if Inf in Pa.Platform.Modules: >> - continue >> - ModuleList.append(Inf) >> - Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdLis= tDict} >> - Pa.DataPipe.DataContainer =3D {"Workspace_timestam= p": >Wa._SrcTimeStamp} >> - Pa.DataPipe.DataContainer =3D {"CommandTarget": se= lf.Target} >> - for Module in ModuleList: >> + if GlobalData.gFdfParser is not None: >> + for InfName in GlobalData.gFdfParser.Profile.InfList: >> + Inf =3D PathClass(NormPath(InfName), self.Workspac= eDir, Arch) >> + if Inf in Pa.Platform.Modules: >> + continue >> + ModuleList.append(Inf) >> + Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDict} >> + Pa.DataPipe.DataContainer =3D {"Workspace_timestamp": >Wa._SrcTimeStamp} >> + Pa.DataPipe.DataContainer =3D {"CommandTarget": self.Targe= t} >> + Pa.CreateLibModuelDirs() >> + Pa.DataPipe.DataContainer =3D >{"LibraryBuildDirectoryList":Pa.LibraryBuildDirectoryList} >> + Pa.DataPipe.DataContainer =3D >{"ModuleBuildDirectoryList":Pa.ModuleBuildDirectoryList} >> + Pa.DataPipe.DataContainer =3D {"FdsCommandDict": >Wa.GenFdsCommandDict} >> + ModuleCodaFile =3D {} >> + for ma in Pa.ModuleAutoGenList: >> + >> + ModuleCodaFile[(ma.MetaFile.File,ma.MetaFile.Root,ma.Arch,ma.MetaFil >> + e.Path)] =3D [item.Target for item in >> ma.CodaTargetList] >> + Pa.DataPipe.DataContainer =3D {"ModuleCodaFile":ModuleCoda= File} >> + for Module in ModuleList: >> # Get ModuleAutoGen object to generate C code = file and >makefile >> - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, = ToolChain, >Arch, self.PlatformFile,Pa.DataPipe) >> + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, >> + ToolChain, Arch, self.PlatformFile,Pa.DataPipe) >> >> - if Ma is None: >> + if Ma is None: >> + continue >> + if Ma.PcdIsDriver: >> + Ma.PlatformInfo =3D Pa >> + Ma.Workspace =3D Wa >> + PcdMaList.append(Ma) >> + TotalModules.append(Ma) >> + # Initialize all modules in tracking to 'FAIL' >> + GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' >> + >> + >> + mqueue =3D mp.Queue() >> + for m in Pa.GetAllModuleInfo: >> + mqueue.put(m) >> + data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVar_%s= _%s.bin" % >(str(Pa.Guid),Pa.Arch)) >> + Pa.DataPipe.dump(data_pipe_file) >> + >> + autogen_rt, errorcode =3D self.StartAutoGen(mqueue, >> + Pa.DataPipe, self.SkipAutoGen, PcdMaList,GlobalData.gCacheIR) >> + >> + # Skip cache hit modules >> + if GlobalData.gBinCacheSource: >> + for Ma in TotalModules: >> + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCach= eIR and \ >> + GlobalData.gCacheIR[(Ma.MetaFile.Path, >Ma.Arch)].PreMakeCacheHit: >> + self.HashSkipModules.append(Ma) >> + continue >> + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCach= eIR and \ >> + GlobalData.gCacheIR[(Ma.MetaFile.Path, >Ma.Arch)].MakeCacheHit: >> + self.HashSkipModules.append(Ma) >> continue >> - if Ma.PcdIsDriver: >> - Ma.PlatformInfo =3D Pa >> - Ma.Workspace =3D Wa >> - PcdMaList.append(Ma) >> - TotalModules.append(Ma) >> - # Initialize all modules in tracking to 'FAIL' >> - GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' >> + BuildModules.append(Ma) >> + else: >> + BuildModules.extend(TotalModules) >> >> - mqueue =3D mp.Queue() >> - for m in Pa.GetAllModuleInfo: >> - mqueue.put(m) >> - data_pipe_file =3D os.path.join(Pa.BuildDir, >"GlobalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) >> - Pa.DataPipe.dump(data_pipe_file) >> - autogen_rt, errorcode =3D self.StartAutoGen(mqueue= , Pa.DataPipe, >self.SkipAutoGen, PcdMaList, >> GlobalData.gCacheIR) >> + if not autogen_rt: >> + self.AutoGenMgr.TerminateWorkers() >> + self.AutoGenMgr.join(1) >> + raise FatalError(errorcode) >> + self.AutoGenTime +=3D int(round((time.time() - AutoGenStar= t))) >> + AutoGenIdFile =3D >os.path.join(GlobalData.gConfDirectory,".AutoGenIdFile.txt") >> + with open(AutoGenIdFile,"w") as fw: >> + fw.write("Arch=3D%s\n" % "|".join((Wa.ArchList))) >> + fw.write("BuildDir=3D%s\n" % Wa.BuildDir) >> + fw.write("PlatformGuid=3D%s\n" % >str(Wa.AutoGenObjectList[0].Guid)) >> + self.Progress.Stop("done!") >> + return Wa, BuildModules >> >> - # Skip cache hit modules >> - if GlobalData.gBinCacheSource: >> - for Ma in TotalModules: >> - if (Ma.MetaFile.Path, Ma.Arch) in GlobalDa= ta.gCacheIR and \ >> - GlobalData.gCacheIR[(Ma.MetaFile.Path, >Ma.Arch)].PreMakeCacheHit: >> - self.HashSkipModules.append(Ma) >> - continue >> - if (Ma.MetaFile.Path, Ma.Arch) in GlobalDa= ta.gCacheIR and \ >> - GlobalData.gCacheIR[(Ma.MetaFile.Path, >Ma.Arch)].MakeCacheHit: >> - self.HashSkipModules.append(Ma) >> - continue >> - self.BuildModules.append(Ma) >> + def _MultiThreadBuildPlatform(self): >> + SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# FI= LE >auto-generated\n', False) >> + for BuildTarget in self.BuildTargetList: >> + GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget >> + index =3D 0 >> + for ToolChain in self.ToolChainList: >> + GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain >> + GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolCh= ain >> + GlobalData.gGlobalDefines['FAMILY'] =3D self.ToolChain= Family[index] >> + index +=3D 1 >> + ExitFlag =3D threading.Event() >> + ExitFlag.clear() >> + if self.SkipAutoGen: >> + Wa =3D self.VerifyAutoGenFiles() >> + if Wa is None: >> + Wa, self.BuildModules =3D >> + self.PerformAutoGen(BuildTarget,ToolChain) >> else: >> - self.BuildModules.extend(TotalModules) >> - >> - if not autogen_rt: >> - self.AutoGenMgr.TerminateWorkers() >> - self.AutoGenMgr.join(0.1) >> - raise FatalError(errorcode) >> - self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) >> - self.Progress.Stop("done!") >> + GlobalData.gAutoGenPhase =3D True >> + self.BuildModules =3D self.SetupMakeSetting(Wa= ) >> + else: >> + Wa, self.BuildModules =3D >self.PerformAutoGen(BuildTarget,ToolChain) >> + Pa =3D Wa.AutoGenObjectList[0] >> + GlobalData.gAutoGenPhase =3D False >> >> if GlobalData.gBinCacheSource: >> EdkLogger.quiet("Total cache hit driver num: %s, >> cache miss driver num: %s" % (len(set(self.HashSkipModules)), >> len(set(self.BuildModules)))) >> CacheHitMa =3D set() >> CacheNotHitMa =3D set() @@ -2136,18 +2246,17 @@ >> class Build(): >> # Start task scheduler >> if not BuildTask.IsOnGoing(): >> >> BuildTask.StartScheduler(self.ThreadNumber, ExitFlag) >> >> # in case there's an interruption. we need a full = version of >makefile for platform >> - Pa.CreateMakeFile(False) >> + >> if BuildTask.HasError(): >> self.invalidateHash() >> EdkLogger.error("build", BUILD_ERROR, "Failed = to build >module", ExtraData=3DGlobalData.gBuildingModule) >> self.MakeTime +=3D int(round((time.time() - >> MakeStart))) >> >> MakeContiue =3D time.time() >> - >> # >> # >> # All modules have been put in build tasks queue. Tell= task >scheduler >> # to exit if all tasks are completed >> # >> @@ -2175,17 +2284,12 @@ class Build(): >> if (Arch =3D=3D 'IA32' or Arch =3D=3D 'ARM') a= nd >> self.LoadFixAddress !=3D 0xFFFFFFFFFFFFFFFF and self.LoadFixAddress >= =3D >0x100000000: >> EdkLogger.error("build", >> PARAMETER_INVALID, "FIX_LOAD_TOP_MEMORY_ADDRESS can't be set to >larger than or equal to 4G for the platorm with IA32 or ARM arch modules"= ) >> # >> # Get Module List >> # >> - ModuleList =3D {} >> - for Pa in Wa.AutoGenObjectList: >> - for Ma in Pa.ModuleAutoGenList: >> - if Ma is None: >> - continue >> - if not Ma.IsLibrary: >> - ModuleList[Ma.Guid.upper()] =3D Ma >> + ModuleList =3D {ma.Guid.upper():ma for ma in >> + self.BuildModules} >> + >> # >> # Rebase module to the preferred memory address be= fore >GenFds >> # >> MapBuffer =3D [] >> if self.LoadFixAddress !=3D 0: >> @@ -2206,33 +2310,17 @@ class Build(): >> self.GenFdsTime +=3D int(round((time.time() - = GenFdsStart))) >> # >> # Save MAP buffer into MAP file. >> # >> self._SaveMapFile(MapBuffer, Wa) >> + self.CreateGuidedSectionToolsFile(Wa) >> self.invalidateHash() >> - >> ## Generate GuidedSectionTools.txt in the FV directories. >> # >> - def CreateGuidedSectionToolsFile(self): >> + def CreateGuidedSectionToolsFile(self,Wa): >> for BuildTarget in self.BuildTargetList: >> for ToolChain in self.ToolChainList: >> - Wa =3D WorkspaceAutoGen( >> - self.WorkspaceDir, >> - self.PlatformFile, >> - BuildTarget, >> - ToolChain, >> - self.ArchList, >> - self.BuildDatabase, >> - self.TargetTxt, >> - self.ToolDef, >> - self.Fdf, >> - self.FdList, >> - self.FvList, >> - self.CapList, >> - self.SkuId, >> - self.UniFlag >> - ) >> FvDir =3D Wa.FvDir >> if not os.path.exists(FvDir): >> continue >> >> for Arch in self.ArchList: >> @@ -2285,11 +2373,10 @@ class Build(): >> if not self.SpawnMode or self.Target not in ["", "all"]: >> self.SpawnMode =3D False >> self._BuildPlatform() >> else: >> self._MultiThreadBuildPlatform() >> - self.CreateGuidedSectionToolsFile() >> else: >> self.SpawnMode =3D False >> self._BuildModule() >> >> if self.Target =3D=3D 'cleanall': >> -- >> 2.20.1.windows.1 >> >> >> > > >