From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.151, mailfrom: bob.c.feng@intel.com) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by groups.io with SMTP; Thu, 05 Sep 2019 07:05:57 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Sep 2019 07:05:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,470,1559545200"; d="scan'208";a="382901841" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga005.fm.intel.com with ESMTP; 05 Sep 2019 07:05:58 -0700 Received: from fmsmsx117.amr.corp.intel.com (10.18.116.17) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 5 Sep 2019 07:05:57 -0700 Received: from shsmsx106.ccr.corp.intel.com (10.239.4.159) by fmsmsx117.amr.corp.intel.com (10.18.116.17) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 5 Sep 2019 07:05:56 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.32]) by SHSMSX106.ccr.corp.intel.com ([169.254.10.86]) with mapi id 14.03.0439.000; Thu, 5 Sep 2019 22:05:55 +0800 From: "Bob Feng" To: "devel@edk2.groups.io" , "Feng, Bob C" , "Gao, Liming" CC: "Shi, Steven" Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Topic: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAutoGen Thread-Index: AQHVY8kKJhoEn+NVaEOxtfk6jvNP1acciT6AgACQx4CAAAQscA== Date: Thu, 5 Sep 2019 14:05:55 +0000 Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D16153133C@SHSMSX104.ccr.corp.intel.com> References: <20190905090458.22976-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E4F3352@SHSMSX104.ccr.corp.intel.com> <15C18FCC38905317.10639@groups.io> In-Reply-To: <15C18FCC38905317.10639@groups.io> Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiN2RkM2RhODYtYzVlZC00ZTc3LWJmNzQtMzcwZjU4YzQ4ZTE5IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoib3dENXprRE1qSTJBZkt4bmp5dUtjU3ZLUDNGT3pmUXlSVEh6dVdlOHpPXC9VUDNid1U5TDVQUWRsWkVuQ3crbVYifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: bob.c.feng@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Sorry for the incomplete mail. I verified this patch on Ovmf, Emulator, MinPlatforms and Intel Server Pla= tform. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io On Behalf Of Bob Feng Sent: Thursday, September 5, 2019 10:04 PM To: Gao, Liming ; devel@edk2.groups.io Cc: Shi, Steven Subject: Re: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAut= oGen Liming, Yes, this change only impact -u option.=20 I separate _MultiThreadBuildPlatform function into two functions and creat= e 2 new functions specially for IgnoreAutoGen, so the there looks big chan= ge for build.py in this patch. I verified this patch on Ovmf, E -Bob -----Original Message----- From: Gao, Liming Sent: Thursday, September 5, 2019 9:12 PM To: devel@edk2.groups.io; Feng, Bob C Cc: Shi, Steven Subject: RE: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for IgnoreAut= oGen Bob:=20 Does this change only impact -u option behavior?=20 Thanks Liming > -----Original Message----- > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of=20 > Bob Feng > Sent: Thursday, September 5, 2019 5:05 PM > To: devel@edk2.groups.io > Cc: Gao, Liming ; Shi, Steven=20 > ; Feng, Bob C > Subject: [edk2-devel] [Patch 1/1] BaseTools: Fixed issue for=20 > IgnoreAutoGen >=20 > https://bugzilla.tianocore.org/show_bug.cgi?id=3D2080 >=20 > This patch is to improve build -u option to re-use=20 > GlobalVar__.bin file which is introduced by=20 > multiple-process-autogen feature. >=20 > Cc: Liming Gao > Cc: Steven Shi > Signed-off-by: Bob Feng > --- > .../Source/Python/AutoGen/AutoGenWorker.py | 2 +- > .../Source/Python/AutoGen/ModuleAutoGen.py | 4 +- > .../Python/AutoGen/ModuleAutoGenHelper.py | 29 ++ > .../Source/Python/AutoGen/PlatformAutoGen.py | 2 + > .../Source/Python/AutoGen/WorkspaceAutoGen.py | 1 - > BaseTools/Source/Python/build/build.py | 355 +++++++++++------- > 6 files changed, 255 insertions(+), 138 deletions(-) >=20 > diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py > b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py > index 2e68538b1cb4..f488ae9d5f80 100755 > --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py > +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py > @@ -236,11 +236,11 @@ class AutoGenWorkerInProcess(mp.Process): > Ma.ReferenceModules =3D Refes[(Ma.MetaFile.File= ,Ma.MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)] > if GlobalData.gBinCacheSource and CommandTarget in [Non= e, "", "all"]: > Ma.GenModuleFilesHash(GlobalData.gCacheIR) > Ma.GenPreMakefileHash(GlobalData.gCacheIR) > if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheIR= ): > - continue > + continue >=20 > Ma.CreateCodeFile(False) > > Ma.CreateMakeFile(False,GenFfsList=3DFfsCmd.get((Ma.MetaFile.File, > Ma.Arch),[])) >=20 > if GlobalData.gBinCacheSource and CommandTarget in [Non= e, "", "all"]: > diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py > index 3bb7e91154ac..5f28681e3146 100755 > --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py > @@ -2058,12 +2058,12 @@ class ModuleAutoGen(AutoGen): > not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDi= gest: > self.GenModuleFilesHash(gDict) >=20 > if not (self.MetaFile.Path, self.Arch) in gDict or \ > not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDi= gest: > - EdkLogger.quiet("[cache warning]: Cannot generate ModuleFile= sHashDigest for module %s[%s]" %(self.MetaFile.Path, > self.Arch)) > - return > + EdkLogger.quiet("[cache warning]: Cannot generate=20 > + ModuleFilesHashDigest for module %s[%s]" %(self.MetaFile.Path, > self.Arch)) > + return >=20 > # Initialze hash object > m =3D hashlib.md5() >=20 > # Add Platform level hash > diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py > b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py > index c7591253debd..9dd93b9beb12 100644 > --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py > +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py > @@ -234,22 +234,51 @@ class AutoGenInfo(object): > # > #This class is the pruned WorkSpaceAutoGen for ModuleAutoGen in=20 > multiple thread # class WorkSpaceInfo(AutoGenInfo): > def __init__(self,Workspace, MetaFile, Target, ToolChain, Arch): > + if not hasattr(self, "_Init"): > + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch) > + self._Init =3D True > + def do_init(self,Workspace, MetaFile, Target, ToolChain, Arch): > self._SrcTimeStamp =3D 0 > self.Db =3D BuildDB > self.BuildDatabase =3D self.Db.BuildObject > self.Target =3D Target > self.ToolChain =3D ToolChain > self.WorkspaceDir =3D Workspace > self.ActivePlatform =3D MetaFile > self.ArchList =3D Arch > + self.AutoGenObjectList =3D [] > + @property > + def BuildDir(self): > + return self.AutoGenObjectList[0].BuildDir >=20 > + @property > + def Name(self): > + return self.AutoGenObjectList[0].Platform.PlatformName > + > + @property > + def FlashDefinition(self): > + return self.AutoGenObjectList[0].Platform.FlashDefinition > + @property > + def GenFdsCommandDict(self): > + FdsCommandDict =3D self.AutoGenObjectList[0].DataPipe.Get("FdsC= ommandDict") > + if FdsCommandDict: > + return FdsCommandDict > + return {} > + > + @cached_property > + def FvDir(self): > + return os.path.join(self.BuildDir, TAB_FV_DIRECTORY) >=20 > class PlatformInfo(AutoGenInfo): > def __init__(self, Workspace, MetaFile, Target, ToolChain, Arch,Dat= aPipe): > + if not hasattr(self, "_Init"): > + self.do_init(Workspace, MetaFile, Target, ToolChain, Arch,D= ataPipe) > + self._Init =3D True > + def do_init(self,Workspace, MetaFile, Target, ToolChain, Arch,DataP= ipe): > self.Wa =3D Workspace > self.WorkspaceDir =3D self.Wa.WorkspaceDir > self.MetaFile =3D MetaFile > self.Arch =3D Arch > self.Target =3D Target > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > index 565424a95ead..1e17b6687129 100644 > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > @@ -186,11 +186,13 @@ class PlatformAutoGen(AutoGen): > key =3D (Ma.MetaFile.File, self.Arch) > if key in FfsCommand: > Ma.CreateMakeFile(CreateModuleMakeFile, FfsCommand[= key]) > else: > Ma.CreateMakeFile(CreateModuleMakeFile) > + self.CreateLibModuelDirs() >=20 > + def CreateLibModuelDirs(self): > # no need to create makefile for the platform more than once > if self.IsMakeFileCreated: > return >=20 > # create library/module build dirs for platform diff --git=20 > a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py > b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py > index 24942674721f..f9d2c216169c 100644 > --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py > @@ -126,11 +126,10 @@ class WorkspaceAutoGen(AutoGen): > self._CheckPcdDefineAndType() >=20 > self.CreateBuildOptionsFile() > self.CreatePcdTokenNumberFile() > self.CreateModuleHashInfo() > - GlobalData.gAutoGenPhase =3D False >=20 > # > # Merge Arch > # > def MergeArch(self): > diff --git a/BaseTools/Source/Python/build/build.py > b/BaseTools/Source/Python/build/build.py > index 0406ac314b65..db57c3e526da 100755 > --- a/BaseTools/Source/Python/build/build.py > +++ b/BaseTools/Source/Python/build/build.py > @@ -52,10 +52,13 @@ from PatchPcdValue.PatchPcdValue import=20 > PatchBinaryFile >=20 > import Common.GlobalData as GlobalData from GenFds.GenFds import=20 > GenFds, GenFdsApi import multiprocessing as mp from multiprocessing=20 > import Manager > +from AutoGen.DataPipe import MemoryDataPipe from=20 > +AutoGen.ModuleAutoGenHelper import WorkSpaceInfo, PlatformInfo from=20 > +GenFds.FdfParser import FdfParser >=20 >=20 > ## standard targets of build command > gSupportedTarget =3D ['all', 'genc', 'genmake', 'modules', 'libraries',= = =20 > 'fds', 'clean', 'cleanall', 'cleanlib', 'run'] >=20 > @@ -837,11 +840,11 @@ class Build(): > for PcdMa in PcdMaList: > if GlobalData.gBinCacheSource and self.Target in [N= one, "", "all"]: > PcdMa.GenModuleFilesHash(share_data) > PcdMa.GenPreMakefileHash(share_data) > if PcdMa.CanSkipbyPreMakefileCache(share_data): > - continue > + continue >=20 > PcdMa.CreateCodeFile(False) > PcdMa.CreateMakeFile(False,GenFfsList =3D=20 > DataPipe.Get("FfsCommand").get((PcdMa.MetaFile.File, > PcdMa.Arch),[])) >=20 > if GlobalData.gBinCacheSource and self.Target in [N= one, "", "all"]: > @@ -1218,18 +1221,28 @@ class Build(): > mqueue =3D mp.Queue() > for m in AutoGenObject.GetAllModuleInfo: > mqueue.put(m) >=20 > AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget":= =20 > self.Target} > + AutoGenObject.DataPipe.DataContainer =3D {"Workspace_timest= amp": AutoGenObject.Workspace._SrcTimeStamp} > + AutoGenObject.CreateLibModuelDirs() > + AutoGenObject.DataPipe.DataContainer =3D {"LibraryBuildDire= ctoryList":AutoGenObject.LibraryBuildDirectoryList} > + AutoGenObject.DataPipe.DataContainer =3D {"ModuleBuildDirec= toryList":AutoGenObject.ModuleBuildDirectoryList} > + AutoGenObject.DataPipe.DataContainer =3D {"FdsCommandDict":= = =20 > + AutoGenObject.Workspace.GenFdsCommandDict} > self.Progress.Start("Generating makefile and code") > data_pipe_file =3D os.path.join(AutoGenObject.BuildDir, > "GlobalVar_%s_%s.bin" % > (str(AutoGenObject.Guid),AutoGenObject.Arch)) > AutoGenObject.DataPipe.dump(data_pipe_file) > autogen_rt,errorcode =3D self.StartAutoGen(mqueue,=20 > AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList, > GlobalData.gCacheIR) > + AutoGenIdFile =3D os.path.join(GlobalData.gConfDirectory,".= AutoGenIdFile.txt") > + with open(AutoGenIdFile,"w") as fw: > + fw.write("Arch=3D%s\n" % "|".join((AutoGenObject.Worksp= ace.ArchList))) > + fw.write("BuildDir=3D%s\n" % AutoGenObject.Workspace.Bu= ildDir) > + fw.write("PlatformGuid=3D%s\n" % > + str(AutoGenObject.Guid)) > self.Progress.Stop("done!") > if not autogen_rt: > self.AutoGenMgr.TerminateWorkers() > - self.AutoGenMgr.join(0.1) > + self.AutoGenMgr.join(1) > raise FatalError(errorcode) > AutoGenObject.CreateCodeFile(False) > AutoGenObject.CreateMakeFile(False) > else: > # always recreate top/platform makefile when clean, just=20 > in case of inconsistency @@ -1778,10 +1791,11 @@ class Build(): > self._CollectFvMapBuffer(MapBuffer, Wa, ModuleL= ist) > # > # Save MAP buffer into MAP file. > # > self._SaveMapFile (MapBuffer, Wa) > + self.CreateGuidedSectionToolsFile(Wa) >=20 > ## Build active module for different build targets, different tool = chains and different archs > # > def _BuildModule(self): > for BuildTarget in self.BuildTargetList: > @@ -1852,13 +1866,13 @@ class Build(): >=20 > if GlobalData.gBinCacheSource and self.Targ= et in [None, "", "all"]: > Ma.GenModuleFilesHash(GlobalData.gCache= IR) > Ma.GenPreMakefileHash(GlobalData.gCache= IR) > if Ma.CanSkipbyPreMakefileCache(GlobalD= ata.gCacheIR): > - self.HashSkipModules.append(Ma) > - EdkLogger.quiet("cache hit: %s[%s]" = % (Ma.MetaFile.Path, Ma.Arch)) > - continue > + self.HashSkipModules.append(Ma) > + EdkLogger.quiet("cache hit: %s[%s]"= % (Ma.MetaFile.Path, Ma.Arch)) > + continue >=20 > # Not to auto-gen for targets 'clean', 'cle= anlib', 'cleanall', 'run', 'fds' > if self.Target not in ['clean', 'cleanlib',= 'cleanall', 'run', 'fds']: > # for target which must generate AutoGe= n code and makefile > if not self.SkipAutoGen or self.Target = = =3D=3D 'genc': > @@ -1987,125 +2001,221 @@ class Build(): > GenFfsDict =3D GenFds.GenFfsMakefile('', GlobalData.gFdfParser,= self, ArchList, GlobalData) > for Cmd in GenFfsDict: > tmpInf, tmpArch =3D GenFfsDict[Cmd] > CmdSetDict[tmpInf, tmpArch].add(Cmd) > return CmdSetDict > + def VerifyAutoGenFiles(self): > + AutoGenIdFile =3D os.path.join(GlobalData.gConfDirectory,".Auto= GenIdFile.txt") > + try: > + with open(AutoGenIdFile) as fd: > + lines =3D fd.readlines() > + except: > + return None > + for line in lines: > + if "Arch" in line: > + ArchList =3D line.strip().split("=3D")[1].split("|") > + if "BuildDir" in line: > + BuildDir =3D line.split("=3D")[1].strip() > + if "PlatformGuid" in line: > + PlatformGuid =3D line.split("=3D")[1].strip() > + GlobalVarList =3D [] > + for arch in ArchList: > + global_var =3D os.path.join(BuildDir, "GlobalVar_%s_%s.bin"= % (str(PlatformGuid),arch)) > + if not os.path.exists(global_var): > + return None > + GlobalVarList.append(global_var) > + for global_var in GlobalVarList: > + data_pipe =3D MemoryDataPipe() > + data_pipe.load(global_var) > + target =3D data_pipe.Get("P_Info").get("Target") > + toolchain =3D data_pipe.Get("P_Info").get("ToolChain") > + archlist =3D data_pipe.Get("P_Info").get("ArchList") > + Arch =3D data_pipe.Get("P_Info").get("Arch") > + active_p =3D data_pipe.Get("P_Info").get("ActivePlatform") > + workspacedir =3D data_pipe.Get("P_Info").get("WorkspaceDir"= ) > + PackagesPath =3D os.getenv("PACKAGES_PATH") > + mws.setWs(workspacedir, PackagesPath) > + LibraryBuildDirectoryList =3D data_pipe.Get("LibraryBuildDi= rectoryList") > + ModuleBuildDirectoryList =3D > + data_pipe.Get("ModuleBuildDirectoryList") > + > + for m_build_dir in LibraryBuildDirectoryList: > + if not os.path.exists(os.path.join(m_build_dir,GenMake.= BuildFile._FILE_NAME_[GenMake.gMakeType])): > + return None > + for m_build_dir in ModuleBuildDirectoryList: > + if not os.path.exists(os.path.join(m_build_dir,GenMake.= BuildFile._FILE_NAME_[GenMake.gMakeType])): > + return None > + Wa =3D WorkSpaceInfo( > + workspacedir,active_p,target,toolchain,archlist > + ) > + Pa =3D PlatformInfo(Wa, active_p, target, toolchain, Arch,d= ata_pipe) > + Wa.AutoGenObjectList.append(Pa) > + return Wa > + def SetupMakeSetting(self,Wa): > + BuildModules =3D [] > + for Pa in Wa.AutoGenObjectList: > + for m in Pa._MbList: > + ma =3D ModuleAutoGen(Wa,m.MetaFile, Pa.BuildTarget, Wa.= ToolChain, Pa.Arch, Pa.MetaFile,Pa.DataPipe) > + BuildModules.append(ma) > + fdf_file =3D Wa.FlashDefinition > + if fdf_file: > + Fdf =3D FdfParser(fdf_file.Path) > + Fdf.ParseFile() > + GlobalData.gFdfParser =3D Fdf > + if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile.F= dDict: > + FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] > + for FdRegion in FdDict.RegionList: > + if str(FdRegion.RegionType) is 'FILE' and self.Plat= form.VpdToolGuid in str(FdRegion.RegionDataList): > + if int(FdRegion.Offset) % 8 !=3D 0: > + EdkLogger.error("build", FORMAT_INVALID,=20 > + 'The VPD Base Address %s must be 8-byte aligned.' % > (FdRegion.Offset)) > + Wa.FdfProfile =3D Fdf.Profile > + self.Fdf =3D Fdf > + else: > + self.Fdf =3D None > + return BuildModules >=20 > ## Build a platform in multi-thread mode > # > - def _MultiThreadBuildPlatform(self): > - SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# FIL= E auto-generated\n', False) > - for BuildTarget in self.BuildTargetList: > - GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget > - index =3D 0 > - for ToolChain in self.ToolChainList: > - WorkspaceAutoGenTime =3D time.time() > - GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain > - GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolCha= in > - GlobalData.gGlobalDefines['FAMILY'] =3D self.ToolChainF= amily[index] > - index +=3D 1 > - Wa =3D WorkspaceAutoGen( > - self.WorkspaceDir, > - self.PlatformFile, > - BuildTarget, > - ToolChain, > - self.ArchList, > - self.BuildDatabase, > - self.TargetTxt, > - self.ToolDef, > - self.Fdf, > - self.FdList, > - self.FvList, > - self.CapList, > - self.SkuId, > - self.UniFlag, > - self.Progress > - ) > - self.Fdf =3D Wa.FdfFile > - self.LoadFixAddress =3D Wa.Platform.LoadFixAddress > - self.BuildReport.AddPlatformReport(Wa) > - Wa.CreateMakeFile(False) > + def PerformAutoGen(self,BuildTarget,ToolChain): > + WorkspaceAutoGenTime =3D time.time() > + Wa =3D WorkspaceAutoGen( > + self.WorkspaceDir, > + self.PlatformFile, > + BuildTarget, > + ToolChain, > + self.ArchList, > + self.BuildDatabase, > + self.TargetTxt, > + self.ToolDef, > + self.Fdf, > + self.FdList, > + self.FvList, > + self.CapList, > + self.SkuId, > + self.UniFlag, > + self.Progress > + ) > + self.Fdf =3D Wa.FdfFile > + self.LoadFixAddress =3D Wa.Platform.LoadFixAddress > + self.BuildReport.AddPlatformReport(Wa) > + Wa.CreateMakeFile(False) >=20 > # Add ffs build to makefile > - CmdListDict =3D {} > - if GlobalData.gEnableGenfdsMultiThread and self.Fdf: > - CmdListDict =3D self._GenFfsCmd(Wa.ArchList) > + CmdListDict =3D {} > + if GlobalData.gEnableGenfdsMultiThread and self.Fdf: > + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) >=20 > - # Add Platform and Package level hash in share_data for= module hash calculation later > - if GlobalData.gBinCacheSource or GlobalData.gBinCacheDe= st: > - GlobalData.gCacheIR[('PlatformHash')] =3D GlobalDat= a.gPlatformHash > - for PkgName in GlobalData.gPackageHash.keys(): > - GlobalData.gCacheIR[(PkgName, 'PackageHash')] = =3D GlobalData.gPackageHash[PkgName] > + # Add Platform and Package level hash in share_data for module = hash calculation later > + if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest: > + GlobalData.gCacheIR[('PlatformHash')] =3D GlobalData.gPlatf= ormHash > + for PkgName in GlobalData.gPackageHash.keys(): > + GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D=20 > + GlobalData.gPackageHash[PkgName] >=20 > - # multi-thread exit flag > - ExitFlag =3D threading.Event() > - ExitFlag.clear() > - self.AutoGenTime +=3D int(round((time.time() - Workspac= eAutoGenTime))) > - self.BuildModules =3D [] > - TotalModules =3D [] > - for Arch in Wa.ArchList: > - PcdMaList =3D [] > - AutoGenStart =3D time.time() > - GlobalData.gGlobalDefines['ARCH'] =3D Arch > - Pa =3D PlatformAutoGen(Wa, self.PlatformFile, Build= Target, ToolChain, Arch) > - if Pa is None: > - continue > - ModuleList =3D [] > - for Inf in Pa.Platform.Modules: > - ModuleList.append(Inf) > + self.AutoGenTime +=3D int(round((time.time() - WorkspaceAutoGen= Time))) > + BuildModules =3D [] > + TotalModules =3D [] > + for Arch in Wa.ArchList: > + PcdMaList =3D [] > + AutoGenStart =3D time.time() > + GlobalData.gGlobalDefines['ARCH'] =3D Arch > + Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTarget, = ToolChain, Arch) > + if Pa is None: > + continue > + ModuleList =3D [] > + for Inf in Pa.Platform.Modules: > + ModuleList.append(Inf) > # Add the INF only list in FDF > - if GlobalData.gFdfParser is not None: > - for InfName in GlobalData.gFdfParser.Profile.In= fList: > - Inf =3D PathClass(NormPath(InfName), self.W= orkspaceDir, Arch) > - if Inf in Pa.Platform.Modules: > - continue > - ModuleList.append(Inf) > - Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdList= Dict} > - Pa.DataPipe.DataContainer =3D {"Workspace_timestamp= ": Wa._SrcTimeStamp} > - Pa.DataPipe.DataContainer =3D {"CommandTarget": sel= f.Target} > - for Module in ModuleList: > + if GlobalData.gFdfParser is not None: > + for InfName in GlobalData.gFdfParser.Profile.InfList: > + Inf =3D PathClass(NormPath(InfName), self.Workspace= Dir, Arch) > + if Inf in Pa.Platform.Modules: > + continue > + ModuleList.append(Inf) > + Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDict} > + Pa.DataPipe.DataContainer =3D {"Workspace_timestamp": Wa._S= rcTimeStamp} > + Pa.DataPipe.DataContainer =3D {"CommandTarget": self.Target= } > + Pa.CreateLibModuelDirs() > + Pa.DataPipe.DataContainer =3D {"LibraryBuildDirectoryList":= Pa.LibraryBuildDirectoryList} > + Pa.DataPipe.DataContainer =3D {"ModuleBuildDirectoryList":P= a.ModuleBuildDirectoryList} > + Pa.DataPipe.DataContainer =3D {"FdsCommandDict": Wa.GenFdsC= ommandDict} > + ModuleCodaFile =3D {} > + for ma in Pa.ModuleAutoGenList: > + > + ModuleCodaFile[(ma.MetaFile.File,ma.MetaFile.Root,ma.Arch,ma.MetaFil > + e.Path)] =3D [item.Target for item in > ma.CodaTargetList] > + Pa.DataPipe.DataContainer =3D {"ModuleCodaFile":ModuleCodaF= ile} > + for Module in ModuleList: > # Get ModuleAutoGen object to generate C code f= ile and makefile > - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, T= oolChain, Arch, self.PlatformFile,Pa.DataPipe) > + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget,=20 > + ToolChain, Arch, self.PlatformFile,Pa.DataPipe) >=20 > - if Ma is None: > + if Ma is None: > + continue > + if Ma.PcdIsDriver: > + Ma.PlatformInfo =3D Pa > + Ma.Workspace =3D Wa > + PcdMaList.append(Ma) > + TotalModules.append(Ma) > + # Initialize all modules in tracking to 'FAIL' > + GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' > + > + > + mqueue =3D mp.Queue() > + for m in Pa.GetAllModuleInfo: > + mqueue.put(m) > + data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVar_%s_= %s.bin" % (str(Pa.Guid),Pa.Arch)) > + Pa.DataPipe.dump(data_pipe_file) > + > + autogen_rt, errorcode =3D self.StartAutoGen(mqueue,=20 > + Pa.DataPipe, self.SkipAutoGen, PcdMaList,GlobalData.gCacheIR) > + > + # Skip cache hit modules > + if GlobalData.gBinCacheSource: > + for Ma in TotalModules: > + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCache= IR and \ > + GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)= ].PreMakeCacheHit: > + self.HashSkipModules.append(Ma) > + continue > + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.gCache= IR and \ > + GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.Arch)= ].MakeCacheHit: > + self.HashSkipModules.append(Ma) > continue > - if Ma.PcdIsDriver: > - Ma.PlatformInfo =3D Pa > - Ma.Workspace =3D Wa > - PcdMaList.append(Ma) > - TotalModules.append(Ma) > - # Initialize all modules in tracking to 'FAIL' > - GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' > + BuildModules.append(Ma) > + else: > + BuildModules.extend(TotalModules) >=20 > - mqueue =3D mp.Queue() > - for m in Pa.GetAllModuleInfo: > - mqueue.put(m) > - data_pipe_file =3D os.path.join(Pa.BuildDir, "Globa= lVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) > - Pa.DataPipe.dump(data_pipe_file) > - autogen_rt, errorcode =3D self.StartAutoGen(mqueue,= Pa.DataPipe, self.SkipAutoGen, PcdMaList, > GlobalData.gCacheIR) > + if not autogen_rt: > + self.AutoGenMgr.TerminateWorkers() > + self.AutoGenMgr.join(1) > + raise FatalError(errorcode) > + self.AutoGenTime +=3D int(round((time.time() - AutoGenStart= ))) > + AutoGenIdFile =3D os.path.join(GlobalData.gConfDirectory,".Auto= GenIdFile.txt") > + with open(AutoGenIdFile,"w") as fw: > + fw.write("Arch=3D%s\n" % "|".join((Wa.ArchList))) > + fw.write("BuildDir=3D%s\n" % Wa.BuildDir) > + fw.write("PlatformGuid=3D%s\n" % str(Wa.AutoGenObjectList[0= ].Guid)) > + self.Progress.Stop("done!") > + return Wa, BuildModules >=20 > - # Skip cache hit modules > - if GlobalData.gBinCacheSource: > - for Ma in TotalModules: > - if (Ma.MetaFile.Path, Ma.Arch) in GlobalDat= a.gCacheIR and \ > - GlobalData.gCacheIR[(Ma.MetaFile.Path, = Ma.Arch)].PreMakeCacheHit: > - self.HashSkipModules.append(Ma) > - continue > - if (Ma.MetaFile.Path, Ma.Arch) in GlobalDat= a.gCacheIR and \ > - GlobalData.gCacheIR[(Ma.MetaFile.Path, = Ma.Arch)].MakeCacheHit: > - self.HashSkipModules.append(Ma) > - continue > - self.BuildModules.append(Ma) > + def _MultiThreadBuildPlatform(self): > + SaveFileOnChange(self.PlatformBuildPath, '# DO NOT EDIT \n# FIL= E auto-generated\n', False) > + for BuildTarget in self.BuildTargetList: > + GlobalData.gGlobalDefines['TARGET'] =3D BuildTarget > + index =3D 0 > + for ToolChain in self.ToolChainList: > + GlobalData.gGlobalDefines['TOOLCHAIN'] =3D ToolChain > + GlobalData.gGlobalDefines['TOOL_CHAIN_TAG'] =3D ToolCha= in > + GlobalData.gGlobalDefines['FAMILY'] =3D self.ToolChainF= amily[index] > + index +=3D 1 > + ExitFlag =3D threading.Event() > + ExitFlag.clear() > + if self.SkipAutoGen: > + Wa =3D self.VerifyAutoGenFiles() > + if Wa is None: > + Wa, self.BuildModules =3D > + self.PerformAutoGen(BuildTarget,ToolChain) > else: > - self.BuildModules.extend(TotalModules) > - > - if not autogen_rt: > - self.AutoGenMgr.TerminateWorkers() > - self.AutoGenMgr.join(0.1) > - raise FatalError(errorcode) > - self.AutoGenTime +=3D int(round((time.time() - AutoGenS= tart))) > - self.Progress.Stop("done!") > + GlobalData.gAutoGenPhase =3D True > + self.BuildModules =3D self.SetupMakeSetting(Wa) > + else: > + Wa, self.BuildModules =3D self.PerformAutoGen(Build= Target,ToolChain) > + Pa =3D Wa.AutoGenObjectList[0] > + GlobalData.gAutoGenPhase =3D False >=20 > if GlobalData.gBinCacheSource: > EdkLogger.quiet("Total cache hit driver num: %s,=20 > cache miss driver num: %s" % (len(set(self.HashSkipModules)), > len(set(self.BuildModules)))) > CacheHitMa =3D set() > CacheNotHitMa =3D set() @@ -2136,18 +2246,17 @@=20 > class Build(): > # Start task scheduler > if not BuildTask.IsOnGoing(): > > BuildTask.StartScheduler(self.ThreadNumber, ExitFlag) >=20 > # in case there's an interruption. we need a full v= ersion of makefile for platform > - Pa.CreateMakeFile(False) > + > if BuildTask.HasError(): > self.invalidateHash() > EdkLogger.error("build", BUILD_ERROR, "Failed t= o build module", ExtraData=3DGlobalData.gBuildingModule) > self.MakeTime +=3D int(round((time.time() - > MakeStart))) >=20 > MakeContiue =3D time.time() > - > # > # > # All modules have been put in build tasks queue. Tell = task scheduler > # to exit if all tasks are completed > # > @@ -2175,17 +2284,12 @@ class Build(): > if (Arch =3D=3D 'IA32' or Arch =3D=3D 'ARM') an= d=20 > self.LoadFixAddress !=3D 0xFFFFFFFFFFFFFFFF and self.LoadFixAddress >=3D= 0x100000000: > EdkLogger.error("build",=20 > PARAMETER_INVALID, "FIX_LOAD_TOP_MEMORY_ADDRESS can't be set to larger t= han or equal to 4G for the platorm with IA32 or ARM arch modules") > # > # Get Module List > # > - ModuleList =3D {} > - for Pa in Wa.AutoGenObjectList: > - for Ma in Pa.ModuleAutoGenList: > - if Ma is None: > - continue > - if not Ma.IsLibrary: > - ModuleList[Ma.Guid.upper()] =3D Ma > + ModuleList =3D {ma.Guid.upper():ma for ma in=20 > + self.BuildModules} > + > # > # Rebase module to the preferred memory address bef= ore GenFds > # > MapBuffer =3D [] > if self.LoadFixAddress !=3D 0: > @@ -2206,33 +2310,17 @@ class Build(): > self.GenFdsTime +=3D int(round((time.time() - G= enFdsStart))) > # > # Save MAP buffer into MAP file. > # > self._SaveMapFile(MapBuffer, Wa) > + self.CreateGuidedSectionToolsFile(Wa) > self.invalidateHash() > - > ## Generate GuidedSectionTools.txt in the FV directories. > # > - def CreateGuidedSectionToolsFile(self): > + def CreateGuidedSectionToolsFile(self,Wa): > for BuildTarget in self.BuildTargetList: > for ToolChain in self.ToolChainList: > - Wa =3D WorkspaceAutoGen( > - self.WorkspaceDir, > - self.PlatformFile, > - BuildTarget, > - ToolChain, > - self.ArchList, > - self.BuildDatabase, > - self.TargetTxt, > - self.ToolDef, > - self.Fdf, > - self.FdList, > - self.FvList, > - self.CapList, > - self.SkuId, > - self.UniFlag > - ) > FvDir =3D Wa.FvDir > if not os.path.exists(FvDir): > continue >=20 > for Arch in self.ArchList: > @@ -2285,11 +2373,10 @@ class Build(): > if not self.SpawnMode or self.Target not in ["", "all"]: > self.SpawnMode =3D False > self._BuildPlatform() > else: > self._MultiThreadBuildPlatform() > - self.CreateGuidedSectionToolsFile() > else: > self.SpawnMode =3D False > self._BuildModule() >=20 > if self.Target =3D=3D 'cleanall': > -- > 2.20.1.windows.1 >=20 >=20 >=20