From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.31; helo=mga06.intel.com; envelope-from=bob.c.feng@intel.com; receiver=edk2-devel@lists.01.org Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 399CA21959CB2 for ; Fri, 29 Mar 2019 05:39:49 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Mar 2019 05:39:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,284,1549958400"; d="scan'208";a="218741400" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga001.jf.intel.com with ESMTP; 29 Mar 2019 05:39:48 -0700 Received: from fmsmsx117.amr.corp.intel.com (10.18.116.17) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 29 Mar 2019 05:39:48 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by fmsmsx117.amr.corp.intel.com (10.18.116.17) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 29 Mar 2019 05:39:47 -0700 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.158]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.134]) with mapi id 14.03.0415.000; Fri, 29 Mar 2019 20:39:45 +0800 From: "Feng, Bob C" To: "Gao, Liming" , "edk2-devel@lists.01.org" Thread-Topic: [Patch] BaseTools: Fixed issue in MultiThread Genfds function Thread-Index: AQHU5h99iyqsQYraR0WD3xHRL13fNKYh9loAgACGUZD//4gWgIAAhscA Date: Fri, 29 Mar 2019 12:39:44 +0000 Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D1600BC667@SHSMSX101.ccr.corp.intel.com> References: <20190329110631.79000-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E40EBB2@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D1600BC604@SHSMSX101.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E40ECCB@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <4A89E2EF3DFEDB4C8BFDE51014F606A14E40ECCB@SHSMSX104.ccr.corp.intel.com> Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Fixed issue in MultiThread Genfds function X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2019 12:39:49 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable That will impact the original GenFds since the command is generate by reusi= ng the GenFds functions. What about enter a new BZ to do that change? Thanks, Bob -----Original Message----- From: Gao, Liming=20 Sent: Friday, March 29, 2019 8:33 PM To: Feng, Bob C ; edk2-devel@lists.01.org Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds function Bob: Could you update the rule to depend on EFI image from DEBUG_DIR? I would = like to remove the additional copy from DEBUG dir to OUTPUT dir and save th= e disk and performance.=20 $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32 : $(DEBUG_DIR)\LogoDxe.efi GenSec -s EFI_SECTION_PE32 -o $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.= pe32 $(DEBUG_DIR)\LogoDxe.efi Thanks Liming > -----Original Message----- > From: Feng, Bob C > Sent: Friday, March 29, 2019 8:00 PM > To: Gao, Liming ; edk2-devel@lists.01.org > Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds=20 > function >=20 > That's to fix the issue that make will fail if enable multiple thread gen= fds. >=20 > For example, If enable Multiple thread Genfds, there will be GenSec comma= nd in the Makefile like below. >=20 > $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32 : $(OUTPUT_DIR)\LogoDxe.e= fi > GenSec -s EFI_SECTION_PE32 -o=20 > $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32=20 > $(OUTPUT_DIR)\LogoDxe.efi >=20 >=20 > The $(OUTPUT_DIR)\LogoDxe.efi is the dependency, but there is no make rul= e for $(OUTPUT_DIR)\LogoDxe.efi generated in the Makefile. > Make program does not know how to make $(OUTPUT_DIR)\LogoDxe.efi. Make f= ails for this case. >=20 > In the build_rule.txt, the $(OUTPUT_DIR)\ )(+)$(MODULE_NAME).efi is gen= erated by the command $(CP) > $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR) under=20 > Dynamic-Library-File section, so $(OUTPUT_DIR)\ )(+)$(MODULE_NAME).efi=20 > should also be a output for Dynamic-Library-File >=20 > Thanks, > Bob >=20 > -----Original Message----- > From: Gao, Liming > Sent: Friday, March 29, 2019 7:42 PM > To: Feng, Bob C ; edk2-devel@lists.01.org > Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds=20 > function >=20 > Bob: > Could you list the more information on why update build_rule.txt? >=20 > > -----Original Message----- > > From: Feng, Bob C > > Sent: Friday, March 29, 2019 7:07 PM > > To: edk2-devel@lists.01.org > > Cc: Feng, Bob C ; Gao, Liming=20 > > > > Subject: [Patch] BaseTools: Fixed issue in MultiThread Genfds=20 > > function > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1450 > > In the Multiple thread Genfds feature, build tool generates GenSec,=20 > > GenFFS command in Makefile. > > > > The Non-Hii Driver does not generate .offset file for uni string=20 > > offset, but the build tool has not knowledge about this in autogen=20 > > phase. So in this patch, I add a check in Makefile for GenSec command. > > If the GenSec input file does not exist, the GenSec will not be=20 > > called. And if GenSec command is not called, its output file, which=20 > > is also the input file of GenFfs command, will also not exist.So for=20 > > GenFfs command, I add a new command parameter -oi which means the=20 > > input file is an optional input file which would not exist. so that=20 > > I can generate GenFfs command with "-oi" parameter in Makefile. > > > > Contributed-under: TianoCore Contribution Agreement 1.1 > > Signed-off-by: Bob Feng > > Cc: Liming Gao > > --- > > BaseTools/Conf/build_rule.template | 14 +++++++------- > > BaseTools/Source/C/GenFfs/GenFfs.c | 17 +++++++++++++++-- > > BaseTools/Source/Python/AutoGen/GenMake.py | 2 +- > > .../Python/GenFds/GenFdsGlobalVariable.py | 10 +++++++++- > > .../Source/Python/Workspace/DscBuildData.py | 3 +++ > > BaseTools/Source/Python/build/build.py | 10 +++++----- > > 6 files changed, 40 insertions(+), 16 deletions(-) > > > > diff --git a/BaseTools/Conf/build_rule.template > > b/BaseTools/Conf/build_rule.template > > index e56b1d9c59..8cd7d0e445 100755 > > --- a/BaseTools/Conf/build_rule.template > > +++ b/BaseTools/Conf/build_rule.template > > @@ -357,16 +357,16 @@ > > [Dynamic-Library-File] > > > > ?.dll > > > > > > - $(DEBUG_DIR)(+)$(MODULE_NAME).efi > > + $(DEBUG_DIR)(+)$(MODULE_NAME).efi=20 > > + $(OUTPUT_DIR)(+)$(MODULE_NAME).efi > > > > > > "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS) > > - $(CP) ${dst} $(OUTPUT_DIR) > > - $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR) > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi=20 > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR) > > -$(CP) $(DEBUG_DIR)(+)*.pdb $(OUTPUT_DIR) > > > > $(CP) ${src} $(DEBUG_DIR)(+)$(MODULE_NAME).debug > > $(OBJCOPY) --strip-unneeded -R .eh_frame ${src} @@ -376,22 > > +376,22 @@ > > # > > -$(OBJCOPY) $(OBJCOPY_ADDDEBUGFLAG) ${src} > > -$(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).debug > > $(BIN_DIR)(+)$(MODULE_NAME_GUID).debug > > > > "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS) > > - $(CP) ${dst} $(OUTPUT_DIR) > > - $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR) > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi=20 > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR) > > > > > > # tool to convert Mach-O to PE/COFF > > "$(MTOC)" -subsystem $(MODULE_TYPE) $(MTOC_FLAGS) ${src} $(= DEBUG_DIR)(+)$(MODULE_NAME).pecoff > > # create symbol file for GDB debug > > -$(DSYMUTIL) ${src} > > "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} $(DEBUG_DIR)(+)$(MODULE= _NAME).pecoff $(GENFW_FLAGS) > > - $(CP) ${dst} $(OUTPUT_DIR) > > - $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR) > > + $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi=20 > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi > > -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR) > > > > [Dependency-Expression-File] > > > > ?.dxs, ?.Dxs, ?.DXS > > diff --git a/BaseTools/Source/C/GenFfs/GenFfs.c > > b/BaseTools/Source/C/GenFfs/GenFfs.c > > index 02c7ac30f0..ea36b22bef 100644 > > --- a/BaseTools/Source/C/GenFfs/GenFfs.c > > +++ b/BaseTools/Source/C/GenFfs/GenFfs.c > > @@ -17,10 +17,14 @@ WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, = EITHER EXPRESS OR IMPLIED. > > #include > > #include > > #include > > #endif > > > > +#ifdef __GNUC__ > > +#include > > +#endif > > + > > #include > > #include > > #include > > > > #include > > @@ -158,10 +162,12 @@ Returns: > > FileAlign points to file alignment, which only= support\n\ > > the following align: 1,2,4,8,16,128,512,1K,4K,= 32K,64K\n\ > > 128K,256K,512K,1M,2M,4M,8M,16M\n"); > > fprintf (stdout, " -i SectionFile, --sectionfile SectionFile\n\ > > Section file will be contained in this FFS=20 > > file.\n"); > > + fprintf (stdout, " -oi SectionFile, --optionalsectionfile SectionFi= le\n\ > > + If the Section file exists, it will be=20 > > + contained in this FFS file, otherwise, it will be ignored.\n"); > > fprintf (stdout, " -n SectionAlign, --sectionalign SectionAlign\n\ > > SectionAlign points to section alignment, whic= h support\n\ > > the alignment scope 0~16M. If SectionAlign is = specified\n\ > > as 0, tool get alignment value from SectionFil= e. It is\n\ > > specified together with sectionfile to=20 > > point its\n\ @@ -734,19 +740,26 @@ Returns: > > argc -=3D 2; > > argv +=3D 2; > > continue; > > } > > > > - if ((stricmp (argv[0], "-i") =3D=3D 0) || (stricmp (argv[0], "--se= ctionfile") =3D=3D 0)) { > > + if ((stricmp (argv[0], "-oi") =3D=3D 0) || (stricmp (argv[0], > > + "--optionalsectionfile") =3D=3D 0) || (stricmp (argv[0], "-i") =3D=3D= 0)=20 > > + || (stricmp (argv[0], > > "--sectionfile") =3D=3D 0)) { > > // > > // Get Input file name and its alignment > > // > > if (argv[1] =3D=3D NULL || argv[1][0] =3D=3D '-') { > > Error (NULL, 0, 1003, "Invalid option value", "input section f= ile is missing for -i option"); > > goto Finish; > > } > > - > > + if ((stricmp (argv[0], "-oi") =3D=3D 0) || (stricmp (argv[0], "-= -optionalsectionfile") =3D=3D 0) ){ > > + if (-1 =3D=3D access(argv[1] , 0)){ > > + Warning(NULL, 0, 0001, "File is not found.", argv[1]); > > + argc -=3D 2; > > + argv +=3D 2; > > + continue; > > + } > > + } > > // > > // Allocate Input file name buffer and its alignment buffer. > > // > > if ((InputFileNum =3D=3D 0) && (InputFileName =3D=3D NULL)) { > > InputFileName =3D (CHAR8 **) malloc (MAXIMUM_INPUT_FILE_NUM *= =20 > > sizeof (CHAR8 *)); diff --git=20 > > a/BaseTools/Source/Python/AutoGen/GenMake.py > > b/BaseTools/Source/Python/AutoGen/GenMake.py > > index b441817b52..2374bcf550 100644 > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > > @@ -714,11 +714,11 @@ cleanlib: > > > > FfsCmdList =3D Cmd[0] > > for index, Str in enumerate(FfsCmdList): > > if '-o' =3D=3D Str: > > OutputFile =3D FfsCmdList[index + 1] > > - if '-i' =3D=3D Str: > > + if '-i' =3D=3D Str or "-oi" =3D=3D Str: > > if DepsFileList =3D=3D []: > > DepsFileList =3D [FfsCmdList[index + 1]] > > else: > > DepsFileList.append(FfsCmdList[index + 1]) > > DepsFileString =3D ' '.join(DepsFileList).strip() diff=20 > > --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py > > b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py > > index 028bcc480c..b42829b698 100644 > > --- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py > > +++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py > > @@ -33,10 +33,11 @@ from Common.ToolDefClassObject import=20 > > ToolDefClassObject, ToolDefDict from AutoGen.BuildEngine import=20 > > BuildRule import Common.DataType as DataType from Common.Misc=20 > > import PathClass from Common.LongFilePathSupport import=20 > > OpenLongFilePath as open from Common.MultipleWorkspace import=20 > > MultipleWorkspace as mws > > +import Common.GlobalData as GlobalData > > > > ## Global variables > > # > > # > > class GenFdsGlobalVariable: > > @@ -499,10 +500,14 @@ class GenFdsGlobalVariable: > > Cmd +=3D ("-o", Output) > > Cmd +=3D Input > > > > SaveFileOnChange(CommandFile, ' '.join(Cmd), False) > > if IsMakefile: > > + if GlobalData.gGlobalDefines.get("FAMILY") =3D=3D "MSF= T": > > + Cmd =3D ['if', 'exist', Input[0]] + Cmd > > + else: > > + Cmd =3D ['test', '-e', Input[0], "&&"] + Cmd > > if ' '.join(Cmd).strip() not in GenFdsGlobalVariable.S= ecCmdList: > > GenFdsGlobalVariable.SecCmdList.append(' '.join(Cm= d).strip()) > > elif GenFdsGlobalVariable.NeedsUpdate(Output, list(Input) = + [CommandFile]): > > GenFdsGlobalVariable.DebugLogger(EdkLogger.DEBUG_5,=20 > > "%s needs update because of newer %s" % (Output, > Input)) > > GenFdsGlobalVariable.CallExternalTool(Cmd, "Failed=20 > > to generate section") @@ -540,11 +545,14 @@ class GenFdsGlobalVariable: > > Align =3D mFfsValidAlign[index + 1] > > Cmd +=3D ("-a", Align) > > > > Cmd +=3D ("-o", Output) > > for I in range(0, len(Input)): > > - Cmd +=3D ("-i", Input[I]) > > + if MakefilePath: > > + Cmd +=3D ("-oi", Input[I]) > > + else: > > + Cmd +=3D ("-i", Input[I]) > > if SectionAlign and SectionAlign[I]: > > Cmd +=3D ("-n", SectionAlign[I]) > > > > CommandFile =3D Output + '.txt' > > SaveFileOnChange(CommandFile, ' '.join(Cmd), False) diff=20 > > --git a/BaseTools/Source/Python/Workspace/DscBuildData.py > > b/BaseTools/Source/Python/Workspace/DscBuildData.py > > index 129c0c950b..ac95c7df17 100644 > > --- a/BaseTools/Source/Python/Workspace/DscBuildData.py > > +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py > > @@ -318,10 +318,13 @@ class DscBuildData(PlatformBuildClassObject): > > > > ## Get architecture > > @property > > def Arch(self): > > return self._Arch > > + @property > > + def Dir(self): > > + return self.MetaFile.Dir > > > > ## Retrieve all information in [Defines] section > > # > > # (Retrieving all [Defines] information in one-shot is just to s= ave time.) > > # > > diff --git a/BaseTools/Source/Python/build/build.py > > b/BaseTools/Source/Python/build/build.py > > index de641fb452..41d405e6ff 100644 > > --- a/BaseTools/Source/Python/build/build.py > > +++ b/BaseTools/Source/Python/build/build.py > > @@ -1667,11 +1667,11 @@ class Build(): > > self.Progress.Stop("done!") > > > > # Add ffs build to makefile > > CmdListDict =3D {} > > if GlobalData.gEnableGenfdsMultiThread and self.Fdf: > > - CmdListDict =3D self._GenFfsCmd() > > + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) > > > > for Arch in Wa.ArchList: > > GlobalData.gGlobalDefines['ARCH'] =3D Arch > > Pa =3D PlatformAutoGen(Wa, self.PlatformFile, Buil= dTarget, ToolChain, Arch) > > for Module in Pa.Platform.Modules: > > @@ -1760,11 +1760,11 @@ class Build(): > > self.LoadFixAddress =3D Wa.Platform.LoadFixAddress > > Wa.CreateMakeFile(False) > > # Add ffs build to makefile > > CmdListDict =3D None > > if GlobalData.gEnableGenfdsMultiThread and self.Fdf: > > - CmdListDict =3D self._GenFfsCmd() > > + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) > > self.Progress.Stop("done!") > > MaList =3D [] > > ExitFlag =3D threading.Event() > > ExitFlag.clear() > > self.AutoGenTime +=3D int(round((time.time() - > > WorkspaceAutoGenTime))) @@ -1879,15 +1879,15 @@ class Build(): > > # > > # Save MAP buffer into MAP file. > > # > > self._SaveMapFile (MapBuffer, Wa) > > > > - def _GenFfsCmd(self): > > + def _GenFfsCmd(self,ArchList): > > # convert dictionary of Cmd:(Inf,Arch) > > # to a new dictionary of (Inf,Arch):Cmd,Cmd,Cmd... > > CmdSetDict =3D defaultdict(set) > > - GenFfsDict =3D GenFds.GenFfsMakefile('', GlobalData.gFdfParser= , self, self.ArchList, GlobalData) > > + GenFfsDict =3D GenFds.GenFfsMakefile('',=20 > > + GlobalData.gFdfParser, self, ArchList, GlobalData) > > for Cmd in GenFfsDict: > > tmpInf, tmpArch =3D GenFfsDict[Cmd] > > CmdSetDict[tmpInf, tmpArch].add(Cmd) > > return CmdSetDict > > > > @@ -1927,11 +1927,11 @@ class Build(): > > Wa.CreateMakeFile(False) > > > > # Add ffs build to makefile > > CmdListDict =3D None > > if GlobalData.gEnableGenfdsMultiThread and self.Fdf: > > - CmdListDict =3D self._GenFfsCmd() > > + CmdListDict =3D self._GenFfsCmd(Wa.ArchList) > > > > # multi-thread exit flag > > ExitFlag =3D threading.Event() > > ExitFlag.clear() > > self.AutoGenTime +=3D int(round((time.time() - > > WorkspaceAutoGenTime))) > > -- > > 2.20.1.windows.1