Re: [Patch] BaseTools: Fixed issue in MultiThread Genfds function

["Feng, Bob C" <bob.c.feng@intel.com>]

BZ for remove additional copy action. 
https://bugzilla.tianocore.org/show_bug.cgi?id=1673


-----Original Message-----
From: Gao, Liming 
Sent: Friday, March 29, 2019 10:09 PM
To: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org
Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds function

Yes. Please submit one BZ for it. 

> -----Original Message-----
> From: Feng, Bob C
> Sent: Friday, March 29, 2019 8:40 PM
> To: Gao, Liming <liming.gao@intel.com>; edk2-devel@lists.01.org
> Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds 
> function
> 
> That will impact the original GenFds since the command is generate by 
> reusing the GenFds functions. What about enter a new BZ to do that change?
> 
> Thanks,
> Bob
> 
> -----Original Message-----
> From: Gao, Liming
> Sent: Friday, March 29, 2019 8:33 PM
> To: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org
> Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds 
> function
> 
> Bob:
>   Could you update the rule to depend on EFI image from DEBUG_DIR? I 
> would like to remove the additional copy from DEBUG dir to OUTPUT dir and save the disk and performance.
> 
> $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32 : $(DEBUG_DIR)\LogoDxe.efi
>  	GenSec -s EFI_SECTION_PE32 -o 
> $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32 
> $(DEBUG_DIR)\LogoDxe.efi
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: Feng, Bob C
> > Sent: Friday, March 29, 2019 8:00 PM
> > To: Gao, Liming <liming.gao@intel.com>; edk2-devel@lists.01.org
> > Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds 
> > function
> >
> > That's to fix the issue that make will fail if enable multiple thread genfds.
> >
> > For example, If enable Multiple thread Genfds, there will be GenSec command in the Makefile like below.
> >
> > $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32 : $(OUTPUT_DIR)\LogoDxe.efi
> > 	GenSec -s EFI_SECTION_PE32 -o
> > $(FFS_OUTPUT_DIR)\$(MODULE_GUID)SEC2.1.1.1.pe32
> > $(OUTPUT_DIR)\LogoDxe.efi
> >
> >
> > The $(OUTPUT_DIR)\LogoDxe.efi is the dependency, but there is no make rule for $(OUTPUT_DIR)\LogoDxe.efi generated in the Makefile.
> > Make program does not know how to make $(OUTPUT_DIR)\LogoDxe.efi.  Make fails for this case.
> >
> > In the build_rule.txt, the $(OUTPUT_DIR)\ )(+)$(MODULE_NAME).efi   is generated by the command $(CP)
> > $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR) under 
> > Dynamic-Library-File section, so $(OUTPUT_DIR)\ 
> > )(+)$(MODULE_NAME).efi should also be a output for 
> > Dynamic-Library-File
> >
> > Thanks,
> > Bob
> >
> > -----Original Message-----
> > From: Gao, Liming
> > Sent: Friday, March 29, 2019 7:42 PM
> > To: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org
> > Subject: RE: [Patch] BaseTools: Fixed issue in MultiThread Genfds 
> > function
> >
> > Bob:
> >   Could you list the more information on why update build_rule.txt?
> >
> > > -----Original Message-----
> > > From: Feng, Bob C
> > > Sent: Friday, March 29, 2019 7:07 PM
> > > To: edk2-devel@lists.01.org
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming 
> > > <liming.gao@intel.com>
> > > Subject: [Patch] BaseTools: Fixed issue in MultiThread Genfds 
> > > function
> > >
> > > https://bugzilla.tianocore.org/show_bug.cgi?id=1450
> > > In the Multiple thread Genfds feature, build tool generates 
> > > GenSec, GenFFS command in Makefile.
> > >
> > > The Non-Hii Driver does not generate .offset file for uni string 
> > > offset, but the build tool has not knowledge about this in autogen 
> > > phase. So in this patch, I add a check in Makefile for GenSec command.
> > > If the GenSec input file does not exist, the GenSec will not be 
> > > called. And if GenSec command is not called, its output file, 
> > > which is also the input file of GenFfs command, will also not 
> > > exist.So for GenFfs command, I add a new command parameter -oi 
> > > which means the input file is an optional input file which would 
> > > not exist. so that I can generate GenFfs command with "-oi" parameter in Makefile.
> > >
> > > Contributed-under: TianoCore Contribution Agreement 1.1
> > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > Cc: Liming Gao <liming.gao@intel.com>
> > > ---
> > >  BaseTools/Conf/build_rule.template              | 14 +++++++-------
> > >  BaseTools/Source/C/GenFfs/GenFfs.c              | 17 +++++++++++++++--
> > >  BaseTools/Source/Python/AutoGen/GenMake.py      |  2 +-
> > >  .../Python/GenFds/GenFdsGlobalVariable.py       | 10 +++++++++-
> > >  .../Source/Python/Workspace/DscBuildData.py     |  3 +++
> > >  BaseTools/Source/Python/build/build.py          | 10 +++++-----
> > >  6 files changed, 40 insertions(+), 16 deletions(-)
> > >
> > > diff --git a/BaseTools/Conf/build_rule.template
> > > b/BaseTools/Conf/build_rule.template
> > > index e56b1d9c59..8cd7d0e445 100755
> > > --- a/BaseTools/Conf/build_rule.template
> > > +++ b/BaseTools/Conf/build_rule.template
> > > @@ -357,16 +357,16 @@
> > >  [Dynamic-Library-File]
> > >      <InputFile>
> > >          ?.dll
> > >
> > >      <OutputFile>
> > > -        $(DEBUG_DIR)(+)$(MODULE_NAME).efi
> > > +        $(DEBUG_DIR)(+)$(MODULE_NAME).efi 
> > > + $(OUTPUT_DIR)(+)$(MODULE_NAME).efi
> > >
> > >      <Command.MSFT, Command.INTEL, Command.RVCT>
> > >          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS)
> > > -        $(CP) ${dst} $(OUTPUT_DIR)
> > > -        $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR)
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi 
> > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > >          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
> > >          -$(CP) $(DEBUG_DIR)(+)*.pdb $(OUTPUT_DIR)
> > >      <Command.GCC>
> > >          $(CP) ${src} $(DEBUG_DIR)(+)$(MODULE_NAME).debug
> > >          $(OBJCOPY) --strip-unneeded -R .eh_frame ${src} @@ 
> > > -376,22
> > > +376,22 @@
> > >          #
> > >          -$(OBJCOPY) $(OBJCOPY_ADDDEBUGFLAG) ${src}
> > >          -$(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).debug
> > > $(BIN_DIR)(+)$(MODULE_NAME_GUID).debug
> > >
> > >          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS)
> > > -        $(CP) ${dst} $(OUTPUT_DIR)
> > > -        $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR)
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi 
> > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > >          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
> > >
> > >      <Command.XCODE>
> > >          # tool to convert Mach-O to PE/COFF
> > >          "$(MTOC)" -subsystem $(MODULE_TYPE)  $(MTOC_FLAGS)  ${src}  $(DEBUG_DIR)(+)$(MODULE_NAME).pecoff
> > >          # create symbol file for GDB debug
> > >          -$(DSYMUTIL) ${src}
> > >          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} $(DEBUG_DIR)(+)$(MODULE_NAME).pecoff $(GENFW_FLAGS)
> > > -        $(CP) ${dst} $(OUTPUT_DIR)
> > > -        $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi $(OUTPUT_DIR)
> > > +        $(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).efi 
> > > + $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
> > >          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
> > >
> > >  [Dependency-Expression-File]
> > >      <InputFile>
> > >          ?.dxs, ?.Dxs, ?.DXS
> > > diff --git a/BaseTools/Source/C/GenFfs/GenFfs.c
> > > b/BaseTools/Source/C/GenFfs/GenFfs.c
> > > index 02c7ac30f0..ea36b22bef 100644
> > > --- a/BaseTools/Source/C/GenFfs/GenFfs.c
> > > +++ b/BaseTools/Source/C/GenFfs/GenFfs.c
> > > @@ -17,10 +17,14 @@ WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
> > >  #include <io.h>
> > >  #include <sys/types.h>
> > >  #include <sys/stat.h>
> > >  #endif
> > >
> > > +#ifdef __GNUC__
> > > +#include <unistd.h>
> > > +#endif
> > > +
> > >  #include <stdio.h>
> > >  #include <stdlib.h>
> > >  #include <string.h>
> > >
> > >  #include <Common/UefiBaseTypes.h> @@ -158,10 +162,12 @@ Returns:
> > >                          FileAlign points to file alignment, which only support\n\
> > >                          the following align: 1,2,4,8,16,128,512,1K,4K,32K,64K\n\
> > >                          128K,256K,512K,1M,2M,4M,8M,16M\n");
> > >    fprintf (stdout, "  -i SectionFile, --sectionfile SectionFile\n\
> > >                          Section file will be contained in this 
> > > FFS file.\n");
> > > +  fprintf (stdout, "  -oi SectionFile, --optionalsectionfile SectionFile\n\
> > > +                        If the Section file exists, it will be 
> > > + contained in this FFS file, otherwise, it will be ignored.\n");
> > >    fprintf (stdout, "  -n SectionAlign, --sectionalign SectionAlign\n\
> > >                          SectionAlign points to section alignment, which support\n\
> > >                          the alignment scope 0~16M. If SectionAlign is specified\n\
> > >                          as 0, tool get alignment value from SectionFile. It is\n\
> > >                          specified together with sectionfile to 
> > > point its\n\ @@ -734,19 +740,26 @@ Returns:
> > >        argc -= 2;
> > >        argv += 2;
> > >        continue;
> > >      }
> > >
> > > -    if ((stricmp (argv[0], "-i") == 0) || (stricmp (argv[0], "--sectionfile") == 0)) {
> > > +    if ((stricmp (argv[0], "-oi") == 0) || (stricmp (argv[0],
> > > + "--optionalsectionfile") == 0) || (stricmp (argv[0], "-i") == 0)
> > > + || (stricmp (argv[0],
> > > "--sectionfile") == 0)) {
> > >        //
> > >        // Get Input file name and its alignment
> > >        //
> > >        if (argv[1] == NULL || argv[1][0] == '-') {
> > >          Error (NULL, 0, 1003, "Invalid option value", "input section file is missing for -i option");
> > >          goto Finish;
> > >        }
> > > -
> > > +      if ((stricmp (argv[0], "-oi") == 0) || (stricmp (argv[0], "--optionalsectionfile") == 0) ){
> > > +        if (-1 == access(argv[1] , 0)){
> > > +          Warning(NULL, 0, 0001, "File is not found.", argv[1]);
> > > +          argc -= 2;
> > > +          argv += 2;
> > > +          continue;
> > > +        }
> > > +      }
> > >        //
> > >        // Allocate Input file name buffer and its alignment buffer.
> > >        //
> > >        if ((InputFileNum == 0) && (InputFileName == NULL)) {
> > >          InputFileName = (CHAR8 **) malloc (MAXIMUM_INPUT_FILE_NUM 
> > > * sizeof (CHAR8 *)); diff --git 
> > > a/BaseTools/Source/Python/AutoGen/GenMake.py
> > > b/BaseTools/Source/Python/AutoGen/GenMake.py
> > > index b441817b52..2374bcf550 100644
> > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> > > @@ -714,11 +714,11 @@ cleanlib:
> > >
> > >              FfsCmdList = Cmd[0]
> > >              for index, Str in enumerate(FfsCmdList):
> > >                  if '-o' == Str:
> > >                      OutputFile = FfsCmdList[index + 1]
> > > -                if '-i' == Str:
> > > +                if '-i' == Str or "-oi" == Str:
> > >                      if DepsFileList == []:
> > >                          DepsFileList = [FfsCmdList[index + 1]]
> > >                      else:
> > >                          DepsFileList.append(FfsCmdList[index + 1])
> > >              DepsFileString = ' '.join(DepsFileList).strip() diff 
> > > --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> > > b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> > > index 028bcc480c..b42829b698 100644
> > > --- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> > > +++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> > > @@ -33,10 +33,11 @@ from Common.ToolDefClassObject import 
> > > ToolDefClassObject, ToolDefDict  from AutoGen.BuildEngine import 
> > > BuildRule  import Common.DataType as DataType  from Common.Misc 
> > > import PathClass  from Common.LongFilePathSupport import 
> > > OpenLongFilePath as open  from Common.MultipleWorkspace import 
> > > MultipleWorkspace as mws
> > > +import Common.GlobalData as GlobalData
> > >
> > >  ## Global variables
> > >  #
> > >  #
> > >  class GenFdsGlobalVariable:
> > > @@ -499,10 +500,14 @@ class GenFdsGlobalVariable:
> > >              Cmd += ("-o", Output)
> > >              Cmd += Input
> > >
> > >              SaveFileOnChange(CommandFile, ' '.join(Cmd), False)
> > >              if IsMakefile:
> > > +                if GlobalData.gGlobalDefines.get("FAMILY") == "MSFT":
> > > +                    Cmd = ['if', 'exist', Input[0]] + Cmd
> > > +                else:
> > > +                    Cmd = ['test', '-e', Input[0], "&&"] + Cmd
> > >                  if ' '.join(Cmd).strip() not in GenFdsGlobalVariable.SecCmdList:
> > >                      GenFdsGlobalVariable.SecCmdList.append(' '.join(Cmd).strip())
> > >              elif GenFdsGlobalVariable.NeedsUpdate(Output, list(Input) + [CommandFile]):
> > >                  
> > > GenFdsGlobalVariable.DebugLogger(EdkLogger.DEBUG_5,
> > > "%s needs update because of newer %s" % (Output,
> > Input))
> > >                  GenFdsGlobalVariable.CallExternalTool(Cmd, 
> > > "Failed to generate section") @@ -540,11 +545,14 @@ class GenFdsGlobalVariable:
> > >                  Align = mFfsValidAlign[index + 1]
> > >              Cmd += ("-a", Align)
> > >
> > >          Cmd += ("-o", Output)
> > >          for I in range(0, len(Input)):
> > > -            Cmd += ("-i", Input[I])
> > > +            if MakefilePath:
> > > +                Cmd += ("-oi", Input[I])
> > > +            else:
> > > +                Cmd += ("-i", Input[I])
> > >              if SectionAlign and SectionAlign[I]:
> > >                  Cmd += ("-n", SectionAlign[I])
> > >
> > >          CommandFile = Output + '.txt'
> > >          SaveFileOnChange(CommandFile, ' '.join(Cmd), False) diff 
> > > --git a/BaseTools/Source/Python/Workspace/DscBuildData.py
> > > b/BaseTools/Source/Python/Workspace/DscBuildData.py
> > > index 129c0c950b..ac95c7df17 100644
> > > --- a/BaseTools/Source/Python/Workspace/DscBuildData.py
> > > +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py
> > > @@ -318,10 +318,13 @@ class DscBuildData(PlatformBuildClassObject):
> > >
> > >      ## Get architecture
> > >      @property
> > >      def Arch(self):
> > >          return self._Arch
> > > +    @property
> > > +    def Dir(self):
> > > +        return self.MetaFile.Dir
> > >
> > >      ## Retrieve all information in [Defines] section
> > >      #
> > >      #   (Retrieving all [Defines] information in one-shot is just to save time.)
> > >      #
> > > diff --git a/BaseTools/Source/Python/build/build.py
> > > b/BaseTools/Source/Python/build/build.py
> > > index de641fb452..41d405e6ff 100644
> > > --- a/BaseTools/Source/Python/build/build.py
> > > +++ b/BaseTools/Source/Python/build/build.py
> > > @@ -1667,11 +1667,11 @@ class Build():
> > >                  self.Progress.Stop("done!")
> > >
> > >                  # Add ffs build to makefile
> > >                  CmdListDict = {}
> > >                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> > > -                    CmdListDict = self._GenFfsCmd()
> > > +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
> > >
> > >                  for Arch in Wa.ArchList:
> > >                      GlobalData.gGlobalDefines['ARCH'] = Arch
> > >                      Pa = PlatformAutoGen(Wa, self.PlatformFile, BuildTarget, ToolChain, Arch)
> > >                      for Module in Pa.Platform.Modules:
> > > @@ -1760,11 +1760,11 @@ class Build():
> > >                  self.LoadFixAddress = Wa.Platform.LoadFixAddress
> > >                  Wa.CreateMakeFile(False)
> > >                  # Add ffs build to makefile
> > >                  CmdListDict = None
> > >                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> > > -                    CmdListDict = self._GenFfsCmd()
> > > +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
> > >                  self.Progress.Stop("done!")
> > >                  MaList = []
> > >                  ExitFlag = threading.Event()
> > >                  ExitFlag.clear()
> > >                  self.AutoGenTime += int(round((time.time() -
> > > WorkspaceAutoGenTime))) @@ -1879,15 +1879,15 @@ class Build():
> > >                      #
> > >                      # Save MAP buffer into MAP file.
> > >                      #
> > >                      self._SaveMapFile (MapBuffer, Wa)
> > >
> > > -    def _GenFfsCmd(self):
> > > +    def _GenFfsCmd(self,ArchList):
> > >          # convert dictionary of Cmd:(Inf,Arch)
> > >          # to a new dictionary of (Inf,Arch):Cmd,Cmd,Cmd...
> > >          CmdSetDict = defaultdict(set)
> > > -        GenFfsDict = GenFds.GenFfsMakefile('', GlobalData.gFdfParser, self, self.ArchList, GlobalData)
> > > +        GenFfsDict = GenFds.GenFfsMakefile('', 
> > > + GlobalData.gFdfParser, self, ArchList, GlobalData)
> > >          for Cmd in GenFfsDict:
> > >              tmpInf, tmpArch = GenFfsDict[Cmd]
> > >              CmdSetDict[tmpInf, tmpArch].add(Cmd)
> > >          return CmdSetDict
> > >
> > > @@ -1927,11 +1927,11 @@ class Build():
> > >                  Wa.CreateMakeFile(False)
> > >
> > >                  # Add ffs build to makefile
> > >                  CmdListDict = None
> > >                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> > > -                    CmdListDict = self._GenFfsCmd()
> > > +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
> > >
> > >                  # multi-thread exit flag
> > >                  ExitFlag = threading.Event()
> > >                  ExitFlag.clear()
> > >                  self.AutoGenTime += int(round((time.time() -
> > > WorkspaceAutoGenTime)))
> > > --
> > > 2.20.1.windows.1