Re: [Patch V2] BaseTools: Fixed issue in MultiThread Genfds function

["Feng, Bob C" <bob.c.feng@intel.com>]

If user does not use multiple thread genfds, there will be no problem. 

-----Original Message-----
From: Gao, Liming 
Sent: Wednesday, April 3, 2019 10:49 AM
To: Feng, Bob C <bob.c.feng@intel.com>
Cc: edk2-devel@lists.01.org
Subject: RE: [Patch V2] BaseTools: Fixed issue in MultiThread Genfds function

Bob:
  Thanks for your updating. If user only update BaseTools, and don't update Conf\build_rule.txt. What problem will happen?

Thanks
Liming
> -----Original Message-----
> From: Feng, Bob C
> Sent: Wednesday, April 3, 2019 10:17 AM
> To: edk2-devel@lists.01.org
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming 
> <liming.gao@intel.com>
> Subject: [Patch V2] BaseTools: Fixed issue in MultiThread Genfds 
> function
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1450
> In the Multiple thread Genfds feature, build tool generates GenSec, 
> GenFFS command in Makefile.
> 
> The Non-Hii Driver does not generate .offset file for uni string 
> offset, but the build tool has not knowledge about this in autogen 
> phase. So in this patch, I add a check in Makefile for GenSec command. 
> If the GenSec input file does not exist, the GenSec will not be 
> called. And if GenSec command is not called, its output file, which is 
> also the input file of GenFfs command, will also not exist.So for 
> GenFfs command, I add a new command parameter -oi which means the 
> input file is an optional input file which would not exist. so that I 
> can generate GenFfs command with "-oi" parameter in Makefile.
> 
> Contributed-under: TianoCore Contribution Agreement 1.1
> Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> ---
>  BaseTools/Conf/build_rule.template              |  8 ++++----
>  BaseTools/Source/C/GenFfs/GenFfs.c              | 17 +++++++++++++++--
>  BaseTools/Source/Python/AutoGen/GenMake.py      |  2 +-
>  .../Python/GenFds/GenFdsGlobalVariable.py       | 10 +++++++++-
>  .../Source/Python/Workspace/DscBuildData.py     |  3 +++
>  BaseTools/Source/Python/build/build.py          | 10 +++++-----
>  6 files changed, 37 insertions(+), 13 deletions(-)
> 
> diff --git a/BaseTools/Conf/build_rule.template 
> b/BaseTools/Conf/build_rule.template
> index e56b1d9c59..17e7353063 100755
> --- a/BaseTools/Conf/build_rule.template
> +++ b/BaseTools/Conf/build_rule.template
> @@ -357,15 +357,15 @@
>  [Dynamic-Library-File]
>      <InputFile>
>          ?.dll
> 
>      <OutputFile>
> -        $(DEBUG_DIR)(+)$(MODULE_NAME).efi
> +        $(OUTPUT_DIR)(+)$(MODULE_NAME).efi
> 
>      <Command.MSFT, Command.INTEL, Command.RVCT>
>          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS)
> -        $(CP) ${dst} $(OUTPUT_DIR)
> +        $(CP) ${dst} $(DEBUG_DIR)
>          $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
>          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
>          -$(CP) $(DEBUG_DIR)(+)*.pdb $(OUTPUT_DIR)
>      <Command.GCC>
>          $(CP) ${src} $(DEBUG_DIR)(+)$(MODULE_NAME).debug
> @@ -376,21 +376,21 @@
>          #
>          -$(OBJCOPY) $(OBJCOPY_ADDDEBUGFLAG) ${src}
>          -$(CP) $(DEBUG_DIR)(+)$(MODULE_NAME).debug 
> $(BIN_DIR)(+)$(MODULE_NAME_GUID).debug
> 
>          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} ${src} $(GENFW_FLAGS)
> -        $(CP) ${dst} $(OUTPUT_DIR)
> +        $(CP) ${dst} $(DEBUG_DIR)
>          $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
>          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
> 
>      <Command.XCODE>
>          # tool to convert Mach-O to PE/COFF
>          "$(MTOC)" -subsystem $(MODULE_TYPE)  $(MTOC_FLAGS)  ${src}  $(DEBUG_DIR)(+)$(MODULE_NAME).pecoff
>          # create symbol file for GDB debug
>          -$(DSYMUTIL) ${src}
>          "$(GENFW)" -e $(MODULE_TYPE) -o ${dst} $(DEBUG_DIR)(+)$(MODULE_NAME).pecoff $(GENFW_FLAGS)
> -        $(CP) ${dst} $(OUTPUT_DIR)
> +        $(CP) ${dst} $(DEBUG_DIR)
>          $(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).efi
>          -$(CP) $(DEBUG_DIR)(+)*.map $(OUTPUT_DIR)
> 
>  [Dependency-Expression-File]
>      <InputFile>
> diff --git a/BaseTools/Source/C/GenFfs/GenFfs.c 
> b/BaseTools/Source/C/GenFfs/GenFfs.c
> index 02c7ac30f0..ea36b22bef 100644
> --- a/BaseTools/Source/C/GenFfs/GenFfs.c
> +++ b/BaseTools/Source/C/GenFfs/GenFfs.c
> @@ -17,10 +17,14 @@ WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
>  #include <io.h>
>  #include <sys/types.h>
>  #include <sys/stat.h>
>  #endif
> 
> +#ifdef __GNUC__
> +#include <unistd.h>
> +#endif
> +
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
> 
>  #include <Common/UefiBaseTypes.h>
> @@ -158,10 +162,12 @@ Returns:
>                          FileAlign points to file alignment, which only support\n\
>                          the following align: 1,2,4,8,16,128,512,1K,4K,32K,64K\n\
>                          128K,256K,512K,1M,2M,4M,8M,16M\n");
>    fprintf (stdout, "  -i SectionFile, --sectionfile SectionFile\n\
>                          Section file will be contained in this FFS 
> file.\n");
> +  fprintf (stdout, "  -oi SectionFile, --optionalsectionfile SectionFile\n\
> +                        If the Section file exists, it will be 
> + contained in this FFS file, otherwise, it will be ignored.\n");
>    fprintf (stdout, "  -n SectionAlign, --sectionalign SectionAlign\n\
>                          SectionAlign points to section alignment, which support\n\
>                          the alignment scope 0~16M. If SectionAlign is specified\n\
>                          as 0, tool get alignment value from SectionFile. It is\n\
>                          specified together with sectionfile to point 
> its\n\ @@ -734,19 +740,26 @@ Returns:
>        argc -= 2;
>        argv += 2;
>        continue;
>      }
> 
> -    if ((stricmp (argv[0], "-i") == 0) || (stricmp (argv[0], "--sectionfile") == 0)) {
> +    if ((stricmp (argv[0], "-oi") == 0) || (stricmp (argv[0], 
> + "--optionalsectionfile") == 0) || (stricmp (argv[0], "-i") == 0) || 
> + (stricmp (argv[0],
> "--sectionfile") == 0)) {
>        //
>        // Get Input file name and its alignment
>        //
>        if (argv[1] == NULL || argv[1][0] == '-') {
>          Error (NULL, 0, 1003, "Invalid option value", "input section file is missing for -i option");
>          goto Finish;
>        }
> -
> +      if ((stricmp (argv[0], "-oi") == 0) || (stricmp (argv[0], "--optionalsectionfile") == 0) ){
> +        if (-1 == access(argv[1] , 0)){
> +          Warning(NULL, 0, 0001, "File is not found.", argv[1]);
> +          argc -= 2;
> +          argv += 2;
> +          continue;
> +        }
> +      }
>        //
>        // Allocate Input file name buffer and its alignment buffer.
>        //
>        if ((InputFileNum == 0) && (InputFileName == NULL)) {
>          InputFileName = (CHAR8 **) malloc (MAXIMUM_INPUT_FILE_NUM * 
> sizeof (CHAR8 *)); diff --git 
> a/BaseTools/Source/Python/AutoGen/GenMake.py 
> b/BaseTools/Source/Python/AutoGen/GenMake.py
> index b441817b52..2374bcf550 100644
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -714,11 +714,11 @@ cleanlib:
> 
>              FfsCmdList = Cmd[0]
>              for index, Str in enumerate(FfsCmdList):
>                  if '-o' == Str:
>                      OutputFile = FfsCmdList[index + 1]
> -                if '-i' == Str:
> +                if '-i' == Str or "-oi" == Str:
>                      if DepsFileList == []:
>                          DepsFileList = [FfsCmdList[index + 1]]
>                      else:
>                          DepsFileList.append(FfsCmdList[index + 1])
>              DepsFileString = ' '.join(DepsFileList).strip() diff 
> --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py 
> b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> index 028bcc480c..b42829b698 100644
> --- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> +++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
> @@ -33,10 +33,11 @@ from Common.ToolDefClassObject import 
> ToolDefClassObject, ToolDefDict  from AutoGen.BuildEngine import 
> BuildRule  import Common.DataType as DataType  from Common.Misc import 
> PathClass  from Common.LongFilePathSupport import OpenLongFilePath as 
> open  from Common.MultipleWorkspace import MultipleWorkspace as mws
> +import Common.GlobalData as GlobalData
> 
>  ## Global variables
>  #
>  #
>  class GenFdsGlobalVariable:
> @@ -499,10 +500,14 @@ class GenFdsGlobalVariable:
>              Cmd += ("-o", Output)
>              Cmd += Input
> 
>              SaveFileOnChange(CommandFile, ' '.join(Cmd), False)
>              if IsMakefile:
> +                if GlobalData.gGlobalDefines.get("FAMILY") == "MSFT":
> +                    Cmd = ['if', 'exist', Input[0]] + Cmd
> +                else:
> +                    Cmd = ['test', '-e', Input[0], "&&"] + Cmd
>                  if ' '.join(Cmd).strip() not in GenFdsGlobalVariable.SecCmdList:
>                      GenFdsGlobalVariable.SecCmdList.append(' '.join(Cmd).strip())
>              elif GenFdsGlobalVariable.NeedsUpdate(Output, list(Input) + [CommandFile]):
>                  GenFdsGlobalVariable.DebugLogger(EdkLogger.DEBUG_5, "%s needs update because of newer %s" % (Output, Input))
>                  GenFdsGlobalVariable.CallExternalTool(Cmd, "Failed to 
> generate section") @@ -540,11 +545,14 @@ class GenFdsGlobalVariable:
>                  Align = mFfsValidAlign[index + 1]
>              Cmd += ("-a", Align)
> 
>          Cmd += ("-o", Output)
>          for I in range(0, len(Input)):
> -            Cmd += ("-i", Input[I])
> +            if MakefilePath:
> +                Cmd += ("-oi", Input[I])
> +            else:
> +                Cmd += ("-i", Input[I])
>              if SectionAlign and SectionAlign[I]:
>                  Cmd += ("-n", SectionAlign[I])
> 
>          CommandFile = Output + '.txt'
>          SaveFileOnChange(CommandFile, ' '.join(Cmd), False) diff 
> --git a/BaseTools/Source/Python/Workspace/DscBuildData.py 
> b/BaseTools/Source/Python/Workspace/DscBuildData.py
> index 129c0c950b..ac95c7df17 100644
> --- a/BaseTools/Source/Python/Workspace/DscBuildData.py
> +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py
> @@ -318,10 +318,13 @@ class DscBuildData(PlatformBuildClassObject):
> 
>      ## Get architecture
>      @property
>      def Arch(self):
>          return self._Arch
> +    @property
> +    def Dir(self):
> +        return self.MetaFile.Dir
> 
>      ## Retrieve all information in [Defines] section
>      #
>      #   (Retrieving all [Defines] information in one-shot is just to save time.)
>      #
> diff --git a/BaseTools/Source/Python/build/build.py 
> b/BaseTools/Source/Python/build/build.py
> index de641fb452..41d405e6ff 100644
> --- a/BaseTools/Source/Python/build/build.py
> +++ b/BaseTools/Source/Python/build/build.py
> @@ -1667,11 +1667,11 @@ class Build():
>                  self.Progress.Stop("done!")
> 
>                  # Add ffs build to makefile
>                  CmdListDict = {}
>                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> -                    CmdListDict = self._GenFfsCmd()
> +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
> 
>                  for Arch in Wa.ArchList:
>                      GlobalData.gGlobalDefines['ARCH'] = Arch
>                      Pa = PlatformAutoGen(Wa, self.PlatformFile, BuildTarget, ToolChain, Arch)
>                      for Module in Pa.Platform.Modules:
> @@ -1760,11 +1760,11 @@ class Build():
>                  self.LoadFixAddress = Wa.Platform.LoadFixAddress
>                  Wa.CreateMakeFile(False)
>                  # Add ffs build to makefile
>                  CmdListDict = None
>                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> -                    CmdListDict = self._GenFfsCmd()
> +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
>                  self.Progress.Stop("done!")
>                  MaList = []
>                  ExitFlag = threading.Event()
>                  ExitFlag.clear()
>                  self.AutoGenTime += int(round((time.time() - 
> WorkspaceAutoGenTime))) @@ -1879,15 +1879,15 @@ class Build():
>                      #
>                      # Save MAP buffer into MAP file.
>                      #
>                      self._SaveMapFile (MapBuffer, Wa)
> 
> -    def _GenFfsCmd(self):
> +    def _GenFfsCmd(self,ArchList):
>          # convert dictionary of Cmd:(Inf,Arch)
>          # to a new dictionary of (Inf,Arch):Cmd,Cmd,Cmd...
>          CmdSetDict = defaultdict(set)
> -        GenFfsDict = GenFds.GenFfsMakefile('', GlobalData.gFdfParser, self, self.ArchList, GlobalData)
> +        GenFfsDict = GenFds.GenFfsMakefile('', GlobalData.gFdfParser, 
> + self, ArchList, GlobalData)
>          for Cmd in GenFfsDict:
>              tmpInf, tmpArch = GenFfsDict[Cmd]
>              CmdSetDict[tmpInf, tmpArch].add(Cmd)
>          return CmdSetDict
> 
> @@ -1927,11 +1927,11 @@ class Build():
>                  Wa.CreateMakeFile(False)
> 
>                  # Add ffs build to makefile
>                  CmdListDict = None
>                  if GlobalData.gEnableGenfdsMultiThread and self.Fdf:
> -                    CmdListDict = self._GenFfsCmd()
> +                    CmdListDict = self._GenFfsCmd(Wa.ArchList)
> 
>                  # multi-thread exit flag
>                  ExitFlag = threading.Event()
>                  ExitFlag.clear()
>                  self.AutoGenTime += int(round((time.time() - 
> WorkspaceAutoGenTime)))
> --
> 2.20.1.windows.1