[Patch 0/2] Enable --genfds-multi-thread to default build

[Patch 0/2] Enable --genfds-multi-thread to default build

[Bob Feng]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

Enable --genfds-multi-thread as default build behavior can improve 
build performance but will not bring nagtive impact.

Feng, Bob C (2):
  BaseTools: Fixed the build fail on Linux with --genfds-multi-thread
  BaseTools: Enable --genfds-multi-thread to default build

 BaseTools/Source/Python/AutoGen/GenMake.py             | 4 ++--
 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py    | 3 +--
 BaseTools/Source/Python/Common/GlobalData.py           | 2 +-
 BaseTools/Source/Python/Common/buildoptions.py         | 3 ++-
 BaseTools/Source/Python/GenFds/GenFds.py               | 9 ++++++---
 BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 4 ++--
 BaseTools/Source/Python/build/build.py                 | 2 +-
 7 files changed, 15 insertions(+), 12 deletions(-)

-- 
2.20.1.windows.1

[Patch 1/2] BaseTools: Fixed the build fail on Linux with --genfds-multi-thread

[Bob Feng]

If GenSec input file not exist, the related command will fail and make
will stop on Linux. GenSec input file is allow to be non-existent.

This patch is to let "make" continue if gensec input file not exist.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
---
 BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
index 037828ea1cca..0f691ae52420 100644
--- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
+++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
@@ -487,11 +487,11 @@ class GenFdsGlobalVariable:
             SaveFileOnChange(CommandFile, ' '.join(Cmd), False)
             if IsMakefile:
                 if GlobalData.gGlobalDefines.get("FAMILY") == "MSFT":
                     Cmd = ['if', 'exist', Input[0]] + Cmd
                 else:
-                    Cmd = ['test', '-e', Input[0], "&&"] + Cmd
+                    Cmd = ['-test', '-e', Input[0], "&&"] + Cmd
                 if ' '.join(Cmd).strip() not in GenFdsGlobalVariable.SecCmdList:
                     GenFdsGlobalVariable.SecCmdList.append(' '.join(Cmd).strip())
             elif GenFdsGlobalVariable.NeedsUpdate(Output, list(Input) + [CommandFile]):
                 GenFdsGlobalVariable.DebugLogger(EdkLogger.DEBUG_5, "%s needs update because of newer %s" % (Output, Input))
                 GenFdsGlobalVariable.CallExternalTool(Cmd, "Failed to generate section")
-- 
2.20.1.windows.1

[Patch 2/2] BaseTools: Enable --genfds-multi-thread to default build

[Bob Feng]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302

This patch enable --genfds-multi-thread to default build.
This patch keep --genfds-multi-thread build option for
compatibility and also add a new build option to disable
genfds-multi-thread as --no-genfds-multi-thread.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
---
 BaseTools/Source/Python/AutoGen/GenMake.py             | 4 ++--
 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py    | 3 +--
 BaseTools/Source/Python/Common/GlobalData.py           | 2 +-
 BaseTools/Source/Python/Common/buildoptions.py         | 3 ++-
 BaseTools/Source/Python/GenFds/GenFds.py               | 9 ++++++---
 BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 2 +-
 BaseTools/Source/Python/build/build.py                 | 2 +-
 7 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 47dae82e1aeb..4f85a93055ab 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -1578,12 +1578,12 @@ class TopLevelMakefile(BuildFile):
         elif LogLevel == EdkLogger.QUIET:
             ExtraOption += " -q"
 
         if GlobalData.gCaseInsensitive:
             ExtraOption += " -c"
-        if GlobalData.gEnableGenfdsMultiThread:
-            ExtraOption += " --genfds-multi-thread"
+        if not GlobalData.gEnableGenfdsMultiThread:
+            ExtraOption += " --no-genfds-multi-thread"
         if GlobalData.gIgnoreSource:
             ExtraOption += " --ignore-sources"
 
         for pcd in GlobalData.BuildOptionPcd:
             if pcd[2]:
diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
index 24942674721f..365cfdefeaa1 100644
--- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
@@ -832,12 +832,11 @@ class WorkspaceAutoGen(AutoGen):
         elif LogLevel <= EdkLogger.DEBUG_9:
             FdsCommandDict["debug"] = LogLevel - 1
         elif LogLevel == EdkLogger.QUIET:
             FdsCommandDict["quiet"] = True
 
-        if GlobalData.gEnableGenfdsMultiThread:
-            FdsCommandDict["GenfdsMultiThread"] = True
+        FdsCommandDict["GenfdsMultiThread"] = GlobalData.gEnableGenfdsMultiThread
         if GlobalData.gIgnoreSource:
             FdsCommandDict["IgnoreSources"] = True
 
         FdsCommandDict["OptionPcd"] = []
         for pcd in GlobalData.BuildOptionPcd:
diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Source/Python/Common/GlobalData.py
index 61327ad8f10e..8eb72aa1d6f5 100755
--- a/BaseTools/Source/Python/Common/GlobalData.py
+++ b/BaseTools/Source/Python/Common/GlobalData.py
@@ -104,11 +104,11 @@ gUseHashCache = None
 gBinCacheDest = None
 gBinCacheSource = None
 gPlatformHash = None
 gPackageHash = {}
 gModuleHash = {}
-gEnableGenfdsMultiThread = False
+gEnableGenfdsMultiThread = True
 gSikpAutoGenCache = set()
 
 # Dictionary for tracking Module build status as success or failure
 # Top Dict:     Key: Arch Type              Value: Dictionary
 # Second Dict:  Key: AutoGen Obj    Value: 'SUCCESS'\'FAIL'\'FAIL_METAFILE'
diff --git a/BaseTools/Source/Python/Common/buildoptions.py b/BaseTools/Source/Python/Common/buildoptions.py
index 7161aa66f23e..a717c58d8c9a 100644
--- a/BaseTools/Source/Python/Common/buildoptions.py
+++ b/BaseTools/Source/Python/Common/buildoptions.py
@@ -82,11 +82,12 @@ def MyOptionParser():
     Parser.add_option("--pcd", action="append", dest="OptionPcd", help="Set PCD value by command line. Format: \"PcdName=Value\" ")
     Parser.add_option("-l", "--cmd-len", action="store", type="int", dest="CommandLength", help="Specify the maximum line length of build command. Default is 4096.")
     Parser.add_option("--hash", action="store_true", dest="UseHashCache", default=False, help="Enable hash-based caching during build process.")
     Parser.add_option("--binary-destination", action="store", type="string", dest="BinCacheDest", help="Generate a cache of binary files in the specified directory.")
     Parser.add_option("--binary-source", action="store", type="string", dest="BinCacheSource", help="Consume a cache of binary files from the specified directory.")
-    Parser.add_option("--genfds-multi-thread", action="store_true", dest="GenfdsMultiThread", default=False, help="Enable GenFds multi thread to generate ffs file.")
+    Parser.add_option("--genfds-multi-thread", action="store_true", dest="GenfdsMultiThread", default=True, help="Enable GenFds multi thread to generate ffs file.")
+    Parser.add_option("--no-genfds-multi-thread", action="store_true", dest="NoGenfdsMultiThread", default=False, help="Disable GenFds multi thread to generate ffs file.")
     Parser.add_option("--disable-include-path-check", action="store_true", dest="DisableIncludePathCheck", default=False, help="Disable the include path check for outside of package.")
     (Opt, Args) = Parser.parse_args()
     return (Opt, Args)
 
 BuildOption, BuildTarget = MyOptionParser()
diff --git a/BaseTools/Source/Python/GenFds/GenFds.py b/BaseTools/Source/Python/GenFds/GenFds.py
index 51943411ad1f..c93c7544220f 100644
--- a/BaseTools/Source/Python/GenFds/GenFds.py
+++ b/BaseTools/Source/Python/GenFds/GenFds.py
@@ -91,11 +91,11 @@ def resetFdsGlobalVariable():
     GenFdsGlobalVariable.GuidToolDefinition = {}
     GenFdsGlobalVariable.FfsCmdDict = {}
     GenFdsGlobalVariable.SecCmdList = []
     GenFdsGlobalVariable.CopyList   = []
     GenFdsGlobalVariable.ModuleFile = ''
-    GenFdsGlobalVariable.EnableGenfdsMultiThread = False
+    GenFdsGlobalVariable.EnableGenfdsMultiThread = True
 
     GenFdsGlobalVariable.LargeFileInFvFlags = []
     GenFdsGlobalVariable.EFI_FIRMWARE_FILE_SYSTEM3_GUID = '5473C07A-3DCB-4dca-BD6F-1E9689E7349A'
     GenFdsGlobalVariable.LARGE_FILE_SIZE = 0x1000000
 
@@ -138,10 +138,12 @@ def GenFdsApi(FdsCommandDict, WorkSpaceDataBase=None):
             GenFdsGlobalVariable.WorkSpaceDir = Workspace
             if FdsCommandDict.get("debug"):
                 GenFdsGlobalVariable.VerboseLogger("Using Workspace:" + Workspace)
             if FdsCommandDict.get("GenfdsMultiThread"):
                 GenFdsGlobalVariable.EnableGenfdsMultiThread = True
+            else:
+                GenFdsGlobalVariable.EnableGenfdsMultiThread = False
         os.chdir(GenFdsGlobalVariable.WorkSpaceDir)
 
         # set multiple workspace
         PackagesPath = os.getenv("PACKAGES_PATH")
         mws.setWs(GenFdsGlobalVariable.WorkSpaceDir, PackagesPath)
@@ -400,11 +402,11 @@ def OptionsToCommandDict(Options):
     FdsCommandDict["verbose"] = Options.verbose
     FdsCommandDict["FixedAddress"] = Options.FixedAddress
     FdsCommandDict["quiet"] = Options.quiet
     FdsCommandDict["debug"] = Options.debug
     FdsCommandDict["Workspace"] = Options.Workspace
-    FdsCommandDict["GenfdsMultiThread"] = Options.GenfdsMultiThread
+    FdsCommandDict["GenfdsMultiThread"] = not Options.NoGenfdsMultiThread
     FdsCommandDict["fdf_file"] = [PathClass(Options.filename)] if Options.filename else []
     FdsCommandDict["build_target"] = Options.BuildTarget
     FdsCommandDict["toolchain_tag"] = Options.ToolChain
     FdsCommandDict["active_platform"] = Options.activePlatform
     FdsCommandDict["OptionPcd"] = Options.OptionPcd
@@ -457,11 +459,12 @@ def myOptionParser():
     Parser.add_option("-D", "--define", action="append", type="string", dest="Macros", help="Macro: \"Name [= Value]\".")
     Parser.add_option("-s", "--specifyaddress", dest="FixedAddress", action="store_true", type=None, help="Specify driver load address.")
     Parser.add_option("--conf", action="store", type="string", dest="ConfDirectory", help="Specify the customized Conf directory.")
     Parser.add_option("--ignore-sources", action="store_true", dest="IgnoreSources", default=False, help="Focus to a binary build and ignore all source files")
     Parser.add_option("--pcd", action="append", dest="OptionPcd", help="Set PCD value by command line. Format: \"PcdName=Value\" ")
-    Parser.add_option("--genfds-multi-thread", action="store_true", dest="GenfdsMultiThread", default=False, help="Enable GenFds multi thread to generate ffs file.")
+    Parser.add_option("--genfds-multi-thread", action="store_true", dest="GenfdsMultiThread", default=True, help="Enable GenFds multi thread to generate ffs file.")
+    Parser.add_option("--no-genfds-multi-thread", action="store_true", dest="NoGenfdsMultiThread", default=False, help="Disable GenFds multi thread to generate ffs file.")
 
     Options, _ = Parser.parse_args()
     return Options
 
 ## The class implementing the EDK2 flash image generation process
diff --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
index 0f691ae52420..3dc73c8f61f9 100644
--- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
+++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py
@@ -67,11 +67,11 @@ class GenFdsGlobalVariable:
     GuidToolDefinition = {}
     FfsCmdDict = {}
     SecCmdList = []
     CopyList   = []
     ModuleFile = ''
-    EnableGenfdsMultiThread = False
+    EnableGenfdsMultiThread = True
 
     #
     # The list whose element are flags to indicate if large FFS or SECTION files exist in FV.
     # At the beginning of each generation of FV, false flag is appended to the list,
     # after the call to GenerateSection returns, check the size of the output file,
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index 0406ac314b65..2a10f99a1e70 100755
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -726,11 +726,11 @@ class Build():
         #Set global flag for build mode
         GlobalData.gIgnoreSource = BuildOptions.IgnoreSources
         GlobalData.gUseHashCache = BuildOptions.UseHashCache
         GlobalData.gBinCacheDest   = BuildOptions.BinCacheDest
         GlobalData.gBinCacheSource = BuildOptions.BinCacheSource
-        GlobalData.gEnableGenfdsMultiThread = BuildOptions.GenfdsMultiThread
+        GlobalData.gEnableGenfdsMultiThread = not BuildOptions.NoGenfdsMultiThread
         GlobalData.gDisableIncludePathCheck = BuildOptions.DisableIncludePathCheck
 
         if GlobalData.gBinCacheDest and not GlobalData.gUseHashCache:
             EdkLogger.error("build", OPTION_NOT_SUPPORTED, ExtraData="--binary-destination must be used together with --hash.")
 
-- 
2.20.1.windows.1

Re: [edk2-devel] [Patch 0/2] Enable --genfds-multi-thread to default build

[Liming Gao]

Bob:
  Can you list the platform list have been verified with this change?

Thanks
Liming> -----Original Message-----
> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Bob Feng
> Sent: Thursday, September 5, 2019 11:11 AM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] [Patch 0/2] Enable --genfds-multi-thread to default build
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302
> 
> Enable --genfds-multi-thread as default build behavior can improve
> build performance but will not bring nagtive impact.
> 
> Feng, Bob C (2):
>   BaseTools: Fixed the build fail on Linux with --genfds-multi-thread
>   BaseTools: Enable --genfds-multi-thread to default build
> 
>  BaseTools/Source/Python/AutoGen/GenMake.py             | 4 ++--
>  BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py    | 3 +--
>  BaseTools/Source/Python/Common/GlobalData.py           | 2 +-
>  BaseTools/Source/Python/Common/buildoptions.py         | 3 ++-
>  BaseTools/Source/Python/GenFds/GenFds.py               | 9 ++++++---
>  BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 4 ++--
>  BaseTools/Source/Python/build/build.py                 | 2 +-
>  7 files changed, 15 insertions(+), 12 deletions(-)
> 
> --
> 2.20.1.windows.1
> 
> 
> 

Re: [edk2-devel] [Patch 0/2] Enable --genfds-multi-thread to default build

[Bob Feng]

Emulator
Ovmf
MinPlatforms
And Intel Server Platform.

-Bob

-----Original Message-----
From: Gao, Liming 
Sent: Thursday, September 5, 2019 9:09 PM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>
Subject: RE: [edk2-devel] [Patch 0/2] Enable --genfds-multi-thread to default build

Bob:
  Can you list the platform list have been verified with this change?

Thanks
Liming> -----Original Message-----
> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of 
> Bob Feng
> Sent: Thursday, September 5, 2019 11:11 AM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] [Patch 0/2] Enable --genfds-multi-thread to 
> default build
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1302
> 
> Enable --genfds-multi-thread as default build behavior can improve 
> build performance but will not bring nagtive impact.
> 
> Feng, Bob C (2):
>   BaseTools: Fixed the build fail on Linux with --genfds-multi-thread
>   BaseTools: Enable --genfds-multi-thread to default build
> 
>  BaseTools/Source/Python/AutoGen/GenMake.py             | 4 ++--
>  BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py    | 3 +--
>  BaseTools/Source/Python/Common/GlobalData.py           | 2 +-
>  BaseTools/Source/Python/Common/buildoptions.py         | 3 ++-
>  BaseTools/Source/Python/GenFds/GenFds.py               | 9 ++++++---
>  BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 4 ++--
>  BaseTools/Source/Python/build/build.py                 | 2 +-
>  7 files changed, 15 insertions(+), 12 deletions(-)
> 
> --
> 2.20.1.windows.1
> 
> 
>