* [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains
@ 2022-01-17 14:42 Steven Shi
2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi
0 siblings, 1 reply; 6+ messages in thread
From: Steven Shi @ 2022-01-17 14:42 UTC (permalink / raw)
To: devel
Cc: bob.c.feng, yuwei.chen, ardb, leif, abner.chang, daniel.schaefer,
Steven Shi
The GNU make parallel jobs flgs (-j or --jobs) can significantly (~30%) improve the edk2 build time in Linux.
This patch enable the GNU make parallel jobs by default.
Below the OVMF test result on GCC5 and CLANGPDB toolchains:
GCC5 build without patch:
jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t GCC5 -DSECURE_BOOT_ENABLE=TRUE
real 0m58.784s
user 5m18.820s
sys 0m46.337s
GCC5 build with patch:
jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t GCC5 -DSECURE_BOOT_ENABLE=TRUE
real 0m41.547s
user 5m23.617s
sys 0m47.397s
GCC5 build time improved by +29%
CLANGPDB build without patch:
jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGPDB -DSECURE_BOOT_ENABLE=TRUE
real 1m4.162s
user 4m15.025s
sys 0m30.480s
CLANGPDB build with patch:
jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGPDB -DSECURE_BOOT_ENABLE=TRUE
real 0m31.307s
user 4m21.952s
sys 0m30.641s
CLANGPDB build time improved by +51%
Steven Shi (1):
BaseTools: Enable GNU make parallel jobs by default
BaseTools/Conf/tools_def.template | 4 ++++
BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
--
2.22.0.windows.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
2022-01-17 14:42 [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains Steven Shi
@ 2022-01-17 14:42 ` Steven Shi
2022-01-17 14:54 ` Steven Shi
2022-01-27 3:00 ` Bob Feng
0 siblings, 2 replies; 6+ messages in thread
From: Steven Shi @ 2022-01-17 14:42 UTC (permalink / raw)
To: devel
Cc: bob.c.feng, yuwei.chen, ardb, leif, abner.chang, daniel.schaefer,
Steven Shi
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807
* Enable GNU make parallel jobs support(--jobs flag) by
default in GCC5 and CLANGPDB toolchains. This gmake flag
can save full features bios linux build time by +20%.
* Explictly distinguish and define the MS nmake flag and
GNU gmake flag in the toolchain which might support
both, e.g. CLANGPDB can use nmake in Windows and gmake
in Linux. General make flags cannot work for such hybrid
make usage scenario.
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Conf/tools_def.template | 4 ++++
BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template
index 2e6b382ab623..84af08cca6ae 100755
--- a/BaseTools/Conf/tools_def.template
+++ b/BaseTools/Conf/tools_def.template
@@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z common-page-size=0x20
*_GCC5_*_*_FAMILY = GCC
*_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
+*_GCC5_*_GMAKE_FLAGS = --jobs
+*_GCC5_*_NMAKE_FLAGS = /nologo
*_GCC5_*_*_DLL = ENV(GCC5_DLL)
*_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN)
@@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl
*_CLANGPDB_*_*_FAMILY = GCC
*_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB
*_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make
+*_CLANGPDB_*_GMAKE_FLAGS = --jobs
+*_CLANGPDB_*_NMAKE_FLAGS = /nologo
*_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL)
*_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index e55efff059f9..c6636d99fc7d 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -732,10 +732,11 @@ cleanlib:
Src, Dst = CopyCmd
Src = self.ReplaceMacro(Src)
Dst = self.ReplaceMacro(Dst)
+ Efi = Src[:-3] + 'efi'
if Dst not in self.ResultFileList:
self.ResultFileList.append(Dst)
if '%s :' %(Dst) not in self.BuildTargetList:
- self.BuildTargetList.append("%s : %s" %(Dst,Src))
+ self.BuildTargetList.append("%s : %s" %(Dst, Efi))
self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst})
FfsCmdList = Cmd[0]
@@ -778,7 +779,7 @@ cleanlib:
SecDepsFileList.append(SecCmdList[index + 1])
index = index + 1
if CmdName == 'Trim':
- SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', os.path.basename(OutputFile).replace('offset', 'efi')))
+ SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', os.path.basename(OutputFile).replace('offset', 'efi')))
if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'):
SecDepsFileList.append(os.path.join('$(MODULE_DIR)', '$(MODULE_FILE)'))
self.FfsOutputFileList.append((OutputFile, ' '.join(SecDepsFileList), SecCmdStr))
diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
index 592d4824a4b3..632c960c8d3d 100644
--- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
@@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen):
RetVal[Tool] = OrderedDict()
if Attr not in RetVal[Tool]:
RetVal[Tool][Attr] = Value
-
+ if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys():
+ if RetVal['MAKE']['PATH'] == "nmake":
+ if 'FLAGS' in RetVal['MAKE'].keys():
+ RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS']
+ else:
+ RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS']
+ else:
+ if 'FLAGS' in RetVal['MAKE'].keys():
+ RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS']
+ else:
+ RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS']
ToolsDef = ''
if GlobalData.gOptions.SilentMode and "MAKE" in RetVal:
if "FLAGS" not in RetVal["MAKE"]:
--
2.22.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi
@ 2022-01-17 14:54 ` Steven Shi
2022-01-19 10:52 ` [edk2-devel] " Steven Shi
2022-01-27 3:00 ` Bob Feng
1 sibling, 1 reply; 6+ messages in thread
From: Steven Shi @ 2022-01-17 14:54 UTC (permalink / raw)
To: ardb@kernel.org, leif@nuviainc.com, Chang, Abner,
Schaefer, Daniel
Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io
Hello Ard, Leif, Abner, Daniel,
I changed the GNU make flag to enable the "--jobs" option in GCC5 toolchain by default for all architectures. Please verify whether this change has impact to the ARM and RISCV GCC5 builds. Thanks!
Thanks
Steven Shi
> -----Original Message-----
> From: Shi, Steven <steven.shi@intel.com>
> Sent: Monday, January 17, 2022 10:43 PM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang,
> Abner <abner.chang@hpe.com>; Schaefer, Daniel
> <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com>
> Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
>
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807
>
> * Enable GNU make parallel jobs support(--jobs flag) by
> default in GCC5 and CLANGPDB toolchains. This gmake flag
> can save full features bios linux build time by +20%.
> * Explictly distinguish and define the MS nmake flag and
> GNU gmake flag in the toolchain which might support
> both, e.g. CLANGPDB can use nmake in Windows and gmake
> in Linux. General make flags cannot work for such hybrid
> make usage scenario.
>
> Signed-off-by: Steven Shi <steven.shi@intel.com>
> ---
> BaseTools/Conf/tools_def.template | 4 ++++
> BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
> BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12
> +++++++++++-
> 3 files changed, 18 insertions(+), 3 deletions(-)
>
> diff --git a/BaseTools/Conf/tools_def.template
> b/BaseTools/Conf/tools_def.template
> index 2e6b382ab623..84af08cca6ae 100755
> --- a/BaseTools/Conf/tools_def.template
> +++ b/BaseTools/Conf/tools_def.template
> @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z
> common-page-size=0x20
> *_GCC5_*_*_FAMILY = GCC
>
>
>
> *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
>
> +*_GCC5_*_GMAKE_FLAGS = --jobs
>
> +*_GCC5_*_NMAKE_FLAGS = /nologo
>
> *_GCC5_*_*_DLL = ENV(GCC5_DLL)
>
> *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN)
>
>
>
> @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS =
> DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl
> *_CLANGPDB_*_*_FAMILY = GCC
>
> *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB
>
> *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make
>
> +*_CLANGPDB_*_GMAKE_FLAGS = --jobs
>
> +*_CLANGPDB_*_NMAKE_FLAGS = /nologo
>
> *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL)
>
> *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN)
>
>
>
> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
> b/BaseTools/Source/Python/AutoGen/GenMake.py
> index e55efff059f9..c6636d99fc7d 100755
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -732,10 +732,11 @@ cleanlib:
> Src, Dst = CopyCmd
>
> Src = self.ReplaceMacro(Src)
>
> Dst = self.ReplaceMacro(Dst)
>
> + Efi = Src[:-3] + 'efi'
>
> if Dst not in self.ResultFileList:
>
> self.ResultFileList.append(Dst)
>
> if '%s :' %(Dst) not in self.BuildTargetList:
>
> - self.BuildTargetList.append("%s : %s" %(Dst,Src))
>
> + self.BuildTargetList.append("%s : %s" %(Dst, Efi))
>
> self.BuildTargetList.append('\t' +
> self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst})
>
>
>
> FfsCmdList = Cmd[0]
>
> @@ -778,7 +779,7 @@ cleanlib:
> SecDepsFileList.append(SecCmdList[index + 1])
>
> index = index + 1
>
> if CmdName == 'Trim':
>
> - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)',
> os.path.basename(OutputFile).replace('offset', 'efi')))
>
> + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)',
> os.path.basename(OutputFile).replace('offset', 'efi')))
>
> if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'):
>
> SecDepsFileList.append(os.path.join('$(MODULE_DIR)',
> '$(MODULE_FILE)'))
>
> self.FfsOutputFileList.append((OutputFile, '
> '.join(SecDepsFileList), SecCmdStr))
>
> diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> index 592d4824a4b3..632c960c8d3d 100644
> --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen):
> RetVal[Tool] = OrderedDict()
>
> if Attr not in RetVal[Tool]:
>
> RetVal[Tool][Attr] = Value
>
> -
>
> + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys():
>
> + if RetVal['MAKE']['PATH'] == "nmake":
>
> + if 'FLAGS' in RetVal['MAKE'].keys():
>
> + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS']
>
> + else:
>
> + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS']
>
> + else:
>
> + if 'FLAGS' in RetVal['MAKE'].keys():
>
> + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS']
>
> + else:
>
> + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS']
>
> ToolsDef = ''
>
> if GlobalData.gOptions.SilentMode and "MAKE" in RetVal:
>
> if "FLAGS" not in RetVal["MAKE"]:
>
> --
> 2.22.0.windows.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
2022-01-17 14:54 ` Steven Shi
@ 2022-01-19 10:52 ` Steven Shi
2022-01-19 15:03 ` Abner Chang
0 siblings, 1 reply; 6+ messages in thread
From: Steven Shi @ 2022-01-19 10:52 UTC (permalink / raw)
To: ardb@kernel.org, leif@nuviainc.com, Chang, Abner,
Schaefer, Daniel
Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io
Hello Ard, Leif, Abner, Daniel,
Are you ok to enable the GNU make "--jobs" option in GCC5 toolchain by default?
Thanks
Steven Shi
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Steven
> Shi
> Sent: Monday, January 17, 2022 10:54 PM
> To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner
> <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> <yuwei.chen@intel.com>; devel@edk2.groups.io
> Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel
> jobs by default
>
> Hello Ard, Leif, Abner, Daniel,
> I changed the GNU make flag to enable the "--jobs" option in GCC5 toolchain
> by default for all architectures. Please verify whether this change has impact
> to the ARM and RISCV GCC5 builds. Thanks!
>
>
> Thanks
> Steven Shi
>
> > -----Original Message-----
> > From: Shi, Steven <steven.shi@intel.com>
> > Sent: Monday, January 17, 2022 10:43 PM
> > To: devel@edk2.groups.io
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> > <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang,
> > Abner <abner.chang@hpe.com>; Schaefer, Daniel
> > <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com>
> > Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
> >
> > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807
> >
> > * Enable GNU make parallel jobs support(--jobs flag) by
> > default in GCC5 and CLANGPDB toolchains. This gmake flag
> > can save full features bios linux build time by +20%.
> > * Explictly distinguish and define the MS nmake flag and
> > GNU gmake flag in the toolchain which might support
> > both, e.g. CLANGPDB can use nmake in Windows and gmake
> > in Linux. General make flags cannot work for such hybrid
> > make usage scenario.
> >
> > Signed-off-by: Steven Shi <steven.shi@intel.com>
> > ---
> > BaseTools/Conf/tools_def.template | 4 ++++
> > BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
> > BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12
> > +++++++++++-
> > 3 files changed, 18 insertions(+), 3 deletions(-)
> >
> > diff --git a/BaseTools/Conf/tools_def.template
> > b/BaseTools/Conf/tools_def.template
> > index 2e6b382ab623..84af08cca6ae 100755
> > --- a/BaseTools/Conf/tools_def.template
> > +++ b/BaseTools/Conf/tools_def.template
> > @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -
> z
> > common-page-size=0x20
> > *_GCC5_*_*_FAMILY = GCC
> >
> >
> >
> > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
> >
> > +*_GCC5_*_GMAKE_FLAGS = --jobs
> >
> > +*_GCC5_*_NMAKE_FLAGS = /nologo
> >
> > *_GCC5_*_*_DLL = ENV(GCC5_DLL)
> >
> > *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN)
> >
> >
> >
> > @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS =
> > DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl
> > *_CLANGPDB_*_*_FAMILY = GCC
> >
> > *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB
> >
> > *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make
> >
> > +*_CLANGPDB_*_GMAKE_FLAGS = --jobs
> >
> > +*_CLANGPDB_*_NMAKE_FLAGS = /nologo
> >
> > *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL)
> >
> > *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN)
> >
> >
> >
> > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
> > b/BaseTools/Source/Python/AutoGen/GenMake.py
> > index e55efff059f9..c6636d99fc7d 100755
> > --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> > @@ -732,10 +732,11 @@ cleanlib:
> > Src, Dst = CopyCmd
> >
> > Src = self.ReplaceMacro(Src)
> >
> > Dst = self.ReplaceMacro(Dst)
> >
> > + Efi = Src[:-3] + 'efi'
> >
> > if Dst not in self.ResultFileList:
> >
> > self.ResultFileList.append(Dst)
> >
> > if '%s :' %(Dst) not in self.BuildTargetList:
> >
> > - self.BuildTargetList.append("%s : %s" %(Dst,Src))
> >
> > + self.BuildTargetList.append("%s : %s" %(Dst, Efi))
> >
> > self.BuildTargetList.append('\t' +
> > self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst})
> >
> >
> >
> > FfsCmdList = Cmd[0]
> >
> > @@ -778,7 +779,7 @@ cleanlib:
> > SecDepsFileList.append(SecCmdList[index + 1])
> >
> > index = index + 1
> >
> > if CmdName == 'Trim':
> >
> > - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)',
> > os.path.basename(OutputFile).replace('offset', 'efi')))
> >
> > + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)',
> > os.path.basename(OutputFile).replace('offset', 'efi')))
> >
> > if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'):
> >
> > SecDepsFileList.append(os.path.join('$(MODULE_DIR)',
> > '$(MODULE_FILE)'))
> >
> > self.FfsOutputFileList.append((OutputFile, '
> > '.join(SecDepsFileList), SecCmdStr))
> >
> > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > index 592d4824a4b3..632c960c8d3d 100644
> > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen):
> > RetVal[Tool] = OrderedDict()
> >
> > if Attr not in RetVal[Tool]:
> >
> > RetVal[Tool][Attr] = Value
> >
> > -
> >
> > + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys():
> >
> > + if RetVal['MAKE']['PATH'] == "nmake":
> >
> > + if 'FLAGS' in RetVal['MAKE'].keys():
> >
> > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS']
> >
> > + else:
> >
> > + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS']
> >
> > + else:
> >
> > + if 'FLAGS' in RetVal['MAKE'].keys():
> >
> > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS']
> >
> > + else:
> >
> > + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS']
> >
> > ToolsDef = ''
> >
> > if GlobalData.gOptions.SilentMode and "MAKE" in RetVal:
> >
> > if "FLAGS" not in RetVal["MAKE"]:
> >
> > --
> > 2.22.0.windows.1
>
>
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
2022-01-19 10:52 ` [edk2-devel] " Steven Shi
@ 2022-01-19 15:03 ` Abner Chang
0 siblings, 0 replies; 6+ messages in thread
From: Abner Chang @ 2022-01-19 15:03 UTC (permalink / raw)
To: Shi, Steven, ardb@kernel.org, leif@nuviainc.com,
Schaefer, Daniel (ROM Janitor)
Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io
Hi Steven,
I don't have opinions with enabling GNU parallel job, that is fine if it does break the build.
However I am not the maintainer of BaseTools, thus I can't give the review tag to this change.
Regards,
Abner
> -----Original Message-----
> From: Shi, Steven <steven.shi@intel.com>
> Sent: Wednesday, January 19, 2022 6:52 PM
> To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner (HPS SW/FW
> Technologist) <abner.chang@hpe.com>; Schaefer, Daniel (ROM Janitor)
> <daniel.schaefer@hpe.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> <yuwei.chen@intel.com>; devel@edk2.groups.io
> Subject: RE: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel
> jobs by default
>
> Hello Ard, Leif, Abner, Daniel,
> Are you ok to enable the GNU make "--jobs" option in GCC5 toolchain by
> default?
>
> Thanks
> Steven Shi
>
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> Steven
> > Shi
> > Sent: Monday, January 17, 2022 10:54 PM
> > To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner
> > <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> > <yuwei.chen@intel.com>; devel@edk2.groups.io
> > Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make
> parallel
> > jobs by default
> >
> > Hello Ard, Leif, Abner, Daniel,
> > I changed the GNU make flag to enable the "--jobs" option in GCC5
> toolchain
> > by default for all architectures. Please verify whether this change has
> impact
> > to the ARM and RISCV GCC5 builds. Thanks!
> >
> >
> > Thanks
> > Steven Shi
> >
> > > -----Original Message-----
> > > From: Shi, Steven <steven.shi@intel.com>
> > > Sent: Monday, January 17, 2022 10:43 PM
> > > To: devel@edk2.groups.io
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine
> > > <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang,
> > > Abner <abner.chang@hpe.com>; Schaefer, Daniel
> > > <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com>
> > > Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by
> default
> > >
> > > BZ:
> INVALID URI REMOVED
> d=3807__;!!NpxR!2rhlQ-
> hgxbtBujz5kn_RG5uJtBl209CFwFmXOlkS0vqky5LS5Yj8zwGnfwSahMs$
> > >
> > > * Enable GNU make parallel jobs support(--jobs flag) by
> > > default in GCC5 and CLANGPDB toolchains. This gmake flag
> > > can save full features bios linux build time by +20%.
> > > * Explictly distinguish and define the MS nmake flag and
> > > GNU gmake flag in the toolchain which might support
> > > both, e.g. CLANGPDB can use nmake in Windows and gmake
> > > in Linux. General make flags cannot work for such hybrid
> > > make usage scenario.
> > >
> > > Signed-off-by: Steven Shi <steven.shi@intel.com>
> > > ---
> > > BaseTools/Conf/tools_def.template | 4 ++++
> > > BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
> > > BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12
> > > +++++++++++-
> > > 3 files changed, 18 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/BaseTools/Conf/tools_def.template
> > > b/BaseTools/Conf/tools_def.template
> > > index 2e6b382ab623..84af08cca6ae 100755
> > > --- a/BaseTools/Conf/tools_def.template
> > > +++ b/BaseTools/Conf/tools_def.template
> > > @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS =
> -
> > z
> > > common-page-size=0x20
> > > *_GCC5_*_*_FAMILY = GCC
> > >
> > >
> > >
> > > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
> > >
> > > +*_GCC5_*_GMAKE_FLAGS = --jobs
> > >
> > > +*_GCC5_*_NMAKE_FLAGS = /nologo
> > >
> > > *_GCC5_*_*_DLL = ENV(GCC5_DLL)
> > >
> > > *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN)
> > >
> > >
> > >
> > > @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS =
> > > DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl
> > > *_CLANGPDB_*_*_FAMILY = GCC
> > >
> > > *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB
> > >
> > > *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make
> > >
> > > +*_CLANGPDB_*_GMAKE_FLAGS = --jobs
> > >
> > > +*_CLANGPDB_*_NMAKE_FLAGS = /nologo
> > >
> > > *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL)
> > >
> > > *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN)
> > >
> > >
> > >
> > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
> > > b/BaseTools/Source/Python/AutoGen/GenMake.py
> > > index e55efff059f9..c6636d99fc7d 100755
> > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> > > @@ -732,10 +732,11 @@ cleanlib:
> > > Src, Dst = CopyCmd
> > >
> > > Src = self.ReplaceMacro(Src)
> > >
> > > Dst = self.ReplaceMacro(Dst)
> > >
> > > + Efi = Src[:-3] + 'efi'
> > >
> > > if Dst not in self.ResultFileList:
> > >
> > > self.ResultFileList.append(Dst)
> > >
> > > if '%s :' %(Dst) not in self.BuildTargetList:
> > >
> > > - self.BuildTargetList.append("%s : %s" %(Dst,Src))
> > >
> > > + self.BuildTargetList.append("%s : %s" %(Dst, Efi))
> > >
> > > self.BuildTargetList.append('\t' +
> > > self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst})
> > >
> > >
> > >
> > > FfsCmdList = Cmd[0]
> > >
> > > @@ -778,7 +779,7 @@ cleanlib:
> > > SecDepsFileList.append(SecCmdList[index + 1])
> > >
> > > index = index + 1
> > >
> > > if CmdName == 'Trim':
> > >
> > > - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)',
> > > os.path.basename(OutputFile).replace('offset', 'efi')))
> > >
> > > + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)',
> > > os.path.basename(OutputFile).replace('offset', 'efi')))
> > >
> > > if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'):
> > >
> > > SecDepsFileList.append(os.path.join('$(MODULE_DIR)',
> > > '$(MODULE_FILE)'))
> > >
> > > self.FfsOutputFileList.append((OutputFile, '
> > > '.join(SecDepsFileList), SecCmdStr))
> > >
> > > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > > index 592d4824a4b3..632c960c8d3d 100644
> > > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
> > > @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen):
> > > RetVal[Tool] = OrderedDict()
> > >
> > > if Attr not in RetVal[Tool]:
> > >
> > > RetVal[Tool][Attr] = Value
> > >
> > > -
> > >
> > > + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys():
> > >
> > > + if RetVal['MAKE']['PATH'] == "nmake":
> > >
> > > + if 'FLAGS' in RetVal['MAKE'].keys():
> > >
> > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS']
> > >
> > > + else:
> > >
> > > + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS']
> > >
> > > + else:
> > >
> > > + if 'FLAGS' in RetVal['MAKE'].keys():
> > >
> > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS']
> > >
> > > + else:
> > >
> > > + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS']
> > >
> > > ToolsDef = ''
> > >
> > > if GlobalData.gOptions.SilentMode and "MAKE" in RetVal:
> > >
> > > if "FLAGS" not in RetVal["MAKE"]:
> > >
> > > --
> > > 2.22.0.windows.1
> >
> >
> >
> >
> >
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi
2022-01-17 14:54 ` Steven Shi
@ 2022-01-27 3:00 ` Bob Feng
1 sibling, 0 replies; 6+ messages in thread
From: Bob Feng @ 2022-01-27 3:00 UTC (permalink / raw)
To: Shi, Steven, devel@edk2.groups.io
Cc: Chen, Christine, ardb@kernel.org, leif@nuviainc.com, Chang, Abner,
Schaefer, Daniel
Hi Steven,
I'd suggest appending the threads number to the --jobs FLAG. The threads number can be got via build.py ThreadNum(),
and suggest testing the incremental build functionality.
Thanks,
Bob
-----Original Message-----
From: Shi, Steven <steven.shi@intel.com>
Sent: Monday, January 17, 2022 10:43 PM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang, Abner <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com>
Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807
* Enable GNU make parallel jobs support(--jobs flag) by
default in GCC5 and CLANGPDB toolchains. This gmake flag
can save full features bios linux build time by +20%.
* Explictly distinguish and define the MS nmake flag and
GNU gmake flag in the toolchain which might support
both, e.g. CLANGPDB can use nmake in Windows and gmake
in Linux. General make flags cannot work for such hybrid
make usage scenario.
Signed-off-by: Steven Shi <steven.shi@intel.com>
---
BaseTools/Conf/tools_def.template | 4 ++++
BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++--
BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template
index 2e6b382ab623..84af08cca6ae 100755
--- a/BaseTools/Conf/tools_def.template
+++ b/BaseTools/Conf/tools_def.template
@@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z common-page-size=0x20
*_GCC5_*_*_FAMILY = GCC
*_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make
+*_GCC5_*_GMAKE_FLAGS = --jobs
+*_GCC5_*_NMAKE_FLAGS = /nologo
*_GCC5_*_*_DLL = ENV(GCC5_DLL)
*_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN)
@@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl
*_CLANGPDB_*_*_FAMILY = GCC
*_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB
*_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make
+*_CLANGPDB_*_GMAKE_FLAGS = --jobs
+*_CLANGPDB_*_NMAKE_FLAGS = /nologo
*_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL)
*_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN)
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index e55efff059f9..c6636d99fc7d 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -732,10 +732,11 @@ cleanlib:
Src, Dst = CopyCmd
Src = self.ReplaceMacro(Src)
Dst = self.ReplaceMacro(Dst)
+ Efi = Src[:-3] + 'efi'
if Dst not in self.ResultFileList:
self.ResultFileList.append(Dst)
if '%s :' %(Dst) not in self.BuildTargetList:
- self.BuildTargetList.append("%s : %s" %(Dst,Src))
+ self.BuildTargetList.append("%s : %s" %(Dst, Efi))
self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst})
FfsCmdList = Cmd[0]
@@ -778,7 +779,7 @@ cleanlib:
SecDepsFileList.append(SecCmdList[index + 1])
index = index + 1
if CmdName == 'Trim':
- SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', os.path.basename(OutputFile).replace('offset', 'efi')))
+ SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', os.path.basename(OutputFile).replace('offset', 'efi')))
if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'):
SecDepsFileList.append(os.path.join('$(MODULE_DIR)', '$(MODULE_FILE)'))
self.FfsOutputFileList.append((OutputFile, ' '.join(SecDepsFileList), SecCmdStr))
diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
index 592d4824a4b3..632c960c8d3d 100644
--- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
@@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen):
RetVal[Tool] = OrderedDict()
if Attr not in RetVal[Tool]:
RetVal[Tool][Attr] = Value
-
+ if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys():
+ if RetVal['MAKE']['PATH'] == "nmake":
+ if 'FLAGS' in RetVal['MAKE'].keys():
+ RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS']
+ else:
+ RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS']
+ else:
+ if 'FLAGS' in RetVal['MAKE'].keys():
+ RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS']
+ else:
+ RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS']
ToolsDef = ''
if GlobalData.gOptions.SilentMode and "MAKE" in RetVal:
if "FLAGS" not in RetVal["MAKE"]:
--
2.22.0.windows.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-01-27 3:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-01-17 14:42 [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains Steven Shi
2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi
2022-01-17 14:54 ` Steven Shi
2022-01-19 10:52 ` [edk2-devel] " Steven Shi
2022-01-19 15:03 ` Abner Chang
2022-01-27 3:00 ` Bob Feng
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox