* [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains @ 2022-01-17 14:42 Steven Shi 2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi 0 siblings, 1 reply; 6+ messages in thread From: Steven Shi @ 2022-01-17 14:42 UTC (permalink / raw) To: devel Cc: bob.c.feng, yuwei.chen, ardb, leif, abner.chang, daniel.schaefer, Steven Shi The GNU make parallel jobs flgs (-j or --jobs) can significantly (~30%) improve the edk2 build time in Linux. This patch enable the GNU make parallel jobs by default. Below the OVMF test result on GCC5 and CLANGPDB toolchains: GCC5 build without patch: jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t GCC5 -DSECURE_BOOT_ENABLE=TRUE real 0m58.784s user 5m18.820s sys 0m46.337s GCC5 build with patch: jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t GCC5 -DSECURE_BOOT_ENABLE=TRUE real 0m41.547s user 5m23.617s sys 0m47.397s GCC5 build time improved by +29% CLANGPDB build without patch: jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGPDB -DSECURE_BOOT_ENABLE=TRUE real 1m4.162s user 4m15.025s sys 0m30.480s CLANGPDB build with patch: jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ time build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGPDB -DSECURE_BOOT_ENABLE=TRUE real 0m31.307s user 4m21.952s sys 0m30.641s CLANGPDB build time improved by +51% Steven Shi (1): BaseTools: Enable GNU make parallel jobs by default BaseTools/Conf/tools_def.template | 4 ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++- 3 files changed, 18 insertions(+), 3 deletions(-) -- 2.22.0.windows.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default 2022-01-17 14:42 [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains Steven Shi @ 2022-01-17 14:42 ` Steven Shi 2022-01-17 14:54 ` Steven Shi 2022-01-27 3:00 ` Bob Feng 0 siblings, 2 replies; 6+ messages in thread From: Steven Shi @ 2022-01-17 14:42 UTC (permalink / raw) To: devel Cc: bob.c.feng, yuwei.chen, ardb, leif, abner.chang, daniel.schaefer, Steven Shi BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807 * Enable GNU make parallel jobs support(--jobs flag) by default in GCC5 and CLANGPDB toolchains. This gmake flag can save full features bios linux build time by +20%. * Explictly distinguish and define the MS nmake flag and GNU gmake flag in the toolchain which might support both, e.g. CLANGPDB can use nmake in Windows and gmake in Linux. General make flags cannot work for such hybrid make usage scenario. Signed-off-by: Steven Shi <steven.shi@intel.com> --- BaseTools/Conf/tools_def.template | 4 ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template index 2e6b382ab623..84af08cca6ae 100755 --- a/BaseTools/Conf/tools_def.template +++ b/BaseTools/Conf/tools_def.template @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z common-page-size=0x20 *_GCC5_*_*_FAMILY = GCC *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make +*_GCC5_*_GMAKE_FLAGS = --jobs +*_GCC5_*_NMAKE_FLAGS = /nologo *_GCC5_*_*_DLL = ENV(GCC5_DLL) *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN) @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl *_CLANGPDB_*_*_FAMILY = GCC *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make +*_CLANGPDB_*_GMAKE_FLAGS = --jobs +*_CLANGPDB_*_NMAKE_FLAGS = /nologo *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL) *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index e55efff059f9..c6636d99fc7d 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -732,10 +732,11 @@ cleanlib: Src, Dst = CopyCmd Src = self.ReplaceMacro(Src) Dst = self.ReplaceMacro(Dst) + Efi = Src[:-3] + 'efi' if Dst not in self.ResultFileList: self.ResultFileList.append(Dst) if '%s :' %(Dst) not in self.BuildTargetList: - self.BuildTargetList.append("%s : %s" %(Dst,Src)) + self.BuildTargetList.append("%s : %s" %(Dst, Efi)) self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) FfsCmdList = Cmd[0] @@ -778,7 +779,7 @@ cleanlib: SecDepsFileList.append(SecCmdList[index + 1]) index = index + 1 if CmdName == 'Trim': - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', os.path.basename(OutputFile).replace('offset', 'efi'))) + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', os.path.basename(OutputFile).replace('offset', 'efi'))) if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'): SecDepsFileList.append(os.path.join('$(MODULE_DIR)', '$(MODULE_FILE)')) self.FfsOutputFileList.append((OutputFile, ' '.join(SecDepsFileList), SecCmdStr)) diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py index 592d4824a4b3..632c960c8d3d 100644 --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen): RetVal[Tool] = OrderedDict() if Attr not in RetVal[Tool]: RetVal[Tool][Attr] = Value - + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys(): + if RetVal['MAKE']['PATH'] == "nmake": + if 'FLAGS' in RetVal['MAKE'].keys(): + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS'] + else: + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS'] + else: + if 'FLAGS' in RetVal['MAKE'].keys(): + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS'] + else: + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS'] ToolsDef = '' if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: if "FLAGS" not in RetVal["MAKE"]: -- 2.22.0.windows.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default 2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi @ 2022-01-17 14:54 ` Steven Shi 2022-01-19 10:52 ` [edk2-devel] " Steven Shi 2022-01-27 3:00 ` Bob Feng 1 sibling, 1 reply; 6+ messages in thread From: Steven Shi @ 2022-01-17 14:54 UTC (permalink / raw) To: ardb@kernel.org, leif@nuviainc.com, Chang, Abner, Schaefer, Daniel Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io Hello Ard, Leif, Abner, Daniel, I changed the GNU make flag to enable the "--jobs" option in GCC5 toolchain by default for all architectures. Please verify whether this change has impact to the ARM and RISCV GCC5 builds. Thanks! Thanks Steven Shi > -----Original Message----- > From: Shi, Steven <steven.shi@intel.com> > Sent: Monday, January 17, 2022 10:43 PM > To: devel@edk2.groups.io > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang, > Abner <abner.chang@hpe.com>; Schaefer, Daniel > <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com> > Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default > > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807 > > * Enable GNU make parallel jobs support(--jobs flag) by > default in GCC5 and CLANGPDB toolchains. This gmake flag > can save full features bios linux build time by +20%. > * Explictly distinguish and define the MS nmake flag and > GNU gmake flag in the toolchain which might support > both, e.g. CLANGPDB can use nmake in Windows and gmake > in Linux. General make flags cannot work for such hybrid > make usage scenario. > > Signed-off-by: Steven Shi <steven.shi@intel.com> > --- > BaseTools/Conf/tools_def.template | 4 ++++ > BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- > BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 > +++++++++++- > 3 files changed, 18 insertions(+), 3 deletions(-) > > diff --git a/BaseTools/Conf/tools_def.template > b/BaseTools/Conf/tools_def.template > index 2e6b382ab623..84af08cca6ae 100755 > --- a/BaseTools/Conf/tools_def.template > +++ b/BaseTools/Conf/tools_def.template > @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z > common-page-size=0x20 > *_GCC5_*_*_FAMILY = GCC > > > > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make > > +*_GCC5_*_GMAKE_FLAGS = --jobs > > +*_GCC5_*_NMAKE_FLAGS = /nologo > > *_GCC5_*_*_DLL = ENV(GCC5_DLL) > > *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = > DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl > *_CLANGPDB_*_*_FAMILY = GCC > > *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB > > *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make > > +*_CLANGPDB_*_GMAKE_FLAGS = --jobs > > +*_CLANGPDB_*_NMAKE_FLAGS = /nologo > > *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL) > > *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py > b/BaseTools/Source/Python/AutoGen/GenMake.py > index e55efff059f9..c6636d99fc7d 100755 > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > @@ -732,10 +732,11 @@ cleanlib: > Src, Dst = CopyCmd > > Src = self.ReplaceMacro(Src) > > Dst = self.ReplaceMacro(Dst) > > + Efi = Src[:-3] + 'efi' > > if Dst not in self.ResultFileList: > > self.ResultFileList.append(Dst) > > if '%s :' %(Dst) not in self.BuildTargetList: > > - self.BuildTargetList.append("%s : %s" %(Dst,Src)) > > + self.BuildTargetList.append("%s : %s" %(Dst, Efi)) > > self.BuildTargetList.append('\t' + > self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) > > > > FfsCmdList = Cmd[0] > > @@ -778,7 +779,7 @@ cleanlib: > SecDepsFileList.append(SecCmdList[index + 1]) > > index = index + 1 > > if CmdName == 'Trim': > > - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', > os.path.basename(OutputFile).replace('offset', 'efi'))) > > + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', > os.path.basename(OutputFile).replace('offset', 'efi'))) > > if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'): > > SecDepsFileList.append(os.path.join('$(MODULE_DIR)', > '$(MODULE_FILE)')) > > self.FfsOutputFileList.append((OutputFile, ' > '.join(SecDepsFileList), SecCmdStr)) > > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > index 592d4824a4b3..632c960c8d3d 100644 > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen): > RetVal[Tool] = OrderedDict() > > if Attr not in RetVal[Tool]: > > RetVal[Tool][Attr] = Value > > - > > + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys(): > > + if RetVal['MAKE']['PATH'] == "nmake": > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS'] > > + else: > > + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS'] > > + else: > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS'] > > + else: > > + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS'] > > ToolsDef = '' > > if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: > > if "FLAGS" not in RetVal["MAKE"]: > > -- > 2.22.0.windows.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default 2022-01-17 14:54 ` Steven Shi @ 2022-01-19 10:52 ` Steven Shi 2022-01-19 15:03 ` Abner Chang 0 siblings, 1 reply; 6+ messages in thread From: Steven Shi @ 2022-01-19 10:52 UTC (permalink / raw) To: ardb@kernel.org, leif@nuviainc.com, Chang, Abner, Schaefer, Daniel Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io Hello Ard, Leif, Abner, Daniel, Are you ok to enable the GNU make "--jobs" option in GCC5 toolchain by default? Thanks Steven Shi > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Steven > Shi > Sent: Monday, January 17, 2022 10:54 PM > To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner > <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > <yuwei.chen@intel.com>; devel@edk2.groups.io > Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel > jobs by default > > Hello Ard, Leif, Abner, Daniel, > I changed the GNU make flag to enable the "--jobs" option in GCC5 toolchain > by default for all architectures. Please verify whether this change has impact > to the ARM and RISCV GCC5 builds. Thanks! > > > Thanks > Steven Shi > > > -----Original Message----- > > From: Shi, Steven <steven.shi@intel.com> > > Sent: Monday, January 17, 2022 10:43 PM > > To: devel@edk2.groups.io > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > > <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang, > > Abner <abner.chang@hpe.com>; Schaefer, Daniel > > <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com> > > Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default > > > > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807 > > > > * Enable GNU make parallel jobs support(--jobs flag) by > > default in GCC5 and CLANGPDB toolchains. This gmake flag > > can save full features bios linux build time by +20%. > > * Explictly distinguish and define the MS nmake flag and > > GNU gmake flag in the toolchain which might support > > both, e.g. CLANGPDB can use nmake in Windows and gmake > > in Linux. General make flags cannot work for such hybrid > > make usage scenario. > > > > Signed-off-by: Steven Shi <steven.shi@intel.com> > > --- > > BaseTools/Conf/tools_def.template | 4 ++++ > > BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- > > BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 > > +++++++++++- > > 3 files changed, 18 insertions(+), 3 deletions(-) > > > > diff --git a/BaseTools/Conf/tools_def.template > > b/BaseTools/Conf/tools_def.template > > index 2e6b382ab623..84af08cca6ae 100755 > > --- a/BaseTools/Conf/tools_def.template > > +++ b/BaseTools/Conf/tools_def.template > > @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = - > z > > common-page-size=0x20 > > *_GCC5_*_*_FAMILY = GCC > > > > > > > > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make > > > > +*_GCC5_*_GMAKE_FLAGS = --jobs > > > > +*_GCC5_*_NMAKE_FLAGS = /nologo > > > > *_GCC5_*_*_DLL = ENV(GCC5_DLL) > > > > *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > > > > > @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = > > DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl > > *_CLANGPDB_*_*_FAMILY = GCC > > > > *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB > > > > *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make > > > > +*_CLANGPDB_*_GMAKE_FLAGS = --jobs > > > > +*_CLANGPDB_*_NMAKE_FLAGS = /nologo > > > > *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL) > > > > *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > > > > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py > > b/BaseTools/Source/Python/AutoGen/GenMake.py > > index e55efff059f9..c6636d99fc7d 100755 > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > > @@ -732,10 +732,11 @@ cleanlib: > > Src, Dst = CopyCmd > > > > Src = self.ReplaceMacro(Src) > > > > Dst = self.ReplaceMacro(Dst) > > > > + Efi = Src[:-3] + 'efi' > > > > if Dst not in self.ResultFileList: > > > > self.ResultFileList.append(Dst) > > > > if '%s :' %(Dst) not in self.BuildTargetList: > > > > - self.BuildTargetList.append("%s : %s" %(Dst,Src)) > > > > + self.BuildTargetList.append("%s : %s" %(Dst, Efi)) > > > > self.BuildTargetList.append('\t' + > > self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) > > > > > > > > FfsCmdList = Cmd[0] > > > > @@ -778,7 +779,7 @@ cleanlib: > > SecDepsFileList.append(SecCmdList[index + 1]) > > > > index = index + 1 > > > > if CmdName == 'Trim': > > > > - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', > > os.path.basename(OutputFile).replace('offset', 'efi'))) > > > > + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', > > os.path.basename(OutputFile).replace('offset', 'efi'))) > > > > if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'): > > > > SecDepsFileList.append(os.path.join('$(MODULE_DIR)', > > '$(MODULE_FILE)')) > > > > self.FfsOutputFileList.append((OutputFile, ' > > '.join(SecDepsFileList), SecCmdStr)) > > > > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > index 592d4824a4b3..632c960c8d3d 100644 > > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen): > > RetVal[Tool] = OrderedDict() > > > > if Attr not in RetVal[Tool]: > > > > RetVal[Tool][Attr] = Value > > > > - > > > > + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys(): > > > > + if RetVal['MAKE']['PATH'] == "nmake": > > > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS'] > > > > + else: > > > > + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS'] > > > > + else: > > > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS'] > > > > + else: > > > > + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS'] > > > > ToolsDef = '' > > > > if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: > > > > if "FLAGS" not in RetVal["MAKE"]: > > > > -- > > 2.22.0.windows.1 > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default 2022-01-19 10:52 ` [edk2-devel] " Steven Shi @ 2022-01-19 15:03 ` Abner Chang 0 siblings, 0 replies; 6+ messages in thread From: Abner Chang @ 2022-01-19 15:03 UTC (permalink / raw) To: Shi, Steven, ardb@kernel.org, leif@nuviainc.com, Schaefer, Daniel (ROM Janitor) Cc: Feng, Bob C, Chen, Christine, devel@edk2.groups.io Hi Steven, I don't have opinions with enabling GNU parallel job, that is fine if it does break the build. However I am not the maintainer of BaseTools, thus I can't give the review tag to this change. Regards, Abner > -----Original Message----- > From: Shi, Steven <steven.shi@intel.com> > Sent: Wednesday, January 19, 2022 6:52 PM > To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner (HPS SW/FW > Technologist) <abner.chang@hpe.com>; Schaefer, Daniel (ROM Janitor) > <daniel.schaefer@hpe.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > <yuwei.chen@intel.com>; devel@edk2.groups.io > Subject: RE: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make parallel > jobs by default > > Hello Ard, Leif, Abner, Daniel, > Are you ok to enable the GNU make "--jobs" option in GCC5 toolchain by > default? > > Thanks > Steven Shi > > > -----Original Message----- > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of > Steven > > Shi > > Sent: Monday, January 17, 2022 10:54 PM > > To: ardb@kernel.org; leif@nuviainc.com; Chang, Abner > > <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > > <yuwei.chen@intel.com>; devel@edk2.groups.io > > Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools: Enable GNU make > parallel > > jobs by default > > > > Hello Ard, Leif, Abner, Daniel, > > I changed the GNU make flag to enable the "--jobs" option in GCC5 > toolchain > > by default for all architectures. Please verify whether this change has > impact > > to the ARM and RISCV GCC5 builds. Thanks! > > > > > > Thanks > > Steven Shi > > > > > -----Original Message----- > > > From: Shi, Steven <steven.shi@intel.com> > > > Sent: Monday, January 17, 2022 10:43 PM > > > To: devel@edk2.groups.io > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine > > > <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang, > > > Abner <abner.chang@hpe.com>; Schaefer, Daniel > > > <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com> > > > Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by > default > > > > > > BZ: > INVALID URI REMOVED > d=3807__;!!NpxR!2rhlQ- > hgxbtBujz5kn_RG5uJtBl209CFwFmXOlkS0vqky5LS5Yj8zwGnfwSahMs$ > > > > > > * Enable GNU make parallel jobs support(--jobs flag) by > > > default in GCC5 and CLANGPDB toolchains. This gmake flag > > > can save full features bios linux build time by +20%. > > > * Explictly distinguish and define the MS nmake flag and > > > GNU gmake flag in the toolchain which might support > > > both, e.g. CLANGPDB can use nmake in Windows and gmake > > > in Linux. General make flags cannot work for such hybrid > > > make usage scenario. > > > > > > Signed-off-by: Steven Shi <steven.shi@intel.com> > > > --- > > > BaseTools/Conf/tools_def.template | 4 ++++ > > > BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- > > > BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 > > > +++++++++++- > > > 3 files changed, 18 insertions(+), 3 deletions(-) > > > > > > diff --git a/BaseTools/Conf/tools_def.template > > > b/BaseTools/Conf/tools_def.template > > > index 2e6b382ab623..84af08cca6ae 100755 > > > --- a/BaseTools/Conf/tools_def.template > > > +++ b/BaseTools/Conf/tools_def.template > > > @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = > - > > z > > > common-page-size=0x20 > > > *_GCC5_*_*_FAMILY = GCC > > > > > > > > > > > > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make > > > > > > +*_GCC5_*_GMAKE_FLAGS = --jobs > > > > > > +*_GCC5_*_NMAKE_FLAGS = /nologo > > > > > > *_GCC5_*_*_DLL = ENV(GCC5_DLL) > > > > > > *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > > > > > > > > > @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = > > > DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl > > > *_CLANGPDB_*_*_FAMILY = GCC > > > > > > *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB > > > > > > *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make > > > > > > +*_CLANGPDB_*_GMAKE_FLAGS = --jobs > > > > > > +*_CLANGPDB_*_NMAKE_FLAGS = /nologo > > > > > > *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL) > > > > > > *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN) > > > > > > > > > > > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py > > > b/BaseTools/Source/Python/AutoGen/GenMake.py > > > index e55efff059f9..c6636d99fc7d 100755 > > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > > > @@ -732,10 +732,11 @@ cleanlib: > > > Src, Dst = CopyCmd > > > > > > Src = self.ReplaceMacro(Src) > > > > > > Dst = self.ReplaceMacro(Dst) > > > > > > + Efi = Src[:-3] + 'efi' > > > > > > if Dst not in self.ResultFileList: > > > > > > self.ResultFileList.append(Dst) > > > > > > if '%s :' %(Dst) not in self.BuildTargetList: > > > > > > - self.BuildTargetList.append("%s : %s" %(Dst,Src)) > > > > > > + self.BuildTargetList.append("%s : %s" %(Dst, Efi)) > > > > > > self.BuildTargetList.append('\t' + > > > self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) > > > > > > > > > > > > FfsCmdList = Cmd[0] > > > > > > @@ -778,7 +779,7 @@ cleanlib: > > > SecDepsFileList.append(SecCmdList[index + 1]) > > > > > > index = index + 1 > > > > > > if CmdName == 'Trim': > > > > > > - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', > > > os.path.basename(OutputFile).replace('offset', 'efi'))) > > > > > > + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', > > > os.path.basename(OutputFile).replace('offset', 'efi'))) > > > > > > if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'): > > > > > > SecDepsFileList.append(os.path.join('$(MODULE_DIR)', > > > '$(MODULE_FILE)')) > > > > > > self.FfsOutputFileList.append((OutputFile, ' > > > '.join(SecDepsFileList), SecCmdStr)) > > > > > > diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > > b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > > index 592d4824a4b3..632c960c8d3d 100644 > > > --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > > +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py > > > @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen): > > > RetVal[Tool] = OrderedDict() > > > > > > if Attr not in RetVal[Tool]: > > > > > > RetVal[Tool][Attr] = Value > > > > > > - > > > > > > + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys(): > > > > > > + if RetVal['MAKE']['PATH'] == "nmake": > > > > > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > > > > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS'] > > > > > > + else: > > > > > > + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS'] > > > > > > + else: > > > > > > + if 'FLAGS' in RetVal['MAKE'].keys(): > > > > > > + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS'] > > > > > > + else: > > > > > > + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS'] > > > > > > ToolsDef = '' > > > > > > if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: > > > > > > if "FLAGS" not in RetVal["MAKE"]: > > > > > > -- > > > 2.22.0.windows.1 > > > > > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default 2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi 2022-01-17 14:54 ` Steven Shi @ 2022-01-27 3:00 ` Bob Feng 1 sibling, 0 replies; 6+ messages in thread From: Bob Feng @ 2022-01-27 3:00 UTC (permalink / raw) To: Shi, Steven, devel@edk2.groups.io Cc: Chen, Christine, ardb@kernel.org, leif@nuviainc.com, Chang, Abner, Schaefer, Daniel Hi Steven, I'd suggest appending the threads number to the --jobs FLAG. The threads number can be got via build.py ThreadNum(), and suggest testing the incremental build functionality. Thanks, Bob -----Original Message----- From: Shi, Steven <steven.shi@intel.com> Sent: Monday, January 17, 2022 10:43 PM To: devel@edk2.groups.io Cc: Feng, Bob C <bob.c.feng@intel.com>; Chen, Christine <yuwei.chen@intel.com>; ardb@kernel.org; leif@nuviainc.com; Chang, Abner <abner.chang@hpe.com>; Schaefer, Daniel <daniel.schaefer@hpe.com>; Shi, Steven <steven.shi@intel.com> Subject: [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3807 * Enable GNU make parallel jobs support(--jobs flag) by default in GCC5 and CLANGPDB toolchains. This gmake flag can save full features bios linux build time by +20%. * Explictly distinguish and define the MS nmake flag and GNU gmake flag in the toolchain which might support both, e.g. CLANGPDB can use nmake in Windows and gmake in Linux. General make flags cannot work for such hybrid make usage scenario. Signed-off-by: Steven Shi <steven.shi@intel.com> --- BaseTools/Conf/tools_def.template | 4 ++++ BaseTools/Source/Python/AutoGen/GenMake.py | 5 +++-- BaseTools/Source/Python/AutoGen/PlatformAutoGen.py | 12 +++++++++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template index 2e6b382ab623..84af08cca6ae 100755 --- a/BaseTools/Conf/tools_def.template +++ b/BaseTools/Conf/tools_def.template @@ -2283,6 +2283,8 @@ RELEASE_GCC49_AARCH64_DLINK_XIPFLAGS = -z common-page-size=0x20 *_GCC5_*_*_FAMILY = GCC *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make +*_GCC5_*_GMAKE_FLAGS = --jobs +*_GCC5_*_NMAKE_FLAGS = /nologo *_GCC5_*_*_DLL = ENV(GCC5_DLL) *_GCC5_*_ASL_PATH = DEF(UNIX_IASL_BIN) @@ -2742,6 +2744,8 @@ RELEASE_CLANG38_AARCH64_DLINK_FLAGS = DEF(CLANG38_AARCH64_DLINK_FLAGS) -flto -Wl *_CLANGPDB_*_*_FAMILY = GCC *_CLANGPDB_*_*_BUILDRULEFAMILY = CLANGPDB *_CLANGPDB_*_MAKE_PATH = ENV(CLANG_HOST_BIN)make +*_CLANGPDB_*_GMAKE_FLAGS = --jobs +*_CLANGPDB_*_NMAKE_FLAGS = /nologo *_CLANGPDB_*_*_DLL = ENV(CLANGPDB_DLL) *_CLANGPDB_*_ASL_PATH = DEF(UNIX_IASL_BIN) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index e55efff059f9..c6636d99fc7d 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -732,10 +732,11 @@ cleanlib: Src, Dst = CopyCmd Src = self.ReplaceMacro(Src) Dst = self.ReplaceMacro(Dst) + Efi = Src[:-3] + 'efi' if Dst not in self.ResultFileList: self.ResultFileList.append(Dst) if '%s :' %(Dst) not in self.BuildTargetList: - self.BuildTargetList.append("%s : %s" %(Dst,Src)) + self.BuildTargetList.append("%s : %s" %(Dst, Efi)) self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) FfsCmdList = Cmd[0] @@ -778,7 +779,7 @@ cleanlib: SecDepsFileList.append(SecCmdList[index + 1]) index = index + 1 if CmdName == 'Trim': - SecDepsFileList.append(os.path.join('$(DEBUG_DIR)', os.path.basename(OutputFile).replace('offset', 'efi'))) + SecDepsFileList.append(os.path.join('$(OUTPUT_DIR)', os.path.basename(OutputFile).replace('offset', 'efi'))) if OutputFile.endswith('.ui') or OutputFile.endswith('.ver'): SecDepsFileList.append(os.path.join('$(MODULE_DIR)', '$(MODULE_FILE)')) self.FfsOutputFileList.append((OutputFile, ' '.join(SecDepsFileList), SecCmdStr)) diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py index 592d4824a4b3..632c960c8d3d 100644 --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py @@ -893,7 +893,17 @@ class PlatformAutoGen(AutoGen): RetVal[Tool] = OrderedDict() if Attr not in RetVal[Tool]: RetVal[Tool][Attr] = Value - + if 'NMAKE' in RetVal.keys() or 'GMAKE' in RetVal.keys(): + if RetVal['MAKE']['PATH'] == "nmake": + if 'FLAGS' in RetVal['MAKE'].keys(): + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['NMAKE']['FLAGS'] + else: + RetVal['MAKE']['FLAGS'] = RetVal['NMAKE']['FLAGS'] + else: + if 'FLAGS' in RetVal['MAKE'].keys(): + RetVal['MAKE']['FLAGS'] += ' ' + RetVal['GMAKE']['FLAGS'] + else: + RetVal['MAKE']['FLAGS'] = RetVal['GMAKE']['FLAGS'] ToolsDef = '' if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: if "FLAGS" not in RetVal["MAKE"]: -- 2.22.0.windows.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-01-27 3:01 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-01-17 14:42 [PATCH 0/1] Enable GNU make parallel jobs by default for GCC5 and CLANGPDB ToolChains Steven Shi 2022-01-17 14:42 ` [PATCH 1/1] BaseTools: Enable GNU make parallel jobs by default Steven Shi 2022-01-17 14:54 ` Steven Shi 2022-01-19 10:52 ` [edk2-devel] " Steven Shi 2022-01-19 15:03 ` Abner Chang 2022-01-27 3:00 ` Bob Feng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox