From: "Gao, Liming" <liming.gao@intel.com>
To: "Feng, Bob C" <bob.c.feng@intel.com>,
"edk2-devel@lists.01.org" <edk2-devel@lists.01.org>
Subject: Re: [Patch] BaseTools:Enable the /MP option of MSVC compiler
Date: Wed, 3 Apr 2019 02:45:12 +0000 [thread overview]
Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E41D00E@SHSMSX104.ccr.corp.intel.com> (raw)
In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D1600D19B0@SHSMSX101.ccr.corp.intel.com>
The clean build performance data is good.
This change updates build rule generation for C source files on MSFT tool chain. It is a base tool behavior change. If it doesn't depend on build rule, could you drop the change in build rule.txt. MP flag will be moved to tools_def.txt.
Thanks
Liming
> -----Original Message-----
> From: Feng, Bob C
> Sent: Wednesday, April 3, 2019 10:12 AM
> To: Gao, Liming <liming.gao@intel.com>; edk2-devel@lists.01.org
> Cc: Fan, ZhijuX <zhijux.fan@intel.com>
> Subject: RE: [Patch] BaseTools:Enable the /MP option of MSVC compiler
>
> Liming,
>
> I add the performance data on the BZ:
> https://bugzilla.tianocore.org/show_bug.cgi?id=1672
>
> Thanks,
> Bob
>
> -----Original Message-----
> From: Gao, Liming
> Sent: Friday, March 29, 2019 10:20 PM
> To: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org
> Cc: Fan, ZhijuX <zhijux.fan@intel.com>
> Subject: RE: [Patch] BaseTools:Enable the /MP option of MSVC compiler
>
> Bob:
> How about use /MP option without process number? The compiler will retrieves the number of effective processors on your computer.
> And, move this option into CC_FLAGS in tools_def.txt. If so, platform can override it in platform DSC file.
>
> Last, please provide the performance data with this option.
>
> Thanks
> Liming
> > -----Original Message-----
> > From: Feng, Bob C
> > Sent: Friday, March 29, 2019 8:30 PM
> > To: edk2-devel@lists.01.org
> > Cc: Fan, ZhijuX <zhijux.fan@intel.com>; Feng, Bob C
> > <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>
> > Subject: [Patch] BaseTools:Enable the /MP option of MSVC compiler
> >
> > From: Zhiju Fan <zhijux.fan@intel.com>
> >
> > https://bugzilla.tianocore.org/show_bug.cgi?id=1672
> > The /MP option of MSVC compiler can reduce the total time to compile
> > the source files on the command line.
> >
> > This patch is going to enable this MSVC option in BaseTools.
> >
> > Cc: Bob Feng <bob.c.feng@intel.com>
> > Cc: Liming Gao <liming.gao@intel.com>
> > Contributed-under: TianoCore Contribution Agreement 1.1
> > Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
> > ---
> > BaseTools/Conf/build_rule.template | 2 +-
> > BaseTools/Source/Python/AutoGen/GenMake.py | 81
> > ++++++++++++++++++++--
> > 2 files changed, 75 insertions(+), 8 deletions(-)
> >
> > diff --git a/BaseTools/Conf/build_rule.template
> > b/BaseTools/Conf/build_rule.template
> > index e56b1d9c59..e7d736740f 100755
> > --- a/BaseTools/Conf/build_rule.template
> > +++ b/BaseTools/Conf/build_rule.template
> > @@ -127,11 +127,11 @@
> >
> > <OutputFile>
> > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.obj
> >
> > <Command.MSFT, Command.INTEL>
> > - "$(CC)" /Fo${dst} $(CC_FLAGS) $(INC) ${src}
> > + "$(CC)" /MP7 /Fo${d_path}\ $(CC_FLAGS) $(INC) ${src}
> >
> > <Command.GCC, Command.RVCT>
> > # For RVCTCYGWIN CC_FLAGS must be first to work around pathing issues
> > "$(CC)" $(CC_FLAGS) -c -o ${dst} $(INC) ${src}
> >
> > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
> > b/BaseTools/Source/Python/AutoGen/GenMake.py
> > index b441817b52..04951346ad 100644
> > --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> > @@ -433,11 +433,11 @@ cleanlib:
> > self.BuildTargetList = [] # [target string]
> > self.PendingBuildTargetList = [] # [FileBuildRule objects]
> > self.CommonFileDependency = []
> > self.FileListMacros = {}
> > self.ListFileMacros = {}
> > -
> > + self.ObjTargetDict = {}
> > self.FileCache = {}
> > self.LibraryBuildCommandList = []
> > self.LibraryFileList = []
> > self.LibraryMakefileList = []
> > self.LibraryBuildDirectoryList = [] @@ -616,10 +616,15 @@
> > cleanlib:
> > ListFileName,
> > "\n".join(self.ListFileMacros[ListFileMacro]),
> > False
> > )
> >
> > + # Generate objlist used to create .obj file
> > + for Type in self.ObjTargetDict:
> > + NewLine = ' '.join(list(self.ObjTargetDict[Type]))
> > + FileMacroList.append("OBJLIST_%s = %s" %
> > + (list(self.ObjTargetDict.keys()).index(Type), NewLine))
> > +
> > BcTargetList = []
> >
> > MakefileName = self._FILE_NAME_[self._FileType]
> > LibraryMakeCommandList = []
> > for D in self.LibraryBuildDirectoryList:
> > @@ -925,17 +930,22 @@ cleanlib:
> > # Extract common files list in the dependency files
> > #
> > for File in DepSet:
> >
> > self.CommonFileDependency.append(self.PlaceMacro(File.Path,
> > self.Macros))
> >
> > + CmdSumDict = {}
> > + CmdTargetDict = {}
> > + CmdCppDict = {}
> > + DependencyDict = FileDependencyDict.copy()
> > for File in FileDependencyDict:
> > # skip non-C files
> > if File.Ext not in [".c", ".C"] or File.Name == "AutoGen.c":
> > continue
> > NewDepSet = set(FileDependencyDict[File])
> > NewDepSet -= DepSet
> > FileDependencyDict[File] = ["$(COMMON_DEPS)"] +
> > list(NewDepSet)
> > + DependencyDict[File] = list(NewDepSet)
> >
> > # Convert target description object to target string in makefile
> > for Type in self._AutoGenObject.Targets:
> > for T in self._AutoGenObject.Targets[Type]:
> > # Generate related macros if needed @@ -943,15
> > +953,25 @@ cleanlib:
> > self.FileListMacros[T.FileListMacro] = []
> > if T.GenListFile and T.ListFileMacro not in self.ListFileMacros:
> > self.ListFileMacros[T.ListFileMacro] = []
> > if T.GenIncListFile and T.IncListFileMacro not in self.ListFileMacros:
> > self.ListFileMacros[T.IncListFileMacro] = []
> > + if self._AutoGenObject.BuildRuleFamily == TAB_COMPILER_MSFT and Type == TAB_C_CODE_FILE:
> > + NewFile = self.PlaceMacro(str(T), self.Macros)
> > + if self.ObjTargetDict.get(T.Target.SubDir):
> > + self.ObjTargetDict[T.Target.SubDir].add(NewFile)
> > + else:
> > + self.ObjTargetDict[T.Target.SubDir] = set()
> > +
> > + self.ObjTargetDict[T.Target.SubDir].add(NewFile)
> >
> > Deps = []
> > + CCodeDeps = []
> > # Add force-dependencies
> > for Dep in T.Dependencies:
> > Deps.append(self.PlaceMacro(str(Dep),
> > self.Macros))
> > + if Dep != '$(MAKE_FILE)':
> > + CCodeDeps.append(self.PlaceMacro(str(Dep),
> > + self.Macros))
> > # Add inclusion-dependencies
> > if len(T.Inputs) == 1 and T.Inputs[0] in FileDependencyDict:
> > for F in FileDependencyDict[T.Inputs[0]]:
> > Deps.append(self.PlaceMacro(str(F), self.Macros))
> > # Add source-dependencies @@ -971,16 +991,63 @@
> > cleanlib:
> > if T.GenFileListMacro:
> > Deps.append("$(%s)" % T.FileListMacro)
> > if Type in [TAB_OBJECT_FILE, TAB_STATIC_LIBRARY]:
> > Deps.append("$(%s)" % T.ListFileMacro)
> >
> > - TargetDict = {
> > - "target" : self.PlaceMacro(T.Target.Path, self.Macros),
> > - "cmd" : "\n\t".join(T.Commands),
> > - "deps" : Deps
> > - }
> > - self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict))
> > + if self._AutoGenObject.BuildRuleFamily == TAB_COMPILER_MSFT and Type == TAB_C_CODE_FILE:
> > + T, CmdTarget, CmdTargetDict, CmdCppDict =
> > + self.ParserCCodeFile(T, Type, CmdSumDict, CmdTargetDict,
> > CmdCppDict, DependencyDict)
> > + TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps":
> > CCodeDeps}
> > + CmdLine = self._BUILD_TARGET_TEMPLATE.Replace(TargetDict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST')
> > + if T.Commands:
> > + CmdLine = '%s%s' %(CmdLine, TAB_LINE_BREAK)
> > + if CCodeDeps or CmdLine:
> > + self.BuildTargetList.append(CmdLine)
> > + else:
> > + TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps}
> > +
> > + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Targ
> > + etDict))
> > +
> > + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict):
> > + if not CmdSumDict:
> > + for item in self._AutoGenObject.Targets[Type]:
> > + CmdSumDict[item.Target.SubDir] = item.Target.BaseName
> > + for CppPath in item.Inputs:
> > + Path = self.PlaceMacro(CppPath.Path, self.Macros)
> > + if CmdCppDict.get(item.Target.SubDir):
> > + CmdCppDict[item.Target.SubDir].append(Path)
> > + else:
> > + CmdCppDict[item.Target.SubDir] = ['$(MAKE_FILE)', Path]
> > + if CppPath.Path in DependencyDict:
> > + for Temp in DependencyDict[CppPath.Path]:
> > + Path = self.PlaceMacro(Temp.Path, self.Macros)
> > + if Path not in (self.CommonFileDependency + CmdCppDict[item.Target.SubDir]):
> > + CmdCppDict[item.Target.SubDir].append(Path)
> > + if T.Commands:
> > + CommandList = T.Commands[:]
> > + for Item in CommandList[:]:
> > + SingleCommandList = Item.split()
> > + if len(SingleCommandList) > 0 and '$(CC)' in SingleCommandList[0]:
> > + for Temp in SingleCommandList:
> > + if Temp.startswith('/Fo'):
> > + CmdSign = '%s%s' % (Temp.rsplit(TAB_SLASH, 1)[0], TAB_SLASH)
> > + break
> > + else: continue
> > + if CmdSign not in list(CmdTargetDict.keys()):
> > + CmdLine = Item.replace(Temp, CmdSign)
> > + if GlobalData.gOptions.ThreadNumber:
> > + CmdLine = CmdLine.replace('/MP7', '%s%d' % ('/MP', GlobalData.gOptions.ThreadNumber))
> > + CmdTargetDict[CmdSign] = CmdLine
> > + else:
> > + CmdTargetDict[CmdSign] = "%s %s" % (CmdTargetDict[CmdSign], SingleCommandList[-1])
> > + Index = CommandList.index(Item)
> > + CommandList.pop(Index)
> > + if SingleCommandList[-1].endswith("%s%s.c" %
> > + (TAB_SLASH, CmdSumDict[CmdSign.lstrip('/Fo').rsplit(TAB_SLASH,
> > 1)[0]])):
> > + Cpplist = CmdCppDict[T.Target.SubDir]
> > + Cpplist.insert(0, '$(OBJLIST_%d): $(COMMON_DEPS)' % list(self.ObjTargetDict.keys()).index(T.Target.SubDir))
> > + T.Commands[Index] = '%s\n\t%s' % (' \\\n\t'.join(Cpplist), CmdTargetDict[CmdSign])
> > + else:
> > + T.Commands.pop(Index)
> > + return T, CmdSumDict, CmdTargetDict, CmdCppDict
> >
> > ## For creating makefile targets for dependent libraries
> > def ProcessDependentLibrary(self):
> > for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList:
> > if not LibraryAutoGen.IsBinaryModule:
> > --
> > 2.20.1.windows.1
next prev parent reply other threads:[~2019-04-03 2:45 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-29 12:30 [Patch] BaseTools:Enable the /MP option of MSVC compiler Feng, Bob C
2019-03-29 14:20 ` Gao, Liming
2019-04-03 2:11 ` Feng, Bob C
2019-04-03 2:45 ` Gao, Liming [this message]
2019-03-30 1:42 ` Ni, Ray
2019-03-30 8:14 ` Minnow Ware
2019-04-01 14:55 ` Feng, Bob C
2019-03-30 11:19 ` Feng, Bob C
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4A89E2EF3DFEDB4C8BFDE51014F606A14E41D00E@SHSMSX104.ccr.corp.intel.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox