From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.151; helo=mga17.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 9CC91211F1E30 for ; Tue, 2 Apr 2019 19:45:16 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Apr 2019 19:45:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,302,1549958400"; d="scan'208";a="334483679" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by fmsmga005.fm.intel.com with ESMTP; 02 Apr 2019 19:45:14 -0700 Received: from fmsmsx162.amr.corp.intel.com (10.18.125.71) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 2 Apr 2019 19:45:15 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by fmsmsx162.amr.corp.intel.com (10.18.125.71) with Microsoft SMTP Server (TLS) id 14.3.408.0; Tue, 2 Apr 2019 19:45:15 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.92]) by shsmsx102.ccr.corp.intel.com ([169.254.2.206]) with mapi id 14.03.0415.000; Wed, 3 Apr 2019 10:45:13 +0800 From: "Gao, Liming" To: "Feng, Bob C" , "edk2-devel@lists.01.org" Thread-Topic: [Patch] BaseTools:Enable the /MP option of MSVC compiler Thread-Index: AQHU5itDe/2xXyYkhEOhUebXOE9lBqYipk7AgAaMYYCAAIamgA== Date: Wed, 3 Apr 2019 02:45:12 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E41D00E@SHSMSX104.ccr.corp.intel.com> References: <20190329123007.52756-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E40ED94@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D1600D19B0@SHSMSX101.ccr.corp.intel.com> In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D1600D19B0@SHSMSX101.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYmNiMWUwNGUtZjI2OC00MzFjLWE3ZDItZTE1ODI0MWUyZWQ0IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiSjVNVTRSV08xcTFyUFBCcnFCVE1hZ0pHYWZxcEtIZUl1K0hGbnN5QW0zbURaVDhWXC9pdnRSNEQxUURBc2lQSXEifQ== dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools:Enable the /MP option of MSVC compiler X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 02:45:16 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The clean build performance data is good.=20 This change updates build rule generation for C source files on MSFT tool c= hain. It is a base tool behavior change. If it doesn't depend on build rule= , could you drop the change in build rule.txt. MP flag will be moved to too= ls_def.txt.=20 Thanks Liming > -----Original Message----- > From: Feng, Bob C > Sent: Wednesday, April 3, 2019 10:12 AM > To: Gao, Liming ; edk2-devel@lists.01.org > Cc: Fan, ZhijuX > Subject: RE: [Patch] BaseTools:Enable the /MP option of MSVC compiler >=20 > Liming, >=20 > I add the performance data on the BZ: > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1672 >=20 > Thanks, > Bob >=20 > -----Original Message----- > From: Gao, Liming > Sent: Friday, March 29, 2019 10:20 PM > To: Feng, Bob C ; edk2-devel@lists.01.org > Cc: Fan, ZhijuX > Subject: RE: [Patch] BaseTools:Enable the /MP option of MSVC compiler >=20 > Bob: > How about use /MP option without process number? The compiler will retr= ieves the number of effective processors on your computer. > And, move this option into CC_FLAGS in tools_def.txt. If so, platform c= an override it in platform DSC file. >=20 > Last, please provide the performance data with this option. >=20 > Thanks > Liming > > -----Original Message----- > > From: Feng, Bob C > > Sent: Friday, March 29, 2019 8:30 PM > > To: edk2-devel@lists.01.org > > Cc: Fan, ZhijuX ; Feng, Bob C > > ; Gao, Liming > > Subject: [Patch] BaseTools:Enable the /MP option of MSVC compiler > > > > From: Zhiju Fan > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1672 > > The /MP option of MSVC compiler can reduce the total time to compile > > the source files on the command line. > > > > This patch is going to enable this MSVC option in BaseTools. > > > > Cc: Bob Feng > > Cc: Liming Gao > > Contributed-under: TianoCore Contribution Agreement 1.1 > > Signed-off-by: Zhiju.Fan > > --- > > BaseTools/Conf/build_rule.template | 2 +- > > BaseTools/Source/Python/AutoGen/GenMake.py | 81 > > ++++++++++++++++++++-- > > 2 files changed, 75 insertions(+), 8 deletions(-) > > > > diff --git a/BaseTools/Conf/build_rule.template > > b/BaseTools/Conf/build_rule.template > > index e56b1d9c59..e7d736740f 100755 > > --- a/BaseTools/Conf/build_rule.template > > +++ b/BaseTools/Conf/build_rule.template > > @@ -127,11 +127,11 @@ > > > > > > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.obj > > > > > > - "$(CC)" /Fo${dst} $(CC_FLAGS) $(INC) ${src} > > + "$(CC)" /MP7 /Fo${d_path}\ $(CC_FLAGS) $(INC) ${src} > > > > > > # For RVCTCYGWIN CC_FLAGS must be first to work around pathing= issues > > "$(CC)" $(CC_FLAGS) -c -o ${dst} $(INC) ${src} > > > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py > > b/BaseTools/Source/Python/AutoGen/GenMake.py > > index b441817b52..04951346ad 100644 > > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > > @@ -433,11 +433,11 @@ cleanlib: > > self.BuildTargetList =3D [] # [target string] > > self.PendingBuildTargetList =3D [] # [FileBuildRule objects= ] > > self.CommonFileDependency =3D [] > > self.FileListMacros =3D {} > > self.ListFileMacros =3D {} > > - > > + self.ObjTargetDict =3D {} > > self.FileCache =3D {} > > self.LibraryBuildCommandList =3D [] > > self.LibraryFileList =3D [] > > self.LibraryMakefileList =3D [] > > self.LibraryBuildDirectoryList =3D [] @@ -616,10 +616,15 @@ > > cleanlib: > > ListFileName, > > "\n".join(self.ListFileMacros[ListFileMacro]), > > False > > ) > > > > + # Generate objlist used to create .obj file > > + for Type in self.ObjTargetDict: > > + NewLine =3D ' '.join(list(self.ObjTargetDict[Type])) > > + FileMacroList.append("OBJLIST_%s =3D %s" % > > + (list(self.ObjTargetDict.keys()).index(Type), NewLine)) > > + > > BcTargetList =3D [] > > > > MakefileName =3D self._FILE_NAME_[self._FileType] > > LibraryMakeCommandList =3D [] > > for D in self.LibraryBuildDirectoryList: > > @@ -925,17 +930,22 @@ cleanlib: > > # Extract common files list in the dependency files > > # > > for File in DepSet: > > > > self.CommonFileDependency.append(self.PlaceMacro(File.Path, > > self.Macros)) > > > > + CmdSumDict =3D {} > > + CmdTargetDict =3D {} > > + CmdCppDict =3D {} > > + DependencyDict =3D FileDependencyDict.copy() > > for File in FileDependencyDict: > > # skip non-C files > > if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoG= en.c": > > continue > > NewDepSet =3D set(FileDependencyDict[File]) > > NewDepSet -=3D DepSet > > FileDependencyDict[File] =3D ["$(COMMON_DEPS)"] + > > list(NewDepSet) > > + DependencyDict[File] =3D list(NewDepSet) > > > > # Convert target description object to target string in makefi= le > > for Type in self._AutoGenObject.Targets: > > for T in self._AutoGenObject.Targets[Type]: > > # Generate related macros if needed @@ -943,15 > > +953,25 @@ cleanlib: > > self.FileListMacros[T.FileListMacro] =3D [] > > if T.GenListFile and T.ListFileMacro not in self.ListF= ileMacros: > > self.ListFileMacros[T.ListFileMacro] =3D [] > > if T.GenIncListFile and T.IncListFileMacro not in self= .ListFileMacros: > > self.ListFileMacros[T.IncListFileMacro] =3D [] > > + if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMP= ILER_MSFT and Type =3D=3D TAB_C_CODE_FILE: > > + NewFile =3D self.PlaceMacro(str(T), self.Macros) > > + if self.ObjTargetDict.get(T.Target.SubDir): > > + self.ObjTargetDict[T.Target.SubDir].add(NewFil= e) > > + else: > > + self.ObjTargetDict[T.Target.SubDir] =3D set() > > + > > + self.ObjTargetDict[T.Target.SubDir].add(NewFile) > > > > Deps =3D [] > > + CCodeDeps =3D [] > > # Add force-dependencies > > for Dep in T.Dependencies: > > Deps.append(self.PlaceMacro(str(Dep), > > self.Macros)) > > + if Dep !=3D '$(MAKE_FILE)': > > + CCodeDeps.append(self.PlaceMacro(str(Dep), > > + self.Macros)) > > # Add inclusion-dependencies > > if len(T.Inputs) =3D=3D 1 and T.Inputs[0] in FileDepen= dencyDict: > > for F in FileDependencyDict[T.Inputs[0]]: > > Deps.append(self.PlaceMacro(str(F), self.Macro= s)) > > # Add source-dependencies @@ -971,16 +991,63 @@ > > cleanlib: > > if T.GenFileListMacro: > > Deps.append("$(%s)" % T.FileListMacro) > > if Type in [TAB_OBJECT_FILE, TAB_STATIC_LIBRARY]: > > Deps.append("$(%s)" % T.ListFileMacro) > > > > - TargetDict =3D { > > - "target" : self.PlaceMacro(T.Target.Path, sel= f.Macros), > > - "cmd" : "\n\t".join(T.Commands), > > - "deps" : Deps > > - } > > - self.BuildTargetList.append(self._BUILD_TARGET_TEMPLAT= E.Replace(TargetDict)) > > + if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMP= ILER_MSFT and Type =3D=3D TAB_C_CODE_FILE: > > + T, CmdTarget, CmdTargetDict, CmdCppDict =3D > > + self.ParserCCodeFile(T, Type, CmdSumDict, CmdTargetDict, > > CmdCppDict, DependencyDict) > > + TargetDict =3D {"target": self.PlaceMacro(T.Target= .Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": > > CCodeDeps} > > + CmdLine =3D self._BUILD_TARGET_TEMPLATE.Replace(Ta= rgetDict).rstrip().replace('\t$(OBJLIST', '$(OBJLIST') > > + if T.Commands: > > + CmdLine =3D '%s%s' %(CmdLine, TAB_LINE_BREAK) > > + if CCodeDeps or CmdLine: > > + self.BuildTargetList.append(CmdLine) > > + else: > > + TargetDict =3D {"target": self.PlaceMacro(T.Target= .Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} > > + > > + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Targ > > + etDict)) > > + > > + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdC= ppDict, DependencyDict): > > + if not CmdSumDict: > > + for item in self._AutoGenObject.Targets[Type]: > > + CmdSumDict[item.Target.SubDir] =3D item.Target.BaseNam= e > > + for CppPath in item.Inputs: > > + Path =3D self.PlaceMacro(CppPath.Path, self.Macros= ) > > + if CmdCppDict.get(item.Target.SubDir): > > + CmdCppDict[item.Target.SubDir].append(Path) > > + else: > > + CmdCppDict[item.Target.SubDir] =3D ['$(MAKE_FI= LE)', Path] > > + if CppPath.Path in DependencyDict: > > + for Temp in DependencyDict[CppPath.Path]: > > + Path =3D self.PlaceMacro(Temp.Path, self.M= acros) > > + if Path not in (self.CommonFileDependency = + CmdCppDict[item.Target.SubDir]): > > + CmdCppDict[item.Target.SubDir].append(= Path) > > + if T.Commands: > > + CommandList =3D T.Commands[:] > > + for Item in CommandList[:]: > > + SingleCommandList =3D Item.split() > > + if len(SingleCommandList) > 0 and '$(CC)' in SingleCom= mandList[0]: > > + for Temp in SingleCommandList: > > + if Temp.startswith('/Fo'): > > + CmdSign =3D '%s%s' % (Temp.rsplit(TAB_SLAS= H, 1)[0], TAB_SLASH) > > + break > > + else: continue > > + if CmdSign not in list(CmdTargetDict.keys()): > > + CmdLine =3D Item.replace(Temp, CmdSign) > > + if GlobalData.gOptions.ThreadNumber: > > + CmdLine =3D CmdLine.replace('/MP7', '%s%d'= % ('/MP', GlobalData.gOptions.ThreadNumber)) > > + CmdTargetDict[CmdSign] =3D CmdLine > > + else: > > + CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTarge= tDict[CmdSign], SingleCommandList[-1]) > > + Index =3D CommandList.index(Item) > > + CommandList.pop(Index) > > + if SingleCommandList[-1].endswith("%s%s.c" % > > + (TAB_SLASH, CmdSumDict[CmdSign.lstrip('/Fo').rsplit(TAB_SLASH, > > 1)[0]])): > > + Cpplist =3D CmdCppDict[T.Target.SubDir] > > + Cpplist.insert(0, '$(OBJLIST_%d): $(COMMON_DEP= S)' % list(self.ObjTargetDict.keys()).index(T.Target.SubDir)) > > + T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.= join(Cpplist), CmdTargetDict[CmdSign]) > > + else: > > + T.Commands.pop(Index) > > + return T, CmdSumDict, CmdTargetDict, CmdCppDict > > > > ## For creating makefile targets for dependent libraries > > def ProcessDependentLibrary(self): > > for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList: > > if not LibraryAutoGen.IsBinaryModule: > > -- > > 2.20.1.windows.1