From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.120; helo=mga04.intel.com; envelope-from=jaben.carsey@intel.com; receiver=edk2-devel@lists.01.org Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1411D21A07A80 for ; Thu, 8 Nov 2018 07:26:28 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Nov 2018 07:26:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,480,1534834800"; d="scan'208";a="94678131" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by FMSMGA003.fm.intel.com with ESMTP; 08 Nov 2018 07:26:26 -0800 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.408.0; Thu, 8 Nov 2018 07:26:20 -0800 Received: from fmsmsx101.amr.corp.intel.com ([169.254.1.202]) by FMSMSX112.amr.corp.intel.com ([169.254.5.114]) with mapi id 14.03.0415.000; Thu, 8 Nov 2018 07:26:19 -0800 From: "Carsey, Jaben" To: "Feng, Bob C" , "edk2-devel@lists.01.org" CC: "Gao, Liming" Thread-Topic: [Patch] BaseTools: Optimize string concatenation Thread-Index: AQHUd0wsD9geLYhxsE+D9ZBManKXOKVGAC4w Date: Thu, 8 Nov 2018 15:26:18 +0000 Message-ID: References: <20181108101625.41364-1-bob.c.feng@intel.com> In-Reply-To: <20181108101625.41364-1-bob.c.feng@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOWM1MzJjNzAtMTM2OC00YmNkLThkMDctNGZhMGQwZjk4NDkyIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiOWZjdzVySzNTQkVqRmJ2eDQ2dW11aWZCQ0JHSm5PVWg0ejBCdnVsVTg4ZjlLNFdPYmMyWmxmQlNFTmwwZ05vTCJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.1.200.107] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Optimize string concatenation X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2018 15:26:29 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Jaben Carsey > -----Original Message----- > From: Feng, Bob C > Sent: Thursday, November 08, 2018 2:16 AM > To: edk2-devel@lists.01.org > Cc: Gao, Liming ; Carsey, Jaben > > Subject: [Patch] BaseTools: Optimize string concatenation > Importance: High >=20 > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1288 >=20 > This patch is one of build tool performance improvement > series patches. >=20 > This patch is going to use join function instead of > string +=3D string2 statement. >=20 > Current code use string +=3D string2 in a loop to combine > a string. while creating a string list in a loop and using > "".join(stringlist) after the loop will be much faster. >=20 > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: BobCF > Cc: Liming Gao > Cc: Jaben Carsey > --- > BaseTools/Source/Python/AutoGen/StrGather.py | 39 +++++++++++++---- > -- > BaseTools/Source/Python/Common/Misc.py | 21 +++++----- > .../Source/Python/Workspace/InfBuildData.py | 4 +- > .../Python/Workspace/WorkspaceCommon.py | 11 ++---- > 4 files changed, 44 insertions(+), 31 deletions(-) >=20 > diff --git a/BaseTools/Source/Python/AutoGen/StrGather.py > b/BaseTools/Source/Python/AutoGen/StrGather.py > index 361d499076..d34a9e9447 100644 > --- a/BaseTools/Source/Python/AutoGen/StrGather.py > +++ b/BaseTools/Source/Python/AutoGen/StrGather.py > @@ -135,11 +135,11 @@ def AscToHexList(Ascii): > # @param UniGenCFlag UniString is generated into AutoGen C file whe= n it > is set to True > # > # @retval Str: A string of .h file content > # > def CreateHFileContent(BaseName, UniObjectClass, IsCompatibleMode, > UniGenCFlag): > - Str =3D '' > + Str =3D [] > ValueStartPtr =3D 60 > Line =3D COMMENT_DEFINE_STR + ' ' + LANGUAGE_NAME_STRING_NAME > + ' ' * (ValueStartPtr - len(DEFINE_STR + > LANGUAGE_NAME_STRING_NAME)) + DecToHexStr(0, 4) + > COMMENT_NOT_REFERENCED > Str =3D WriteLine(Str, Line) > Line =3D COMMENT_DEFINE_STR + ' ' + > PRINTABLE_LANGUAGE_NAME_STRING_NAME + ' ' * (ValueStartPtr - > len(DEFINE_STR + PRINTABLE_LANGUAGE_NAME_STRING_NAME)) + > DecToHexStr(1, 4) + COMMENT_NOT_REFERENCED > Str =3D WriteLine(Str, Line) > @@ -164,16 +164,16 @@ def CreateHFileContent(BaseName, > UniObjectClass, IsCompatibleMode, UniGenCFlag): > Line =3D COMMENT_DEFINE_STR + ' ' + Name + ' ' + > DecToHexStr(Token, 4) + COMMENT_NOT_REFERENCED > else: > Line =3D COMMENT_DEFINE_STR + ' ' + Name + ' ' * (Va= lueStartPtr - > len(DEFINE_STR + Name)) + DecToHexStr(Token, 4) + > COMMENT_NOT_REFERENCED > UnusedStr =3D WriteLine(UnusedStr, Line) >=20 > - Str =3D ''.join([Str, UnusedStr]) > + Str.extend( UnusedStr) >=20 > Str =3D WriteLine(Str, '') > if IsCompatibleMode or UniGenCFlag: > Str =3D WriteLine(Str, 'extern unsigned char ' + BaseName + 'Str= ings[];') > - return Str > + return "".join(Str) >=20 > ## Create a complete .h file > # > # Create a complet .h file with file header and file content > # > @@ -185,11 +185,11 @@ def CreateHFileContent(BaseName, > UniObjectClass, IsCompatibleMode, UniGenCFlag): > # @retval Str: A string of complete .h file > # > def CreateHFile(BaseName, UniObjectClass, IsCompatibleMode, > UniGenCFlag): > HFile =3D WriteLine('', CreateHFileContent(BaseName, UniObjectClass, > IsCompatibleMode, UniGenCFlag)) >=20 > - return HFile > + return "".join(HFile) >=20 > ## Create a buffer to store all items in an array > # > # @param BinBuffer Buffer to contain Binary data. > # @param Array: The array need to be formatted > @@ -209,11 +209,11 @@ def CreateBinBuffer(BinBuffer, Array): > # > def CreateArrayItem(Array, Width =3D 16): > MaxLength =3D Width > Index =3D 0 > Line =3D ' ' > - ArrayItem =3D '' > + ArrayItem =3D [] >=20 > for Item in Array: > if Index < MaxLength: > Line =3D Line + Item + ', ' > Index =3D Index + 1 > @@ -221,11 +221,11 @@ def CreateArrayItem(Array, Width =3D 16): > ArrayItem =3D WriteLine(ArrayItem, Line) > Line =3D ' ' + Item + ', ' > Index =3D 1 > ArrayItem =3D Write(ArrayItem, Line.rstrip()) >=20 > - return ArrayItem > + return "".join(ArrayItem) >=20 > ## CreateCFileStringValue > # > # Create a line with string value > # > @@ -236,11 +236,11 @@ def CreateArrayItem(Array, Width =3D 16): >=20 > def CreateCFileStringValue(Value): > Value =3D [StringBlockType] + Value > Str =3D WriteLine('', CreateArrayItem(Value)) >=20 > - return Str > + return "".join(Str) >=20 > ## GetFilteredLanguage > # > # apply get best language rules to the UNI language code list > # > @@ -438,11 +438,11 @@ def CreateCFileContent(BaseName, > UniObjectClass, IsCompatibleMode, UniBinBuffer, > # > # Join package data > # > AllStr =3D Write(AllStr, Str) >=20 > - return AllStr > + return "".join(AllStr) >=20 > ## Create end of .c file > # > # Create end of .c file > # > @@ -465,11 +465,11 @@ def CreateCFileEnd(): > # > def CreateCFile(BaseName, UniObjectClass, IsCompatibleMode, FilterInfo): > CFile =3D '' > CFile =3D WriteLine(CFile, CreateCFileContent(BaseName, UniObjectCla= ss, > IsCompatibleMode, None, FilterInfo)) > CFile =3D WriteLine(CFile, CreateCFileEnd()) > - return CFile > + return "".join(CFile) >=20 > ## GetFileList > # > # Get a list for all files > # > @@ -572,17 +572,34 @@ def GetStringFiles(UniFilList, SourceFileList, > IncludeList, IncludePathList, Ski >=20 > # > # Write an item > # > def Write(Target, Item): > - return ''.join([Target, Item]) > + if isinstance(Target,str): > + Target =3D [Target] > + if not Target: > + Target =3D [] > + if isinstance(Item,list): > + Target.extend(Item) > + else: > + Target.append(Item) > + return Target >=20 > # > # Write an item with a break line > # > def WriteLine(Target, Item): > - return ''.join([Target, Item, '\n']) > + if isinstance(Target,str): > + Target =3D [Target] > + if not Target: > + Target =3D [] > + if isinstance(Item, list): > + Target.extend(Item) > + else: > + Target.append(Item) > + Target.append('\n') > + return Target >=20 > # This acts like the main() function for the script, unless it is 'impor= t'ed into > another > # script. > if __name__ =3D=3D '__main__': > EdkLogger.info('start') > diff --git a/BaseTools/Source/Python/Common/Misc.py > b/BaseTools/Source/Python/Common/Misc.py > index 80236db160..8dcbe141ae 100644 > --- a/BaseTools/Source/Python/Common/Misc.py > +++ b/BaseTools/Source/Python/Common/Misc.py > @@ -777,21 +777,21 @@ class TemplateString(object): >=20 > return "".join(StringList) >=20 > ## Constructor > def __init__(self, Template=3DNone): > - self.String =3D '' > + self.String =3D [] > self.IsBinary =3D False > self._Template =3D Template > self._TemplateSectionList =3D self._Parse(Template) >=20 > ## str() operator > # > # @retval string The string replaced > # > def __str__(self): > - return self.String > + return "".join(self.String) >=20 > ## Split the template string into fragments per the ${BEGIN} and ${E= ND} > flags > # > # @retval list A list of TemplateString.Section objects > # > @@ -835,13 +835,16 @@ class TemplateString(object): > # @param Dictionary The placeholder dictionaries > # > def Append(self, AppendString, Dictionary=3DNone): > if Dictionary: > SectionList =3D self._Parse(AppendString) > - self.String +=3D "".join(S.Instantiate(Dictionary) for S in = SectionList) > + self.String.append( "".join(S.Instantiate(Dictionary) for S = in > SectionList)) > else: > - self.String +=3D AppendString > + if isinstance(AppendString,list): > + self.String.extend(AppendString) > + else: > + self.String.append(AppendString) >=20 > ## Replace the string template with dictionary of placeholders > # > # @param Dictionary The placeholder dictionaries > # > @@ -1741,27 +1744,21 @@ class PathClass(object): > # > # @retval False The two PathClass are different > # @retval True The two PathClass are the same > # > def __eq__(self, Other): > - if isinstance(Other, type(self)): > - return self.Path =3D=3D Other.Path > - else: > - return self.Path =3D=3D str(Other) > + return self.Path =3D=3D str(Other) >=20 > ## Override __cmp__ function > # > # Customize the comparsion operation of two PathClass > # > # @retval 0 The two PathClass are different > # @retval -1 The first PathClass is less than the second PathClas= s > # @retval 1 The first PathClass is Bigger than the second PathCl= ass > def __cmp__(self, Other): > - if isinstance(Other, type(self)): > - OtherKey =3D Other.Path > - else: > - OtherKey =3D str(Other) > + OtherKey =3D str(Other) >=20 > SelfKey =3D self.Path > if SelfKey =3D=3D OtherKey: > return 0 > elif SelfKey > OtherKey: > diff --git a/BaseTools/Source/Python/Workspace/InfBuildData.py > b/BaseTools/Source/Python/Workspace/InfBuildData.py > index 44d44d24eb..d615cccdf7 100644 > --- a/BaseTools/Source/Python/Workspace/InfBuildData.py > +++ b/BaseTools/Source/Python/Workspace/InfBuildData.py > @@ -612,11 +612,13 @@ class InfBuildData(ModuleBuildClassObject): > for Record in RecordList: > Lib =3D Record[0] > Instance =3D Record[1] > if Instance: > Instance =3D NormPath(Instance, self._Macros) > - RetVal[Lib] =3D Instance > + RetVal[Lib] =3D Instance > + else: > + RetVal[Lib] =3D None > return RetVal >=20 > ## Retrieve library names (for Edk.x style of modules) > @cached_property > def Libraries(self): > diff --git a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > b/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > index 8d8a3e2789..55d01fa4b2 100644 > --- a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > +++ b/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > @@ -126,17 +126,14 @@ def GetModuleLibInstances(Module, Platform, > BuildDatabase, Arch, Target, Toolcha > while len(LibraryConsumerList) > 0: > M =3D LibraryConsumerList.pop() > for LibraryClassName in M.LibraryClasses: > if LibraryClassName not in LibraryInstance: > # override library instance for this module > - if LibraryClassName in > Platform.Modules[str(Module)].LibraryClasses: > - LibraryPath =3D > Platform.Modules[str(Module)].LibraryClasses[LibraryClassName] > - else: > - LibraryPath =3D Platform.LibraryClasses[LibraryClass= Name, > ModuleType] > - if LibraryPath is None or LibraryPath =3D=3D "": > - LibraryPath =3D M.LibraryClasses[LibraryClassName] > - if LibraryPath is None or LibraryPath =3D=3D "": > + LibraryPath =3D > Platform.Modules[str(Module)].LibraryClasses.get(LibraryClassName,Platfor > m.LibraryClasses[LibraryClassName, ModuleType]) > + if LibraryPath is None: > + LibraryPath =3D M.LibraryClasses.get(LibraryClassNam= e) > + if LibraryPath is None: > if FileName: > EdkLogger.error("build", RESOURCE_NOT_AVAILA= BLE, > "Instance of library class [= %s] is not found" % > LibraryClassName, > File=3DFileName, > ExtraData=3D"in [%s] [%s]\n\= tconsumed by module > [%s]" % (str(M), Arch, str(Module))) > -- > 2.19.1.windows.1