From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.24; helo=mga09.intel.com; envelope-from=michael.d.kinney@intel.com; receiver=edk2-devel@lists.01.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 65E4621A07A92 for ; Fri, 9 Nov 2018 09:01:25 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Nov 2018 09:01:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,483,1534834800"; d="scan'208";a="85321565" Received: from orsmsx102.amr.corp.intel.com ([10.22.225.129]) by fmsmga008.fm.intel.com with ESMTP; 09 Nov 2018 09:01:24 -0800 Received: from orsmsx114.amr.corp.intel.com (10.22.240.10) by ORSMSX102.amr.corp.intel.com (10.22.225.129) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 9 Nov 2018 09:01:23 -0800 Received: from orsmsx113.amr.corp.intel.com ([169.254.9.125]) by ORSMSX114.amr.corp.intel.com ([169.254.8.128]) with mapi id 14.03.0415.000; Fri, 9 Nov 2018 09:01:22 -0800 From: "Kinney, Michael D" To: "Gao, Liming" , "Feng, Bob C" , "edk2-devel@lists.01.org" , "Kinney, Michael D" CC: "Carsey, Jaben" Thread-Topic: [edk2] [Patch] BaseTools: Optimize string concatenation Thread-Index: AQHUd0w2DVq+M6EpAUenVkvbnPSY3qVGFLkQgAHweID//6dIEA== Date: Fri, 9 Nov 2018 17:01:22 +0000 Message-ID: References: <20181108101625.41364-1-bob.c.feng@intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E368206@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <4A89E2EF3DFEDB4C8BFDE51014F606A14E368206@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.22.254.140] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Optimize string concatenation X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 17:01:26 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Liming, If we can support both Python2 and Python3 equally, then I agree that these types of performance enhancements can be added to edk2/master after the stable tag. Let's make sure we enter a BZ for each performance improvement and as Leif has asked, put evidence of the performance improvement in the BZ. Mike > -----Original Message----- > From: Gao, Liming > Sent: Friday, November 9, 2018 6:17 AM > To: Kinney, Michael D ; > Feng, Bob C ; edk2- > devel@lists.01.org > Cc: Carsey, Jaben > Subject: RE: [edk2] [Patch] BaseTools: Optimize string > concatenation >=20 > Mike: > This patch bases on edk2 master with Python27. > Seemly, most people are not aware that Python3 > migration > (https://bugzilla.tianocore.org/show_bug.cgi?id=3D55) is > added for next edk2-stable201903 tag planning. In BZ > 55, I propose to keep Python2 and Python3 both, and new > feature and patches are created based on Python3. I > will send the mail to collect the feedback on this > approach. >=20 > Thanks > Liming > > -----Original Message----- > > From: Kinney, Michael D > > Sent: Friday, November 9, 2018 12:41 AM > > To: Feng, Bob C ; edk2- > devel@lists.01.org; Kinney, Michael D > > > Cc: Carsey, Jaben ; Gao, > Liming > > Subject: RE: [edk2] [Patch] BaseTools: Optimize > string concatenation > > > > Bob, > > > > Is this for edk2/master or for the Python 3 > conversion in the > > edk2-staging branch? If it is for the edk-staging > branch, then > > the Subject is not correct. > > > > Thanks, > > > > Mike > > > > > -----Original Message----- > > > From: edk2-devel [mailto:edk2-devel- > > > bounces@lists.01.org] On Behalf Of BobCF > > > Sent: Thursday, November 8, 2018 2:16 AM > > > To: edk2-devel@lists.01.org > > > Cc: Carsey, Jaben ; Gao, > Liming > > > > > > Subject: [edk2] [Patch] BaseTools: Optimize string > > > concatenation > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1288 > > > > > > This patch is one of build tool performance > improvement > > > series patches. > > > > > > This patch is going to use join function instead of > > > string +=3D string2 statement. > > > > > > Current code use string +=3D string2 in a loop to > combine > > > a string. while creating a string list in a loop > and > > > using > > > "".join(stringlist) after the loop will be much > faster. > > > > > > Contributed-under: TianoCore Contribution Agreement > 1.1 > > > Signed-off-by: BobCF > > > Cc: Liming Gao > > > Cc: Jaben Carsey > > > --- > > > BaseTools/Source/Python/AutoGen/StrGather.py | 39 > > > +++++++++++++------ > > > BaseTools/Source/Python/Common/Misc.py | 21 > > > +++++----- > > > .../Source/Python/Workspace/InfBuildData.py | 4 > +- > > > .../Python/Workspace/WorkspaceCommon.py | 11 > ++- > > > --- > > > 4 files changed, 44 insertions(+), 31 deletions(-) > > > > > > diff --git > > > a/BaseTools/Source/Python/AutoGen/StrGather.py > > > b/BaseTools/Source/Python/AutoGen/StrGather.py > > > index 361d499076..d34a9e9447 100644 > > > --- a/BaseTools/Source/Python/AutoGen/StrGather.py > > > +++ b/BaseTools/Source/Python/AutoGen/StrGather.py > > > @@ -135,11 +135,11 @@ def AscToHexList(Ascii): > > > # @param UniGenCFlag UniString is generated > into > > > AutoGen C file when it is set to True > > > # > > > # @retval Str: A string of .h file > content > > > # > > > def CreateHFileContent(BaseName, UniObjectClass, > > > IsCompatibleMode, UniGenCFlag): > > > - Str =3D '' > > > + Str =3D [] > > > ValueStartPtr =3D 60 > > > Line =3D COMMENT_DEFINE_STR + ' ' + > > > LANGUAGE_NAME_STRING_NAME + ' ' * (ValueStartPtr - > > > len(DEFINE_STR + LANGUAGE_NAME_STRING_NAME)) + > > > DecToHexStr(0, 4) + COMMENT_NOT_REFERENCED > > > Str =3D WriteLine(Str, Line) > > > Line =3D COMMENT_DEFINE_STR + ' ' + > > > PRINTABLE_LANGUAGE_NAME_STRING_NAME + ' ' * > > > (ValueStartPtr - len(DEFINE_STR + > > > PRINTABLE_LANGUAGE_NAME_STRING_NAME)) + > DecToHexStr(1, > > > 4) + COMMENT_NOT_REFERENCED > > > Str =3D WriteLine(Str, Line) > > > @@ -164,16 +164,16 @@ def > CreateHFileContent(BaseName, > > > UniObjectClass, IsCompatibleMode, UniGenCFlag): > > > Line =3D COMMENT_DEFINE_STR + ' > ' + > > > Name + ' ' + DecToHexStr(Token, 4) + > > > COMMENT_NOT_REFERENCED > > > else: > > > Line =3D COMMENT_DEFINE_STR + ' > ' + > > > Name + ' ' * (ValueStartPtr - len(DEFINE_STR + > Name)) + > > > DecToHexStr(Token, 4) + COMMENT_NOT_REFERENCED > > > UnusedStr =3D WriteLine(UnusedStr, > Line) > > > > > > - Str =3D ''.join([Str, UnusedStr]) > > > + Str.extend( UnusedStr) > > > > > > Str =3D WriteLine(Str, '') > > > if IsCompatibleMode or UniGenCFlag: > > > Str =3D WriteLine(Str, 'extern unsigned char > ' + > > > BaseName + 'Strings[];') > > > - return Str > > > + return "".join(Str) > > > > > > ## Create a complete .h file > > > # > > > # Create a complet .h file with file header and > file > > > content > > > # > > > @@ -185,11 +185,11 @@ def > CreateHFileContent(BaseName, > > > UniObjectClass, IsCompatibleMode, UniGenCFlag): > > > # @retval Str: A string of complete .h > file > > > # > > > def CreateHFile(BaseName, UniObjectClass, > > > IsCompatibleMode, UniGenCFlag): > > > HFile =3D WriteLine('', > CreateHFileContent(BaseName, > > > UniObjectClass, IsCompatibleMode, UniGenCFlag)) > > > > > > - return HFile > > > + return "".join(HFile) > > > > > > ## Create a buffer to store all items in an array > > > # > > > # @param BinBuffer Buffer to contain Binary > data. > > > # @param Array: The array need to be > formatted > > > @@ -209,11 +209,11 @@ def > CreateBinBuffer(BinBuffer, > > > Array): > > > # > > > def CreateArrayItem(Array, Width =3D 16): > > > MaxLength =3D Width > > > Index =3D 0 > > > Line =3D ' ' > > > - ArrayItem =3D '' > > > + ArrayItem =3D [] > > > > > > for Item in Array: > > > if Index < MaxLength: > > > Line =3D Line + Item + ', ' > > > Index =3D Index + 1 > > > @@ -221,11 +221,11 @@ def CreateArrayItem(Array, > Width > > > =3D 16): > > > ArrayItem =3D WriteLine(ArrayItem, Line) > > > Line =3D ' ' + Item + ', ' > > > Index =3D 1 > > > ArrayItem =3D Write(ArrayItem, Line.rstrip()) > > > > > > - return ArrayItem > > > + return "".join(ArrayItem) > > > > > > ## CreateCFileStringValue > > > # > > > # Create a line with string value > > > # > > > @@ -236,11 +236,11 @@ def CreateArrayItem(Array, > Width > > > =3D 16): > > > > > > def CreateCFileStringValue(Value): > > > Value =3D [StringBlockType] + Value > > > Str =3D WriteLine('', CreateArrayItem(Value)) > > > > > > - return Str > > > + return "".join(Str) > > > > > > ## GetFilteredLanguage > > > # > > > # apply get best language rules to the UNI > language > > > code list > > > # > > > @@ -438,11 +438,11 @@ def > CreateCFileContent(BaseName, > > > UniObjectClass, IsCompatibleMode, UniBinBuffer, > > > # > > > # Join package data > > > # > > > AllStr =3D Write(AllStr, Str) > > > > > > - return AllStr > > > + return "".join(AllStr) > > > > > > ## Create end of .c file > > > # > > > # Create end of .c file > > > # > > > @@ -465,11 +465,11 @@ def CreateCFileEnd(): > > > # > > > def CreateCFile(BaseName, UniObjectClass, > > > IsCompatibleMode, FilterInfo): > > > CFile =3D '' > > > CFile =3D WriteLine(CFile, > > > CreateCFileContent(BaseName, UniObjectClass, > > > IsCompatibleMode, None, FilterInfo)) > > > CFile =3D WriteLine(CFile, CreateCFileEnd()) > > > - return CFile > > > + return "".join(CFile) > > > > > > ## GetFileList > > > # > > > # Get a list for all files > > > # > > > @@ -572,17 +572,34 @@ def > GetStringFiles(UniFilList, > > > SourceFileList, IncludeList, IncludePathList, Ski > > > > > > # > > > # Write an item > > > # > > > def Write(Target, Item): > > > - return ''.join([Target, Item]) > > > + if isinstance(Target,str): > > > + Target =3D [Target] > > > + if not Target: > > > + Target =3D [] > > > + if isinstance(Item,list): > > > + Target.extend(Item) > > > + else: > > > + Target.append(Item) > > > + return Target > > > > > > # > > > # Write an item with a break line > > > # > > > def WriteLine(Target, Item): > > > - return ''.join([Target, Item, '\n']) > > > + if isinstance(Target,str): > > > + Target =3D [Target] > > > + if not Target: > > > + Target =3D [] > > > + if isinstance(Item, list): > > > + Target.extend(Item) > > > + else: > > > + Target.append(Item) > > > + Target.append('\n') > > > + return Target > > > > > > # This acts like the main() function for the > script, > > > unless it is 'import'ed into another > > > # script. > > > if __name__ =3D=3D '__main__': > > > EdkLogger.info('start') > > > diff --git a/BaseTools/Source/Python/Common/Misc.py > > > b/BaseTools/Source/Python/Common/Misc.py > > > index 80236db160..8dcbe141ae 100644 > > > --- a/BaseTools/Source/Python/Common/Misc.py > > > +++ b/BaseTools/Source/Python/Common/Misc.py > > > @@ -777,21 +777,21 @@ class TemplateString(object): > > > > > > return "".join(StringList) > > > > > > ## Constructor > > > def __init__(self, Template=3DNone): > > > - self.String =3D '' > > > + self.String =3D [] > > > self.IsBinary =3D False > > > self._Template =3D Template > > > self._TemplateSectionList =3D > > > self._Parse(Template) > > > > > > ## str() operator > > > # > > > # @retval string The string replaced > > > # > > > def __str__(self): > > > - return self.String > > > + return "".join(self.String) > > > > > > ## Split the template string into fragments > per > > > the ${BEGIN} and ${END} flags > > > # > > > # @retval list A list of > > > TemplateString.Section objects > > > # > > > @@ -835,13 +835,16 @@ class TemplateString(object): > > > # @param Dictionary The > placeholder > > > dictionaries > > > # > > > def Append(self, AppendString, > Dictionary=3DNone): > > > if Dictionary: > > > SectionList =3D > self._Parse(AppendString) > > > - self.String +=3D > > > "".join(S.Instantiate(Dictionary) for S in > SectionList) > > > + self.String.append( > > > "".join(S.Instantiate(Dictionary) for S in > > > SectionList)) > > > else: > > > - self.String +=3D AppendString > > > + if isinstance(AppendString,list): > > > + self.String.extend(AppendString) > > > + else: > > > + self.String.append(AppendString) > > > > > > ## Replace the string template with dictionary > of > > > placeholders > > > # > > > # @param Dictionary The > placeholder > > > dictionaries > > > # > > > @@ -1741,27 +1744,21 @@ class PathClass(object): > > > # > > > # @retval False The two PathClass are > different > > > # @retval True The two PathClass are the same > > > # > > > def __eq__(self, Other): > > > - if isinstance(Other, type(self)): > > > - return self.Path =3D=3D Other.Path > > > - else: > > > - return self.Path =3D=3D str(Other) > > > + return self.Path =3D=3D str(Other) > > > > > > ## Override __cmp__ function > > > # > > > # Customize the comparsion operation of two > > > PathClass > > > # > > > # @retval 0 The two PathClass are > different > > > # @retval -1 The first PathClass is less > than > > > the second PathClass > > > # @retval 1 The first PathClass is Bigger > than > > > the second PathClass > > > def __cmp__(self, Other): > > > - if isinstance(Other, type(self)): > > > - OtherKey =3D Other.Path > > > - else: > > > - OtherKey =3D str(Other) > > > + OtherKey =3D str(Other) > > > > > > SelfKey =3D self.Path > > > if SelfKey =3D=3D OtherKey: > > > return 0 > > > elif SelfKey > OtherKey: > > > diff --git > > > a/BaseTools/Source/Python/Workspace/InfBuildData.py > > > b/BaseTools/Source/Python/Workspace/InfBuildData.py > > > index 44d44d24eb..d615cccdf7 100644 > > > --- > a/BaseTools/Source/Python/Workspace/InfBuildData.py > > > +++ > b/BaseTools/Source/Python/Workspace/InfBuildData.py > > > @@ -612,11 +612,13 @@ class > > > InfBuildData(ModuleBuildClassObject): > > > for Record in RecordList: > > > Lib =3D Record[0] > > > Instance =3D Record[1] > > > if Instance: > > > Instance =3D NormPath(Instance, > > > self._Macros) > > > - RetVal[Lib] =3D Instance > > > + RetVal[Lib] =3D Instance > > > + else: > > > + RetVal[Lib] =3D None > > > return RetVal > > > > > > ## Retrieve library names (for Edk.x style of > > > modules) > > > @cached_property > > > def Libraries(self): > > > diff --git > > > > a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > > > > b/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > > > index 8d8a3e2789..55d01fa4b2 100644 > > > --- > > > > a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > > > +++ > > > > b/BaseTools/Source/Python/Workspace/WorkspaceCommon.py > > > @@ -126,17 +126,14 @@ def > GetModuleLibInstances(Module, > > > Platform, BuildDatabase, Arch, Target, Toolcha > > > while len(LibraryConsumerList) > 0: > > > M =3D LibraryConsumerList.pop() > > > for LibraryClassName in M.LibraryClasses: > > > if LibraryClassName not in > > > LibraryInstance: > > > # override library instance for > this > > > module > > > - if LibraryClassName in > > > Platform.Modules[str(Module)].LibraryClasses: > > > - LibraryPath =3D > > > > Platform.Modules[str(Module)].LibraryClasses[LibraryCla > > > ssName] > > > - else: > > > - LibraryPath =3D > > > Platform.LibraryClasses[LibraryClassName, > ModuleType] > > > - if LibraryPath is None or > LibraryPath > > > =3D=3D "": > > > - LibraryPath =3D > > > M.LibraryClasses[LibraryClassName] > > > - if LibraryPath is None or > > > LibraryPath =3D=3D "": > > > + LibraryPath =3D > > > > Platform.Modules[str(Module)].LibraryClasses.get(Librar > > > > yClassName,Platform.LibraryClasses[LibraryClassName, > > > ModuleType]) > > > + if LibraryPath is None: > > > + LibraryPath =3D > > > M.LibraryClasses.get(LibraryClassName) > > > + if LibraryPath is None: > > > if FileName: > > > > EdkLogger.error("build", > > > RESOURCE_NOT_AVAILABLE, > > > > "Instance > > > of library class [%s] is not found" % > LibraryClassName, > > > > > > File=3DFileName, > > > > > > ExtraData=3D"in [%s] [%s]\n\tconsumed by module [%s]" > % > > > (str(M), Arch, str(Module))) > > > -- > > > 2.19.1.windows.1 > > > > > > _______________________________________________ > > > edk2-devel mailing list > > > edk2-devel@lists.01.org > > > https://lists.01.org/mailman/listinfo/edk2-devel