From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.20; helo=mga02.intel.com; envelope-from=jaben.carsey@intel.com; receiver=edk2-devel@lists.01.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 79F7321A07A80 for ; Thu, 8 Nov 2018 07:22:48 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Nov 2018 07:22:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,480,1534834800"; d="scan'208";a="106408675" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga001.fm.intel.com with ESMTP; 08 Nov 2018 07:22:47 -0800 Received: from fmsmsx126.amr.corp.intel.com (10.18.125.43) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Thu, 8 Nov 2018 07:22:47 -0800 Received: from fmsmsx101.amr.corp.intel.com ([169.254.1.202]) by FMSMSX126.amr.corp.intel.com ([169.254.1.5]) with mapi id 14.03.0415.000; Thu, 8 Nov 2018 07:22:47 -0800 From: "Carsey, Jaben" To: "Feng, Bob C" , "edk2-devel@lists.01.org" CC: "Gao, Liming" Thread-Topic: [edk2] [Patch] BaseTools: Customize deepcopy function. Thread-Index: AQHUdyjdyh9n9kpqckWXsH46/mAMrKVF/3XQ Date: Thu, 8 Nov 2018 15:22:45 +0000 Message-ID: References: <20181108060338.40404-1-bob.c.feng@intel.com> In-Reply-To: <20181108060338.40404-1-bob.c.feng@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiODNjYjk4YjMtYWJiOC00Y2EwLThkYzQtOThhZmYxNDg5MmQ4IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoidytWWmZxMHQxc0k4dUlVT3E2VkRTK2RHQ3BpMVFXVzhuT0dSVm83MlJHS0Q5NlArYjhxWHQ1SFZIZHdaOXJ1SyJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.1.200.107] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Customize deepcopy function. X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2018 15:22:48 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Jaben Carsey > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > BobCF > Sent: Wednesday, November 07, 2018 10:04 PM > To: edk2-devel@lists.01.org > Cc: Carsey, Jaben ; Gao, Liming > > Subject: [edk2] [Patch] BaseTools: Customize deepcopy function. > Importance: High >=20 > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1288 >=20 > This patch is one of build tool performance improvement > series patches. >=20 > This patch is going to customize the deepcopy function for > SkuClass, PcdClassObject and python dictionary. >=20 > python deepcopy copy everything of a object, but for our current > usage we just need to copy the data we care about recursively. >=20 > By implementing __deepcopy__ for SkuClass, PcdClassObject, we can > customize > deepcopy function for them. >=20 > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: BobCF > Cc: Liming Gao > Cc: Jaben Carsey > --- > BaseTools/Source/Python/Common/Expression.py | 4 +- > BaseTools/Source/Python/Common/Misc.py | 16 ++++++ > .../Python/CommonDataClass/CommonClass.py | 15 ++++++ > .../Python/Workspace/BuildClassObject.py | 52 +++++++++++++++++++ > .../Source/Python/Workspace/DscBuildData.py | 6 +-- > 5 files changed, 88 insertions(+), 5 deletions(-) >=20 > diff --git a/BaseTools/Source/Python/Common/Expression.py > b/BaseTools/Source/Python/Common/Expression.py > index a21ab5daa7..091c0f4296 100644 > --- a/BaseTools/Source/Python/Common/Expression.py > +++ b/BaseTools/Source/Python/Common/Expression.py > @@ -15,11 +15,11 @@ > from __future__ import print_function > from __future__ import absolute_import > from Common.GlobalData import * > from CommonDataClass.Exceptions import BadExpression > from CommonDataClass.Exceptions import WrnExpression > -from .Misc import GuidStringToGuidStructureString, ParseFieldValue > +from .Misc import GuidStringToGuidStructureString, > ParseFieldValue,CopyDict > import Common.EdkLogger as EdkLogger > import copy > from Common.DataType import * > import sys > from random import sample > @@ -353,11 +353,11 @@ class ValueExpression(BaseExpression): > raise BadExpression(ERR_EMPTY_EXPR) >=20 > # > # The symbol table including PCD and macro mapping > # > - self._Symb =3D copy.deepcopy(SymbolTable) > + self._Symb =3D CopyDict(SymbolTable) > self._Symb.update(self.LogicalOperators) > self._Idx =3D 0 > self._Len =3D len(self._Expr) > self._Token =3D '' > self._WarnExcept =3D None > diff --git a/BaseTools/Source/Python/Common/Misc.py > b/BaseTools/Source/Python/Common/Misc.py > index 3b8efb2e71..80236db160 100644 > --- a/BaseTools/Source/Python/Common/Misc.py > +++ b/BaseTools/Source/Python/Common/Misc.py > @@ -39,10 +39,11 @@ from Common.LongFilePathSupport import > OpenLongFilePath as open > from Common.MultipleWorkspace import MultipleWorkspace as mws > import uuid > from CommonDataClass.Exceptions import BadExpression > from Common.caching import cached_property > import subprocess > +from collections import OrderedDict > ## Regular expression used to find out place holders in string template > gPlaceholderPattern =3D re.compile("\$\{([^$()\s]+)\}", re.MULTILINE | > re.UNICODE) >=20 > ## regular expressions for map file processing > startPatternGeneral =3D re.compile("^Start[' ']+Length[' ']+Name[' ']+Cl= ass") > @@ -2129,10 +2130,25 @@ def PackByteFormatGUID(Guid): > Guid[8], > Guid[9], > Guid[10], > ) >=20 > +## DeepCopy dict/OrderedDict recusively > +# > +# @param ori_dict a nested dict or ordereddict > +# > +# @retval new dict or orderdict > +# > +def CopyDict(ori_dict): > + dict_type =3D ori_dict.__class__ > + new_dict =3D dict_type() > + for key in ori_dict: > + if isinstance(ori_dict[key],(dict,OrderedDict)): > + new_dict[key] =3D CopyDict(ori_dict[key]) > + else: > + new_dict[key] =3D ori_dict[key] > + return new_dict > ## > # > # This acts like the main() function for the script, unless it is 'impor= t'ed into > another > # script. > # > diff --git a/BaseTools/Source/Python/CommonDataClass/CommonClass.py > b/BaseTools/Source/Python/CommonDataClass/CommonClass.py > index a98cf8a7c5..336bb11671 100644 > --- a/BaseTools/Source/Python/CommonDataClass/CommonClass.py > +++ b/BaseTools/Source/Python/CommonDataClass/CommonClass.py > @@ -78,5 +78,20 @@ class SkuInfoClass(object): > 'VariableOffset =3D ' + str(self.VariableOffset) + "= ," + \ > 'HiiDefaultValue =3D ' + str(self.HiiDefaultValue) += "," + \ > 'VpdOffset =3D ' + str(self.VpdOffset) + "," + \ > 'DefaultValue =3D ' + str(self.DefaultValue) + "," > return Rtn > + > + def __deepcopy__(self,memo): > + new_sku =3D SkuInfoClass() > + new_sku.SkuIdName =3D self.SkuIdName > + new_sku.SkuId =3D self.SkuId > + new_sku.VariableName =3D self.VariableName > + new_sku.VariableGuid =3D self.VariableGuid > + new_sku.VariableGuidValue =3D self.VariableGuidValue > + new_sku.VariableOffset =3D self.VariableOffset > + new_sku.HiiDefaultValue =3D self.HiiDefaultValue > + new_sku.VariableAttribute =3D self.VariableAttribute > + new_sku.DefaultStoreDict =3D {key:value for key,value in > self.DefaultStoreDict.items()} > + new_sku.VpdOffset =3D self.VpdOffset > + new_sku.DefaultValue =3D self.DefaultValue > + return new_sku > diff --git a/BaseTools/Source/Python/Workspace/BuildClassObject.py > b/BaseTools/Source/Python/Workspace/BuildClassObject.py > index 95edc376fe..7d10391c1e 100644 > --- a/BaseTools/Source/Python/Workspace/BuildClassObject.py > +++ b/BaseTools/Source/Python/Workspace/BuildClassObject.py > @@ -14,10 +14,12 @@ > from collections import OrderedDict, namedtuple > from Common.DataType import * > import collections > import re > from collections import OrderedDict > +from Common.Misc import CopyDict > +import copy > StructPattern =3D re.compile(r'[_a-zA-Z][0-9A-Za-z_\[\]]*$') > ArrayIndex =3D re.compile("\[\s*\d{0,1}\s*\]") > ## PcdClassObject > # > # This Class is used for PcdObject > @@ -152,10 +154,41 @@ class PcdClassObject(object): > # @retval truple() Key for hash table > # > def __hash__(self): > return hash((self.TokenCName, self.TokenSpaceGuidCName)) >=20 > + def sharedcopy(self,new_pcd): > + new_pcd.TokenCName =3D self.TokenCName > + new_pcd.TokenSpaceGuidCName =3D self.TokenSpaceGuidCName > + new_pcd.TokenSpaceGuidValue =3D self.TokenSpaceGuidValue > + new_pcd.Type =3D self.Type > + new_pcd.DatumType =3D self.DatumType > + new_pcd.DefaultValue =3D self.DefaultValue > + new_pcd.TokenValue =3D self.TokenValue > + new_pcd.MaxDatumSize =3D self.MaxDatumSize > + > + new_pcd.Phase =3D self.Phase > + new_pcd.Pending =3D self.Pending > + new_pcd.IsOverrided =3D self.IsOverrided > + new_pcd.IsFromBinaryInf =3D self.IsFromBinaryInf > + new_pcd.IsFromDsc =3D self.IsFromDsc > + new_pcd.PcdValueFromComm =3D self.PcdValueFromComm > + new_pcd.PcdValueFromFdf =3D self.PcdValueFromFdf > + new_pcd.UserDefinedDefaultStoresFlag =3D > self.UserDefinedDefaultStoresFlag > + new_pcd.DscRawValue =3D self.DscRawValue > + new_pcd.CustomAttribute =3D self.CustomAttribute > + new_pcd.validateranges =3D [item for item in self.validateranges= ] > + new_pcd.validlists =3D [item for item in self.validlists] > + new_pcd.expressions =3D [item for item in self.expressions] > + new_pcd.SkuInfoList =3D {key: copy.deepcopy(skuobj) for key,skuo= bj in > self.SkuInfoList.items()} > + return new_pcd > + > + def __deepcopy__(self,memo): > + new_pcd =3D PcdClassObject() > + self.sharedcopy(new_pcd) > + return new_pcd > + > class StructurePcd(PcdClassObject): > def __init__(self, StructuredPcdIncludeFile=3DNone, Packages=3DNone, > Name=3DNone, Guid=3DNone, Type=3DNone, DatumType=3DNone, Value=3DNone, > Token=3DNone, MaxDatumSize=3DNone, SkuInfoList=3DNone, IsOverrided=3DFals= e, > GuidValue=3DNone, validateranges=3DNone, validlists=3DNone, > expressions=3DNone,default_store =3D TAB_DEFAULT_STORES_DEFAULT): > if SkuInfoList is None: > SkuInfoList =3D {} > if validateranges is None: > @@ -240,10 +273,29 @@ class StructurePcd(PcdClassObject): > self.PkgPath =3D PcdObject.PkgPath if PcdObject.PkgPath else > self.PkgPath > self.ValueChain =3D PcdObject.ValueChain if PcdObject.ValueC= hain else > self.ValueChain > self.PcdFieldValueFromComm =3D PcdObject.PcdFieldValueFromCo= mm > if PcdObject.PcdFieldValueFromComm else self.PcdFieldValueFromComm > self.PcdFieldValueFromFdf =3D PcdObject.PcdFieldValueFromFdf= if > PcdObject.PcdFieldValueFromFdf else self.PcdFieldValueFromFdf >=20 > + def __deepcopy__(self,memo): > + new_pcd =3D StructurePcd() > + self.sharedcopy(new_pcd) > + > + new_pcd.DefaultValueFromDec =3D self.DefaultValueFromDec > + new_pcd.PcdMode =3D self.PcdMode > + new_pcd.StructName =3D self.DatumType > + new_pcd.PcdDefineLineNo =3D self.PcdDefineLineNo > + new_pcd.PkgPath =3D self.PkgPath > + new_pcd.StructuredPcdIncludeFile =3D [item for item in > self.StructuredPcdIncludeFile] > + new_pcd.PackageDecs =3D [item for item in self.PackageDecs] > + new_pcd.DefaultValues =3D CopyDict(self.DefaultValues) > + new_pcd.DefaultFromDSC=3DCopyDict(self.DefaultFromDSC) > + new_pcd.SkuOverrideValues =3D CopyDict(self.SkuOverrideValues) > + new_pcd.PcdFieldValueFromComm =3D > CopyDict(self.PcdFieldValueFromComm) > + new_pcd.PcdFieldValueFromFdf =3D > CopyDict(self.PcdFieldValueFromFdf) > + new_pcd.ValueChain =3D {item for item in self.ValueChain} > + return new_pcd > + > LibraryClassObject =3D namedtuple('LibraryClassObject', > ['LibraryClass','SupModList'], verbose=3DFalse) >=20 > ## ModuleBuildClassObject > # > # This Class defines ModuleBuildClass > diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py > b/BaseTools/Source/Python/Workspace/DscBuildData.py > index 11aa63fb26..5655034b88 100644 > --- a/BaseTools/Source/Python/Workspace/DscBuildData.py > +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py > @@ -970,11 +970,11 @@ class DscBuildData(PlatformBuildClassObject): > for skuid in pcd.SkuInfoList: > skuobj =3D pcd.SkuInfoList.get(skuid) > if TAB_DEFAULT_STORES_DEFAULT not in skuobj.DefaultStore= Dict: > PcdDefaultStoreSet =3D set(defaultstorename for def= aultstorename > in skuobj.DefaultStoreDict) > mindefaultstorename =3D > DefaultStoreMgr.GetMin(PcdDefaultStoreSet) > - skuobj.DefaultStoreDict[TAB_DEFAULT_STORES_DEFAULT] = =3D > copy.deepcopy(skuobj.DefaultStoreDict[mindefaultstorename]) > + skuobj.DefaultStoreDict[TAB_DEFAULT_STORES_DEFAULT] = =3D > CopyDict(skuobj.DefaultStoreDict[mindefaultstorename]) > return Pcds >=20 > def RecoverCommandLinePcd(self): > def UpdateCommandLineValue(pcd): > if pcd.Type in > [self._PCD_TYPE_STRING_[MODEL_PCD_FIXED_AT_BUILD], > @@ -1501,11 +1501,11 @@ class DscBuildData(PlatformBuildClassObject): > PcdDefaultStoreSet =3D set(defaultstorename for def= aultstorename > in stru_pcd.SkuOverrideValues[nextskuid]) > mindefaultstorename =3D > DefaultStoreMgr.GetMin(PcdDefaultStoreSet) >=20 > for defaultstoreid in DefaultStores: > if defaultstoreid not in stru_pcd.SkuOverrideVal= ues[skuid]: > - stru_pcd.SkuOverrideValues[skuid][defaultsto= reid] =3D > copy.deepcopy(stru_pcd.SkuOverrideValues[nextskuid][mindefaultstorena > me]) > + stru_pcd.SkuOverrideValues[skuid][defaultsto= reid] =3D > CopyDict(stru_pcd.SkuOverrideValues[nextskuid][mindefaultstorename]) > stru_pcd.ValueChain.add((skuid, defaultstore= id)) > S_pcd_set =3D DscBuildData.OverrideByFdf(S_pcd_set,self.Workspac= eDir) > S_pcd_set =3D DscBuildData.OverrideByComm(S_pcd_set) > Str_Pcd_Values =3D self.GenerateByteArrayValue(S_pcd_set) > if Str_Pcd_Values: > @@ -2597,11 +2597,11 @@ class DscBuildData(PlatformBuildClassObject): > for skuid in PcdObj.SkuInfoList: > skuobj =3D PcdObj.SkuInfoList[skuid] > mindefaultstorename =3D > DefaultStoreObj.GetMin(set(defaultstorename for defaultstorename in > skuobj.DefaultStoreDict)) > for defaultstorename in DefaultStores: > if defaultstorename not in skuobj.DefaultStoreDi= ct: > - skuobj.DefaultStoreDict[defaultstorename] = =3D > copy.deepcopy(skuobj.DefaultStoreDict[mindefaultstorename]) > + skuobj.DefaultStoreDict[defaultstorename] = =3D > CopyDict(skuobj.DefaultStoreDict[mindefaultstorename]) > skuobj.HiiDefaultValue =3D > skuobj.DefaultStoreDict[mindefaultstorename] > for skuname, skuid in SkuIds.items(): > if skuname not in PcdObj.SkuInfoList: > nextskuid =3D self.SkuIdMgr.GetNextSkuId(skuname) > while nextskuid not in PcdObj.SkuInfoList: > -- > 2.19.1.windows.1 >=20 > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel