From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.85.221.43; helo=mail-wr1-f43.google.com; envelope-from=philmd@redhat.com; receiver=edk2-devel@lists.01.org Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 8E12F2119A84A for ; Mon, 17 Dec 2018 02:05:25 -0800 (PST) Received: by mail-wr1-f43.google.com with SMTP id z5so11583833wrt.11 for ; Mon, 17 Dec 2018 02:05:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:openpgp:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=Cvf0G2I/C8RFZ9PTJIcw1tA+U1BslTvhITW0SQFlGF4=; b=fmRZqOVY+gIQhjxA/kxicp9PN1e09cwmgDhcX93ea58ldbr6DEJ0fr7W+RxBLweESD kTw67eR5Xut0REjjeNmN51QmJ0VokbYxxiVCl0GeVo1jjOwugAh3lrWhUEuf6qAGKl06 58hTbepAcA9FgyyCMLgODZWzZweAW2oRxXM7X8V0L/dRWi1Bk6thbxrfPhJP7866x5rZ bbbwLdCxp3gjzh0kqsgRZ1I5CFiTdWOJtB9vyKsQ7kack9NLiPwovYJnq4RGfTRJh4b7 nx5n5tH1UX37ShI8gr0g/x3Im3T1OpV3BqjwVgdjZFasKUkLPHAtmnUl5PzAs9YEYB5e xutQ== X-Gm-Message-State: AA+aEWYiXP9xkY0D8G2tzeg2kzfHh3ugSklx63cY1djk9TXgtWhmp9yY +0AFBQ3LcuyORtTJ+1gO3oidBQ== X-Google-Smtp-Source: AFSGD/WX8Ofxd3H4xUjCj87qOvb0KlilhPiKXjqGGMsZnfdDj6mcsgDv4/6u6mr8/t7CU0Lcai2r+g== X-Received: by 2002:adf:e08c:: with SMTP id c12mr9329216wri.199.1545041123973; Mon, 17 Dec 2018 02:05:23 -0800 (PST) Received: from [192.168.1.34] (172.red-88-21-202.staticip.rima-tde.net. [88.21.202.172]) by smtp.gmail.com with ESMTPSA id o81sm16436220wmd.10.2018.12.17.02.05.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 02:05:23 -0800 (PST) To: BobCF , edk2-devel@lists.01.org Cc: Liming Gao , Ard Biesheuvel References: <20181214161521.23236-1-bob.c.feng@intel.com> From: =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= Openpgp: id=89C1E78F601EE86C867495CBA2A3FD6EDEADC0DE; url=http://pgp.mit.edu/pks/lookup?op=get&search=0xA2A3FD6EDEADC0DE Message-ID: <91b38ee2-7680-9276-6904-ac3a1e6ebfdc@redhat.com> Date: Mon, 17 Dec 2018 11:05:22 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: <20181214161521.23236-1-bob.c.feng@intel.com> Subject: Re: [Patch V2] BaseTools: Fix PcdArray issue X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 10:05:27 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit On 12/14/18 5:15 PM, BobCF wrote: > From: "Feng, Bob C" > > https://bugzilla.tianocore.org/show_bug.cgi?id=1390 > > 1. support hex number for array index > 2. support Non-Dynamic Pcd for array data type > 3. support {} and {CODE()} for array data type > 4. Change GetStructurePcdMaxSize to be a static function since it need to > be called in another static function. And this function does not depend on > it's class instance. > 5. Add unittest for RemoveCComments function and > ArrayIndex regular expression. Thanks for this Bob! > > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Bob Feng > Cc: Liming Gao > Cc: Ard Biesheuvel > Cc: Philippe Mathieu-Daudé Reviewed-by: Philippe Mathieu-Daudé > --- > BaseTools/Source/Python/Common/Misc.py | 6 ++ > .../Python/Workspace/BuildClassObject.py | 3 +- > .../Source/Python/Workspace/DscBuildData.py | 59 ++++++++++++------- > BaseTools/Tests/TestRegularExpression.py | 54 +++++++++++++++++ > 4 files changed, 99 insertions(+), 23 deletions(-) > create mode 100644 BaseTools/Tests/TestRegularExpression.py > > diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py > index b063f064fb..ea09f85e70 100644 > --- a/BaseTools/Source/Python/Common/Misc.py > +++ b/BaseTools/Source/Python/Common/Misc.py > @@ -2144,10 +2144,16 @@ def CopyDict(ori_dict): > if isinstance(ori_dict[key],(dict,OrderedDict)): > new_dict[key] = CopyDict(ori_dict[key]) > else: > new_dict[key] = ori_dict[key] > return new_dict > + > +# > +# Remove the c/c++ comments: // and /* */ > +# > +def RemoveCComments(ctext): > + return re.sub('//.*?\n|/\*.*?\*/', '\n', ctext, flags=re.S) > ## > # > # This acts like the main() function for the script, unless it is 'import'ed into another > # script. > # > diff --git a/BaseTools/Source/Python/Workspace/BuildClassObject.py b/BaseTools/Source/Python/Workspace/BuildClassObject.py > index 008eee1a16..e9a1195fd2 100644 > --- a/BaseTools/Source/Python/Workspace/BuildClassObject.py > +++ b/BaseTools/Source/Python/Workspace/BuildClassObject.py > @@ -17,11 +17,11 @@ import collections > import re > from collections import OrderedDict > from Common.Misc import CopyDict > import copy > StructPattern = re.compile(r'[_a-zA-Z][0-9A-Za-z_\[\]]*$') > -ArrayIndex = re.compile("\[\s*\d{0,1}\s*\]") > +ArrayIndex = re.compile("\[\s*[0-9a-fA-FxX]*\s*\]") > ## PcdClassObject > # > # This Class is used for PcdObject > # > # @param object: Inherited from object class > @@ -82,10 +82,11 @@ class PcdClassObject(object): > dimension = ArrayIndex.findall(self._DatumType) > for item in dimension: > maxsize = item.lstrip("[").rstrip("]").strip() > if not maxsize: > maxsize = "-1" > + maxsize = str(int(maxsize,16)) if maxsize.startswith(("0x","0X")) else maxsize > self._Capacity.append(maxsize) > if hasattr(self, "SkuOverrideValues"): > for sku in self.SkuOverrideValues: > for defaultstore in self.SkuOverrideValues[sku]: > fields = self.SkuOverrideValues[sku][defaultstore] > diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/Source/Python/Workspace/DscBuildData.py > index b485c75a84..37fb8d56b6 100644 > --- a/BaseTools/Source/Python/Workspace/DscBuildData.py > +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py > @@ -31,11 +31,11 @@ from .MetaDataTable import * > from .MetaFileTable import * > from .MetaFileParser import * > > from .WorkspaceCommon import GetDeclaredPcd > from Common.Misc import AnalyzeDscPcd > -from Common.Misc import ProcessDuplicatedInf > +from Common.Misc import ProcessDuplicatedInf,RemoveCComments > import re > from Common.Parsing import IsValidWord > from Common.VariableAttributes import VariableAttributes > import Common.GlobalData as GlobalData > import subprocess > @@ -1573,11 +1573,11 @@ class DscBuildData(PlatformBuildClassObject): > mindefaultstorename = DefaultStoreObj.GetMin(PcdDefaultStoreSet) > str_pcd_obj.SkuInfoList[self.SkuIdMgr.SystemSkuId].HiiDefaultValue = str_pcd_obj.SkuInfoList[self.SkuIdMgr.SystemSkuId].DefaultStoreDict[mindefaultstorename] > > for str_pcd_obj in S_pcd_set.values(): > > - str_pcd_obj.MaxDatumSize = self.GetStructurePcdMaxSize(str_pcd_obj) > + str_pcd_obj.MaxDatumSize = DscBuildData.GetStructurePcdMaxSize(str_pcd_obj) > Pcds[str_pcd_obj.TokenCName, str_pcd_obj.TokenSpaceGuidCName] = str_pcd_obj > Pcds[str_pcd_obj.TokenCName, str_pcd_obj.TokenSpaceGuidCName].CustomAttribute['IsStru']=True > > for pcdkey in Pcds: > pcd = Pcds[pcdkey] > @@ -1687,13 +1687,14 @@ class DscBuildData(PlatformBuildClassObject): > if SkuName not in Pcds[PcdCName, TokenSpaceGuid].DscRawValue: > Pcds[PcdCName, TokenSpaceGuid].DscRawValue[SkuName] = {} > Pcds[PcdCName, TokenSpaceGuid].DscRawValue[SkuName][TAB_DEFAULT_STORES_DEFAULT] = Settings[0] > return Pcds > > - def GetStructurePcdMaxSize(self, str_pcd): > + @staticmethod > + def GetStructurePcdMaxSize(str_pcd): > pcd_default_value = str_pcd.DefaultValue > - sku_values = [skuobj.HiiDefaultValue if str_pcd.Type in [self._PCD_TYPE_STRING_[MODEL_PCD_DYNAMIC_HII], self._PCD_TYPE_STRING_[MODEL_PCD_DYNAMIC_EX_HII]] else skuobj.DefaultValue for skuobj in str_pcd.SkuInfoList.values()] > + sku_values = [skuobj.HiiDefaultValue if str_pcd.Type in [DscBuildData._PCD_TYPE_STRING_[MODEL_PCD_DYNAMIC_HII], DscBuildData._PCD_TYPE_STRING_[MODEL_PCD_DYNAMIC_EX_HII]] else skuobj.DefaultValue for skuobj in str_pcd.SkuInfoList.values()] > sku_values.append(pcd_default_value) > > def get_length(value): > Value = value.strip() > if len(value) > 1: > @@ -1701,11 +1702,14 @@ class DscBuildData(PlatformBuildClassObject): > return 16 > if Value.startswith('L"') and Value.endswith('"'): > return len(Value[2:-1]) > if Value[0] == '"' and Value[-1] == '"': > return len(Value) - 2 > - if Value[0] == '{' and Value[-1] == '}': > + if Value.strip().startswith("{CODE("): > + tmpValue = RemoveCComments(Value) > + return len(tmpValue.split(",")) > + if (Value[0] == '{' and Value[-1] == '}'): > return len(Value.split(",")) > if Value.startswith("L'") and Value.endswith("'") and len(list(Value[2:-1])) > 1: > return len(list(Value[2:-1])) > if Value[0] == "'" and Value[-1] == "'" and len(list(Value[1:-1])) > 1: > return len(Value) - 2 > @@ -1864,11 +1868,12 @@ class DscBuildData(PlatformBuildClassObject): > for dem in Pcd.Capacity: > if dem == '0': > r_datatype.append("[1]") > else: > r_datatype.append("[" + dem + "]") > - CApp = ' Size = sizeof(%s);\n' % ("".join(r_datatype)) > + sizebasevalue = "(%s / sizeof(%s) + 1)" % ((DscBuildData.GetStructurePcdMaxSize(Pcd), Pcd.BaseDatumType)) > + CApp = ' Size = sizeof(%s) > %s?sizeof(%s) : %s ;\n' % ( ("".join(r_datatype), sizebasevalue, "".join(r_datatype), sizebasevalue) ) > else: > CApp = ' Size = sizeof(%s);\n' % (Pcd.DatumType) > CApp = CApp + ' Cal_%s_%s_Size(&Size);\n' % (Pcd.TokenSpaceGuidCName, Pcd.TokenCName) > return CApp > > @@ -2246,11 +2251,11 @@ class DscBuildData(PlatformBuildClassObject): > > CApp = CApp + '\n' > > PcdDefaultValue = StringToArray(Pcd.DefaultValueFromDec.strip()) > > - InitByteValue += '%s.%s.%s.%s|%s|%s\n' % (SkuName, DefaultStoreName, Pcd.TokenSpaceGuidCName, Pcd.TokenCName, Pcd.BaseDatumType, PcdDefaultValue) > + InitByteValue += '%s.%s.%s.%s|%s|%s\n' % (SkuName, DefaultStoreName, Pcd.TokenSpaceGuidCName, Pcd.TokenCName, Pcd.DatumType, PcdDefaultValue) > > # > # Get current PCD value and size > # > CApp = CApp + ' OriginalPcd = PcdGetPtr (%s, %s, %s, %s, &OriginalSize);\n' % (SkuName, DefaultStoreName, Pcd.TokenSpaceGuidCName, Pcd.TokenCName) > @@ -2316,33 +2321,43 @@ class DscBuildData(PlatformBuildClassObject): > return CApp > if not Pcd.IsArray(): > return CApp > Demesion = "" > for d in Pcd.Capacity: > - if d == "0": > - Demesion += "[]" > - else: > - Demesion += "["+d+"]" > + Demesion += "[]" > > Value = Pcd.DefaultValueFromDec > if "{CODE(" in Pcd.DefaultValueFromDec: > realvalue = Pcd.DefaultValueFromDec.strip()[6:-2] # "{CODE(").rstrip(")}" > - CApp += "static %s %s_%s_INIT_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,Demesion,realvalue) > + else: > + realvalue = Pcd.DefaultValueFromDec.strip() > + CApp += "static %s %s_%s_INIT_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,Demesion,realvalue) > > - for skuname in Pcd.SkuInfoList: > - skuinfo = Pcd.SkuInfoList[skuname] > - if skuinfo.VariableName: > - for defaultstore in skuinfo.DefaultStoreDict: > - Value = skuinfo[defaultstore] > + if Pcd.Type in PCD_DYNAMIC_TYPE_SET | PCD_DYNAMIC_EX_TYPE_SET: > + for skuname in Pcd.SkuInfoList: > + skuinfo = Pcd.SkuInfoList[skuname] > + if skuinfo.VariableName: > + for defaultstore in skuinfo.DefaultStoreDict: > + Value = skuinfo[defaultstore] > + if "{CODE(" in Value: > + realvalue = Value.strip()[6:-2] # "{CODE(").rstrip(")}" > + else: > + realvalue = Value.strip() > + CApp += "static %s %s_%s_%s_%s_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,skuname,defaultstore,Demesion,realvalue) > + else: > + Value = skuinfo.DefaultValue > if "{CODE(" in Value: > realvalue = Value.strip()[6:-2] # "{CODE(").rstrip(")}" > - CApp += "static %s %s_%s_%s_%s_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,skuname,defaultstore,Demesion,realvalue) > - else: > - Value = skuinfo.DefaultValue > - if "{CODE(" in Value: > - realvalue = Value.strip()[6:-2] # "{CODE(").rstrip(")}" > + else: > + realvalue = Value.strip() > CApp += "static %s %s_%s_%s_%s_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,skuname,TAB_DEFAULT_STORES_DEFAULT,Demesion,realvalue) > + else: > + if "{CODE(" in Pcd.DefaultValue: > + realvalue = Pcd.DefaultValue.strip()[6:-2] # "{CODE(").rstrip(")}" > + else: > + realvalue = Pcd.DefaultValue.strip() > + CApp += "static %s %s_%s_DEFAULT_STANDARD_Value%s = %s;\n" % (Pcd.BaseDatumType,Pcd.TokenSpaceGuidCName,Pcd.TokenCName,Demesion,realvalue) > return CApp > def SkuOverrideValuesEmpty(self,OverrideValues): > if not OverrideValues: > return True > for key in OverrideValues: > diff --git a/BaseTools/Tests/TestRegularExpression.py b/BaseTools/Tests/TestRegularExpression.py > new file mode 100644 > index 0000000000..07143d6a76 > --- /dev/null > +++ b/BaseTools/Tests/TestRegularExpression.py > @@ -0,0 +1,54 @@ > +## @file > +# Routines for generating Pcd Database > +# > +# Copyright (c) 2018, Intel Corporation. All rights reserved.
> +# This program and the accompanying materials > +# are licensed and made available under the terms and conditions of the BSD License > +# which accompanies this distribution. The full text of the license may be found at > +# http://opensource.org/licenses/bsd-license.php > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, > +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +import unittest > +from Common.Misc import RemoveCComments > +from Workspace.BuildClassObject import ArrayIndex > + > +class TestRe(unittest.TestCase): > + def test_ccomments(self): > + TestStr1 = """ {0x01,0x02} """ > + self.assertEquals(TestStr1, RemoveCComments(TestStr1)) > + > + TestStr2 = """ L'TestString' """ > + self.assertEquals(TestStr2, RemoveCComments(TestStr2)) > + > + TestStr3 = """ 'TestString' """ > + self.assertEquals(TestStr3, RemoveCComments(TestStr3)) > + > + TestStr4 = """ > + {CODE({ > + {0x01, {0x02, 0x03, 0x04 }},// Data comment > + {0x01, {0x02, 0x03, 0x04 }},// Data comment > + }) > + } /* > + This is multiple line comments > + The seconde line comment > + */ > + // This is a comment > + """ > + Expect_TestStr4 = """{CODE({ > + {0x01, {0x02, 0x03, 0x04 }}, > + {0x01, {0x02, 0x03, 0x04 }}, > + }) > + }""" > + self.assertEquals(Expect_TestStr4, RemoveCComments(TestStr4).strip()) > + > + def Test_ArrayIndex(self): > + TestStr1 = """[1]""" > + self.assertEquals(['[1]'], ArrayIndex.findall(TestStr1)) > + > + TestStr2 = """[1][2][0x1][0x01][]""" > + self.assertEquals(['[1]','[2]','[0x1]','[0x01]','[]'], ArrayIndex.findall(TestStr2)) > + > +if __name__ == '__main__': > + unittest.main() >