[PATCH v1 0/5] BaseTools: RegEx refactoring

[PATCH v1 0/5] BaseTools: RegEx refactoring

[Jaben Carsey]

move RegEx compile out of functions/loops
share RegEx when have identical data

Jaben Carsey (5):
  BaseTools: Autogen - move RegEx compile
  BaseTools: GenFds - move RegEx compile
  BaseTools: Add new RegEx pattern to GlobalData
  BaseTools: AutoGen - use the new shared RegEx
  BaseTools: remove redundant check

 BaseTools/Source/Python/AutoGen/AutoGen.py        | 22 +++++++++++++-------
 BaseTools/Source/Python/AutoGen/IdfClassObject.py |  5 +++--
 BaseTools/Source/Python/AutoGen/UniClassObject.py |  8 +++----
 BaseTools/Source/Python/Common/GlobalData.py      |  3 +++
 BaseTools/Source/Python/GenFds/FdfParser.py       |  5 ++---
 5 files changed, 27 insertions(+), 16 deletions(-)

-- 
2.16.2.windows.1

[PATCH v1 1/5] BaseTools: Autogen - move RegEx compile

[Jaben Carsey]

compile each RegEx once not in loops/functions

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
---
 BaseTools/Source/Python/AutoGen/AutoGen.py | 22 +++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
index 3384fdb70b7e..7dd7c35e6685 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -49,6 +49,16 @@ from GenVar import VariableMgr,var_info
 ## Regular expression for splitting Dependency Expression string into tokens
 gDepexTokenPattern = re.compile("(\(|\)|\w+| \S+\.inf)")
 
+## Regular expression for match: PCD(xxxx.yyy)
+gPCDAsGuidPattern = re.compile(r"^PCD\(.+\..+\)$")
+
+#
+# Regular expression for finding Include Directories, the difference between MSFT and INTEL/GCC/RVCT
+# is the former use /I , the Latter used -I to specify include directories
+#
+gBuildOptIncludePatternMsft = re.compile(r"(?:.*?)/I[ \t]*([^ ]*)", re.MULTILINE | re.DOTALL)
+gBuildOptIncludePatternOther = re.compile(r"(?:.*?)-I[ \t]*([^ ]*)", re.MULTILINE | re.DOTALL)
+
 #
 # Match name = variable
 #
@@ -818,13 +828,11 @@ class WorkspaceAutoGen(AutoGen):
                         InfFoundFlag = False
 
                 if FfsFile.NameGuid is not None:
-                    _CheckPCDAsGuidPattern = re.compile("^PCD\(.+\..+\)$")
-
                     #
                     # If the NameGuid reference a PCD name. 
                     # The style must match: PCD(xxxx.yyy)
                     #
-                    if _CheckPCDAsGuidPattern.match(FfsFile.NameGuid):
+                    if gPCDAsGuidPattern.match(FfsFile.NameGuid):
                         #
                         # Replace the PCD value.
                         #
@@ -3315,9 +3323,9 @@ class ModuleAutoGen(AutoGen):
             # is the former use /I , the Latter used -I to specify include directories
             #
             if self.PlatformInfo.ToolChainFamily in ('MSFT'):
-                gBuildOptIncludePattern = re.compile(r"(?:.*?)/I[ \t]*([^ ]*)", re.MULTILINE | re.DOTALL)
+                BuildOptIncludeRegEx = gBuildOptIncludePatternMsft
             elif self.PlatformInfo.ToolChainFamily in ('INTEL', 'GCC', 'RVCT'):
-                gBuildOptIncludePattern = re.compile(r"(?:.*?)-I[ \t]*([^ ]*)", re.MULTILINE | re.DOTALL)
+                BuildOptIncludeRegEx = gBuildOptIncludePatternOther
             else:
                 #
                 # New ToolChainFamily, don't known whether there is option to specify include directories
@@ -3334,13 +3342,13 @@ class ModuleAutoGen(AutoGen):
                     FlagOption = ''
                 
                 if self.PlatformInfo.ToolChainFamily != 'RVCT':
-                    IncPathList = [NormPath(Path, self.Macros) for Path in gBuildOptIncludePattern.findall(FlagOption)]
+                    IncPathList = [NormPath(Path, self.Macros) for Path in BuildOptIncludeRegEx.findall(FlagOption)]
                 else:
                     #
                     # RVCT may specify a list of directory seperated by commas
                     #
                     IncPathList = []
-                    for Path in gBuildOptIncludePattern.findall(FlagOption):
+                    for Path in BuildOptIncludeRegEx.findall(FlagOption):
                         PathList = GetSplitList(Path, TAB_COMMA_SPLIT)
                         IncPathList += [NormPath(PathEntry, self.Macros) for PathEntry in PathList]
 
-- 
2.16.2.windows.1

[PATCH v1 2/5] BaseTools: GenFds - move RegEx compile

[Jaben Carsey]

compile each RegEx once not in loops/functions

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
---
 BaseTools/Source/Python/GenFds/FdfParser.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/BaseTools/Source/Python/GenFds/FdfParser.py b/BaseTools/Source/Python/GenFds/FdfParser.py
index 9b7e898570a3..00e03446421d 100644
--- a/BaseTools/Source/Python/GenFds/FdfParser.py
+++ b/BaseTools/Source/Python/GenFds/FdfParser.py
@@ -85,6 +85,7 @@ RegionSizePattern = re.compile("\s*(?P<base>(?:0x|0X)?[a-fA-F0-9]+)\s*\|\s*(?P<s
 RegionSizeGuidPattern = re.compile("\s*(?P<base>\w+\.\w+)\s*\|\s*(?P<size>\w+\.\w+)\s*")
 RegionOffsetPcdPattern = re.compile("\s*(?P<base>\w+\.\w+)\s*$")
 ShortcutPcdPattern = re.compile("\s*\w+\s*=\s*(?P<value>(?:0x|0X)?[a-fA-F0-9]+)\s*\|\s*(?P<name>\w+\.\w+)\s*")
+BaseAddrValuePattern = re.compile('^0[xX][0-9a-fA-F]+')
 
 AllIncludeFileList = []
 
@@ -2211,9 +2212,7 @@ class FdfParser:
         if not self.__GetNextToken():
             raise Warning("expected FV base address value", self.FileName, self.CurrentLineNumber)
 
-        IsValidBaseAddrValue = re.compile('^0[x|X][0-9a-fA-F]+')
-
-        if not IsValidBaseAddrValue.match(self.__Token.upper()):
+        if not BaseAddrValuePattern.match(self.__Token.upper()):
             raise Warning("Unknown FV base address value '%s'" % self.__Token, self.FileName, self.CurrentLineNumber)
         Obj.FvBaseAddress = self.__Token
         return True  
-- 
2.16.2.windows.1

[PATCH v1 3/5] BaseTools: Add new RegEx pattern to GlobalData

[Jaben Carsey]

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
---
 BaseTools/Source/Python/Common/GlobalData.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Source/Python/Common/GlobalData.py
index fab67601a9e5..f58dc5a8dda2 100644
--- a/BaseTools/Source/Python/Common/GlobalData.py
+++ b/BaseTools/Source/Python/Common/GlobalData.py
@@ -63,6 +63,9 @@ gGuidPatternEnd = re.compile(r'{}$'.format(_GuidPattern))
 g4HexChar = re.compile(r'{}{{4}}'.format(_HexChar))
 gHexPattern = re.compile(r'0[xX]{}+'.format(_HexChar))
 
+## Regular expressions for string identifier checking
+gIdentifierPattern = re.compile('^[a-zA-Z][a-zA-Z0-9_]*$', re.UNICODE)
+
 #
 # A global variable for whether current build in AutoGen phase or not.
 #
-- 
2.16.2.windows.1

[PATCH v1 4/5] BaseTools: AutoGen - use the new shared RegEx

[Jaben Carsey]

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
---
 BaseTools/Source/Python/AutoGen/IdfClassObject.py | 3 ++-
 BaseTools/Source/Python/AutoGen/UniClassObject.py | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/IdfClassObject.py b/BaseTools/Source/Python/AutoGen/IdfClassObject.py
index cb72219b40d5..a028aceee927 100644
--- a/BaseTools/Source/Python/AutoGen/IdfClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/IdfClassObject.py
@@ -21,6 +21,7 @@ from Common.Misc import PathClass
 from Common.LongFilePathSupport import LongFilePath
 import re
 import os
+from Common.GlobalData import gIdentifierPattern
 
 IMAGE_TOKEN = re.compile('IMAGE_TOKEN *\(([A-Z0-9_]+) *\)', re.MULTILINE | re.UNICODE)
 
@@ -105,7 +106,7 @@ class IdfFileClassObject(object):
                     EdkLogger.error("Image Definition File Parser", PARSER_ERROR, 'The format is not match #image IMAGE_ID [TRANSPARENT] ImageFileName in Line %s of File %s.' % (LineNo, File.Path))
                 if Len == 4 and LineDetails[2] != 'TRANSPARENT':
                     EdkLogger.error("Image Definition File Parser", PARSER_ERROR, 'Please use the keyword "TRANSPARENT" to describe the transparency setting in Line %s of File %s.' % (LineNo, File.Path))
-                MatchString = re.match('^[a-zA-Z][a-zA-Z0-9_]*$', LineDetails[1], re.UNICODE)
+                MatchString = gIdentifierPattern.match(LineDetails[1])
                 if MatchString is None or MatchString.end(0) != len(LineDetails[1]):
                     EdkLogger.error('Image Definition  File Parser', FORMAT_INVALID, 'The Image token name %s defined in Idf file %s contains the invalid character.' % (LineDetails[1], File.Path))
                 if LineDetails[1] not in self.ImageIDList:
diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index 242402dfaeeb..8b0c563a8c88 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -351,7 +351,7 @@ class UniFileClassObject(object):
         Name = Item.split()[1]
         # Check the string name
         if Name != '':
-            MatchString = re.match('^[a-zA-Z][a-zA-Z0-9_]*$', Name, re.UNICODE)
+            MatchString = gIdentifierPattern.match(Name)
             if MatchString is None or MatchString.end(0) != len(Name):
                 EdkLogger.error('Unicode File Parser', FORMAT_INVALID, 'The string token name %s defined in UNI file %s contains the invalid character.' % (Name, self.File))
         LanguageList = Item.split(u'#language ')
@@ -521,7 +521,7 @@ class UniFileClassObject(object):
                 Language = GetLanguageCode(Language, self.IsCompatibleMode, self.File)
                 # Check the string name
                 if not self.IsCompatibleMode and Name != '':
-                    MatchString = re.match('^[a-zA-Z][a-zA-Z0-9_]*$', Name, re.UNICODE)
+                    MatchString = gIdentifierPattern.match(Name)
                     if MatchString is None or MatchString.end(0) != len(Name):
                         EdkLogger.error('Unicode File Parser', FORMAT_INVALID, 'The string token name %s defined in UNI file %s contains the invalid character.' % (Name, self.File))
                 self.AddStringToList(Name, Language, Value)
-- 
2.16.2.windows.1

[PATCH v1 5/5] BaseTools: remove redundant check

[Jaben Carsey]

The RegEx matches begining and end of the string, dont then check length.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Jaben Carsey <jaben.carsey@intel.com>
---
 BaseTools/Source/Python/AutoGen/IdfClassObject.py | 2 +-
 BaseTools/Source/Python/AutoGen/UniClassObject.py | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/IdfClassObject.py b/BaseTools/Source/Python/AutoGen/IdfClassObject.py
index a028aceee927..6953854a5247 100644
--- a/BaseTools/Source/Python/AutoGen/IdfClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/IdfClassObject.py
@@ -107,7 +107,7 @@ class IdfFileClassObject(object):
                 if Len == 4 and LineDetails[2] != 'TRANSPARENT':
                     EdkLogger.error("Image Definition File Parser", PARSER_ERROR, 'Please use the keyword "TRANSPARENT" to describe the transparency setting in Line %s of File %s.' % (LineNo, File.Path))
                 MatchString = gIdentifierPattern.match(LineDetails[1])
-                if MatchString is None or MatchString.end(0) != len(LineDetails[1]):
+                if MatchString is None:
                     EdkLogger.error('Image Definition  File Parser', FORMAT_INVALID, 'The Image token name %s defined in Idf file %s contains the invalid character.' % (LineDetails[1], File.Path))
                 if LineDetails[1] not in self.ImageIDList:
                     self.ImageIDList.append(LineDetails[1])
diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index 8b0c563a8c88..5b879d784d9c 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -352,7 +352,7 @@ class UniFileClassObject(object):
         # Check the string name
         if Name != '':
             MatchString = gIdentifierPattern.match(Name)
-            if MatchString is None or MatchString.end(0) != len(Name):
+            if MatchString is None:
                 EdkLogger.error('Unicode File Parser', FORMAT_INVALID, 'The string token name %s defined in UNI file %s contains the invalid character.' % (Name, self.File))
         LanguageList = Item.split(u'#language ')
         for IndexI in range(len(LanguageList)):
@@ -522,7 +522,7 @@ class UniFileClassObject(object):
                 # Check the string name
                 if not self.IsCompatibleMode and Name != '':
                     MatchString = gIdentifierPattern.match(Name)
-                    if MatchString is None or MatchString.end(0) != len(Name):
+                    if MatchString is None:
                         EdkLogger.error('Unicode File Parser', FORMAT_INVALID, 'The string token name %s defined in UNI file %s contains the invalid character.' % (Name, self.File))
                 self.AddStringToList(Name, Language, Value)
                 continue
-- 
2.16.2.windows.1

Re: [PATCH v1 0/5] BaseTools: RegEx refactoring

[Zhu, Yonghong]

The patches are good to me.

Reviewed-by: Yonghong Zhu <yonghong.zhu@intel.com> 

Best Regards,
Zhu Yonghong


-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Jaben Carsey
Sent: Wednesday, April 04, 2018 6:34 AM
To: edk2-devel@lists.01.org
Subject: [edk2] [PATCH v1 0/5] BaseTools: RegEx refactoring

move RegEx compile out of functions/loops share RegEx when have identical data

Jaben Carsey (5):
  BaseTools: Autogen - move RegEx compile
  BaseTools: GenFds - move RegEx compile
  BaseTools: Add new RegEx pattern to GlobalData
  BaseTools: AutoGen - use the new shared RegEx
  BaseTools: remove redundant check

 BaseTools/Source/Python/AutoGen/AutoGen.py        | 22 +++++++++++++-------
 BaseTools/Source/Python/AutoGen/IdfClassObject.py |  5 +++--  BaseTools/Source/Python/AutoGen/UniClassObject.py |  8 +++----
 BaseTools/Source/Python/Common/GlobalData.py      |  3 +++
 BaseTools/Source/Python/GenFds/FdfParser.py       |  5 ++---
 5 files changed, 27 insertions(+), 16 deletions(-)

--
2.16.2.windows.1

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel