From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.120, mailfrom: bob.c.feng@intel.com) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by groups.io with SMTP; Mon, 10 Jun 2019 19:54:35 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Jun 2019 19:54:34 -0700 X-ExtLoop1: 1 Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by fmsmga008.fm.intel.com with ESMTP; 10 Jun 2019 19:54:34 -0700 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 10 Jun 2019 19:54:34 -0700 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.10]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.6]) with mapi id 14.03.0415.000; Tue, 11 Jun 2019 10:54:32 +0800 From: "Bob Feng" To: "Fan, ZhijuX" , "devel@edk2.groups.io" CC: "Gao, Liming" Subject: Re: [PATCH V2] BaseTools:add UniTool.py to Edk2\BaseTools\Scripts Thread-Topic: [PATCH V2] BaseTools:add UniTool.py to Edk2\BaseTools\Scripts Thread-Index: AdUV872e9SFT1+KSSZeCoQej1V1piQKCuxYg Date: Tue, 11 Jun 2019 02:54:31 +0000 Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D1601354E4@SHSMSX101.ccr.corp.intel.com> References: In-Reply-To: Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: bob.c.feng@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Zhiju, For overall, the functions exit method are not consistent in this patch. Some function uses print(message ...) and return, some function uses exit(m= essage), some function uses except(message...) and some function uses retur= n(message...) Please well define the functions in this patch. And for specific,=20 + try: + outputFile =3D codecs.open(destFileName, 'w', Encoding) + print(destFileName + " did not exist. Creating new file.") + platformUQI=3D'' + except: + print("Error creating " + destFileName + ".") + return 1 The code just open a file but do not close it. Suggest using with statement= . There are still similar issues in this patch. + try: + Encoding =3D GetUniFileEncoding (destFileName) + with codecs.open(destFileName, 'r+', Encoding) as destFile: + destFileBuffer =3D destFile.read() + except IOError as e: + print("ERROR: " + e.args[1]) + raise RuntimeError + return The return after raise is not necessary. There are also other such issue in= this patch. Thanks, Bob -----Original Message----- From: Fan, ZhijuX=20 Sent: Wednesday, May 29, 2019 3:55 PM To: devel@edk2.groups.io Cc: Gao, Liming ; Feng, Bob C Subject: [PATCH V2] BaseTools:add UniTool.py to Edk2\BaseTools\Scripts BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D1855 UniTool is one python script to generate UQI (Universal Question Identifier) unicode string for HII question PROMPT string. UQI string can b= e used to identify each HII question. The scripts function will sync up UQI definitions with uni files based on v= fi/vfr/hfr/sd/sdi in the tree. This script can be run in both Py2 and Py3. Cc: Bob Feng Cc: Liming Gao Signed-off-by: Zhiju.Fan --- BaseTools/Scripts/UniTool.py | 485 +++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 485 insertions(+) create mode 100644 BaseTools/Scripts/UniTool.py diff --git a/BaseTools/Scripts/UniTool.py b/BaseTools/Scripts/UniTool.py ne= w file mode 100644 index 0000000000..2f76d305cd --- /dev/null +++ b/BaseTools/Scripts/UniTool.py @@ -0,0 +1,485 @@ +## @file +# generate UQI (Universal Question Identifier) unicode string for HII=20 +question PROMPT string. UQI string can be used to # identify each HII ques= tion. +# +# Copyright (c) 2019, Intel Corporation. All rights reserved. +# +# SPDX-License-Identifier: BSD-2-Clause-Patent # + +import re, sys, os, getopt, codecs, fnmatch + +# global variable declarations +questionError =3D False +uqiList =3D re.compile('^#string[ \t]+([A-Z_0-9]+)[ \t]+#language[=20 +\t]+uqi[ \t\r\n]+"(?:[x\S]{1,2})([0-9a-fA-F]{4,5})"',re.M).findall +allUqis =3D {} +stringDict =3D {} +GlobalVarId =3D {} +options =3D {} + +#********************************************************************** +# description: Prints help information +# +# arguments: none +# +# returns: none +# + +def usage () : + sys.exit("Syntax: %s [-b] [-u] [-l] [-x] [-h] [-d 'rootDirectory1']=20 +[-d 'rootDirectory2'] [-d 'rootDirectory3']... [-q e|w] \ 'rootDirectory0'= 'uqiFile'|'uqiFileDirectory' ['excludedDirectory1'] ['excludedDirectory2']= ['excludedDirectory3']...\n%s" % + (os.path.basename(sys.argv[0]), + """\nFunction will sync up UQI definitions with uni files based=20 +on vfi/vfr/hfr/sd/sdi in the tree.\n Required Arguments: + 'rootdirectory0' path to root directory + 'uqiFileDirectory' path to UQI file(UqiList.uni) + 'uqiFile' UQI file + +Options: + -h Show this help + -b Build option returns error if any new UQI needs as= signing + based on vfi/vfr/hfr/sd/sdi when no -u option is s= pecified + -u Create new UQIs that does not already exist in uqi= File for + any string requiring a UQI based on vfi/vfr/hfr/sd= /sdi + NOTE: 'uqiFile' cannot be readonly! + -l Language deletion option (keeps only English and u= qi) + moves all UQIs to 'uqiFile' + NOTE: Uni files cannot be readonly! + -x Exclude 'rootDirectory'/'excludedDirectory1' & + 'rootDirectory'/'excludedDirectory2'... from UQI l= ist build + NOTE: Cannot be the same as rootDirectory + -d Add multiple root directories to process + -q Print warning(w) or return error(e) if different H= II questions + are referring same string token + +Return error if any duplicated UQI string or value in UQI list or if no=20 +definition for any string referred by HII question when -b or -u is=20 +specified + +NOTE: Options must be specified before parameters +""")) + +#********************************************************************** +# description: Get uni file encoding +# +# arguments: filename - name of uni file +# +# returns: utf-8 or utf-16 +# +def GetUniFileEncoding(filename): + # + # Detect Byte Order Mark at beginning of file. Default to UTF-8 + # + Encoding =3D 'utf-8' + + # + # Read file + # + try: + with open(filename, mode=3D'rb') as UniFile: + FileIn =3D UniFile.read() + except: + return Encoding + + if (FileIn.startswith(codecs.BOM_UTF16_BE) or FileIn.startswith(codecs.B= OM_UTF16_LE)): + Encoding =3D 'utf-16' + + return Encoding + +# rewrite function os.path.walk +def Walk(top, func, arg): + try: + names =3D os.listdir(top) + except os.error: + return + func(arg, top, names) + for name in names: + name =3D os.path.join(top, name) + if os.path.isdir(name): + Walk(name, func, arg) + +#********************************************************************** +# description: Parses commandline arguments and options +# Calls function processUni to build dictionary of strings +# Calls other functions according to user specified options +# +# arguments: argv - contains all input from command line +# - must contain path to root directory +# - may contain options -h, -u, -l, -b or -x before path +# +# returns: none +# +def main(argv) : +##### Read input arguments and options + global allUqis, uqiList, questionError + try: + opts, args =3D getopt.getopt(argv[1:], "hulbxd:q:") # each letter is=20 +an optional argument + except getopt.GetoptError: + usage() + try: + dirNameList =3D [args[0]] + QuestionOption =3D None + for eachOpt in opts: + if eachOpt[0] =3D=3D '-d': + dirNameList.append(eachOpt[1]) + if eachOpt[0] =3D=3D '-q': + QuestionOption =3D eachOpt[1] + if (QuestionOption !=3D "e") and (QuestionOption !=3D "w"): + print("\nERROR: invalid option value for -q option\n") + raise Exception + destname =3D args[1] + if len(args) > 2: + exDirList =3D args[2:] + except: + usage() + + UpdateUQIs =3D False + LangOption =3D False + BuildOption =3D False + ExcludeOption =3D False + exPathList =3D [] + + for o,a in opts: + if o =3D=3D "-h": + usage() + if o =3D=3D "-b": + BuildOption =3D True + if o =3D=3D "-u": + BuildOption =3D True + UpdateUQIs =3D True + if o =3D=3D "-l": + LangOption =3D True + if o =3D=3D "-x": + ExcludeOption =3D True + try: + for eachExDir in exDirList: + for eachRootDir in dirNameList: + if eachExDir =3D=3D eachRootDir: + print("\nERROR: excludedDirectory is same as rootDirectory\n= ") + raise Exception + exPathList.append(eachRootDir + os.sep + eachExDir) + except: + usage() + + global options + options =3D {'destname':destname, 'dirNameList':dirNameList, 'exPathList= ':exPathList, 'BuildOption':BuildOption, 'UpdateUQIs':UpdateUQIs, + 'LangOption':LangOption, 'ExcludeOption':ExcludeOption,=20 + 'QuestionOption':QuestionOption} print("UQI file: %s" %destname) for=20 + eachDirName in dirNameList: + Walk(eachDirName, processUni, None) if questionError: + raise RuntimeError + return + if os.path.isdir(options['destname']): + destFileName =3D options['destname']+os.sep+'UqiList.uni' + else: + destFileName =3D options['destname'] + if os.path.exists(destFileName) and (destFileName not in list(allUqis.ke= ys())): + try: + Encoding =3D GetUniFileEncoding (destFileName) + with codecs.open(destFileName, 'r+', Encoding) as destFile: + destFileBuffer =3D destFile.read() + except IOError as e: + print("ERROR: " + e.args[1]) + raise RuntimeError + return + allUqis[destFileName]=3D uqiList(destFileBuffer) returnVal =3D 0 if= =20 + BuildOption: + returnVal =3D newUqi() + if (returnVal =3D=3D 1): + raise(RuntimeError, 'Please fix UQI ERROR(s) above before proceeding= .') + else: + print("No UQI issues detected\n") return + +#********************************************************************** +# description: newUqi collects a list of all currently used uqi values in = the tree +# Halt build if any duplicated string or value in UQI list. +# If -u option was specified, creates new UQIs that does not +# already exist in uqiFile for any string requiring a UQI. +# +# arguments: none +# +# returns: 0 on success +# 1 on error - this should cause the build to halt +# + +syntax =3D "S" +syntaxRE =3D re.compile('#string[ \t]+[A-Z_0-9]+[ \t]+#language[ \t]+uqi[= =20 +\t\r\n]+"([x\S]{1,2}).*',re.DOTALL).findall + +def newUqi(): + global options, GlobalVarId, allUqis, syntax, syntaxRE + uqiRange=3D[] + uqiStringList=3D[] + createUQI=3D[] + returnVal =3D 0 + BaseNumSpaces =3D 47 # Used to line up the UQI values in the resulting=20 +uqiFile + + # Look for duplication in the current UQIs and collect current range=20 + of UQIs for path in allUqis.keys(): + for uqiString in allUqis[path]: # path contains the path and filename = of each uni file + #Checks for duplicated strings in UQI list + for tempString in uqiStringList: + if tempString =3D=3D uqiString[0]: + print("ERROR: UQI string %s was assigned more than once and will= cause corruption!" %uqiString[0]) + print("Delete one occurrence of the string and rerun tool.") + returnVal =3D 1 #halt build + + uqiStringList.append(uqiString[0]) + + #Checks for duplicated UQI values in UQI list + if int(uqiString[1],16) in uqiRange: + print("ERROR: UQI value %04x was assigned more than once and will = cause corruption!" %int(uqiString[1],16)) + print("Delete one occurrance of the UQI and rerun tool to create a= lternate value.") + returnVal =3D 1 #halt build + uqiRange.append(int(uqiString[1],16)) + + for stringValue in GlobalVarId.keys(): + stringFound =3D False + for path in stringDict.keys(): + for uniString in stringDict[path]: # path contains the path and file= name of each uni file + if (stringValue =3D=3D uniString): + stringFound =3D True + break + if not stringFound: + print("ERROR: No definition for %s referred by HII question" %(strin= gValue)) + returnVal =3D 1 #halt build + + # Require a UQI for any string in vfr/vfi files for stringValue in=20 + GlobalVarId.keys(): + # Ignore strings defined as STRING_TOKEN(0) + if (stringValue !=3D "0"): + # Check if this string already exists in the UQI list + if (stringValue not in uqiStringList) and (stringValue not in create= UQI): + createUQI.append(stringValue) + if not options['UpdateUQIs']: + print("ERROR: No UQI for %s referred by HII question" %(stringVa= lue)) + returnVal =3D 1 # halt build after printing all error messages + + if (returnVal =3D=3D 1): + return returnVal + + # Update uqiFile with necessary UQIs + if options['UpdateUQIs'] and createUQI: + if os.path.isdir(options['destname']): + destFileName =3D options['destname']+os.sep+'UqiList.uni' + else: + destFileName =3D options['destname'] + try: + Encoding =3D GetUniFileEncoding (destFileName) + outputFile =3D codecs.open(destFileName, 'r+', Encoding) + platformUQI=3DoutputFile.read() + except IOError as e: + print("ERROR: " + e.args[1]) + if (e.args[0]=3D=3D2): + try: + outputFile =3D codecs.open(destFileName, 'w', Encoding) + print(destFileName + " did not exist. Creating new file.") + platformUQI=3D'' + except: + print("Error creating " + destFileName + ".") + return 1 + if (e.args[1]=3D=3D"Permission denied"): + print("\n%s is Readonly. You must uncheck the ReadOnly attibute t= o run the -u option.\n" %destFileName) + return 1 + + #Determines and sets the UQI number format + #TODO: there is probably a more elegant way to do this... + syntaxL =3D syntaxRE(platformUQI) + if len(syntaxL) !=3D 0: + syntax =3D syntaxL[0] + + # script is reading the file in and writing it back instead of appendi= ng because the codecs module + # automatically adds a BOM wherever you start writing. This caused bui= ld failure. + uqiRange.sort() + if (uqiRange =3D=3D []): + nextUqi =3D 0 + else: + nextUqi =3D uqiRange[len(uqiRange) - 1] + 1 + + for stringValue in createUQI: + print("%s will be assigned a new UQI value" %stringValue) + uqiRange.append(nextUqi) + # + # Lines up the UQI values in the resulting uqiFile + # + spaces =3D " "*(BaseNumSpaces - len(stringValue)) + platformUQI +=3D '#string %s%s #language uqi \"%s%04x\"\r\n' %(strin= gValue,spaces,syntax,nextUqi) + print("#string %s%s #language uqi \"%s%04X\"" %(stringValue, spaces= , syntax, nextUqi)) + nextUqi +=3D 1 + + outputFile.seek(0) + outputFile.write(platformUQI) + outputFile.close() + + return 0 + + +#********************************************************************** +# description: Parses each uni file to collect dictionary of strings +# Removes additional languages and overwrites current uni fil= es +# if -l option was specified +# +# arguments: path - directory location of file including file name +# filename - name of file to be modified +# +# returns: error string if failure occurred; +# none if completed sucessfully +# +# the following are global so that parsefile is quicker + +findUniString =3D re.compile('^#string[ \t]+([A-Z_0-9]+)(?:[=20 +\t\r\n]+#language[ \t]+[a-zA-Z-]{2,5}[ \t\r\n]+".*"[=20 +\t]*[\r]?[\n]?)*',re.M).findall + +otherLang =3D re.compile('^#string[ \t]+[A-Z_0-9]+(?:[ \t\r\n]+#language[= =20 +\t]+[a-zA-Z-]{2,5}[ \t\r\n]+".*"[ \t]*[\r]?[\n]?)*',re.M).findall=20 +eachLang =3D re.compile('[ \t\r\n]+#language[ \t]+([a-zA-Z-]{2,5})[=20 +\t\r\n]+".*"[ \t]*[\r]?[\n]?').findall + +uqiStrings =3D re.compile('^#string[ \t]+[A-Z_0-9]+[ \t]+#language[=20 +\t]+uqi[ \t\r\n]+".*"[ \t]*[\r]?[\n]?',re.M) + +def parsefile(path,filename): + global options, stringDict, allUqis, uqiList, findUniString,=20 +otherLang, eachLang, uqiStrings + + FullPath =3D path+os.sep+filename + + try: + UniEncoding =3D GetUniFileEncoding (FullPath) + with codecs.open(FullPath, 'r', UniEncoding) as UniFile: + databuffer =3D UniFile.read() + except: + return ("Error opening " + FullPath + " for reading.") writeFile =3D= =20 + False + + if os.path.isdir(options['destname']): + destFileName =3D options['destname']+os.sep+'UqiList.uni' + else: + destFileName =3D options['destname'] + + if options['LangOption']: + try: + UqiEncoding =3D GetUniFileEncoding (destFileName) + outputFile =3D codecs.open(destFileName, 'r+', UqiEncoding) + platformUQI=3DoutputFile.read() + except IOError as e: + print("ERROR: " + e.args[1]) + if (e.args[0]=3D=3D2): + try: + outputFile =3D codecs.open(destFileName, 'w', UqiEncoding) + print(destFileName + " did not exist. Creating new file.") + platformUQI=3D'' + except: + return ("Error creating " + destFileName + ".") + else: + return ("Error opening " + destFileName + " for appending.") + + if (filename!=3DdestFileName.split(os.sep)[-1]): + Uqis =3D re.findall(uqiStrings,databuffer) + if Uqis: + for uqi in Uqis: + platformUQI +=3D uqi + outputFile.seek(0) + outputFile.write(platformUQI) + outputFile.close() + databuffer =3D re.sub(uqiStrings, '', databuffer) + if Uqis: + writeFile =3D True + print("Deleted uqis from %s" %FullPath) + stringlist =3D otherLang(databuffer) + for stringfound in stringlist: + thisString =3D eachLang(stringfound) + for languageFound in thisString: + if ((languageFound !=3D 'en') and (languageFound !=3D 'en-US') a= nd (languageFound !=3D 'eng') and (languageFound !=3D 'uqi')): + databuffer =3D re.sub(re.escape(stringfound), '', databuffer) + writeFile =3D True + print("Deleted %s from %s" %(languageFound, FullPath)) if=20 + (filename!=3DdestFileName.split(os.sep)[-1]): + #adding strings to dictionary + stringDict[r'%s' %FullPath]=3D findUniString(databuffer) #adding=20 + UQIs to dictionary allUqis[r'%s' %FullPath]=3D uqiList(databuffer) + + if writeFile: + try: + with codecs.open(FullPath, 'w', UniEncoding) as UniFile: + UniFile.write(databuffer) + except: + return ("Error opening " + FullPath + " for writing.") return + +#********************************************************************** +# description: Searches tree for uni files +# Calls parsefile to collect dictionary of strings in each un= i file +# Calls searchVfiFile for each vfi or vfr file found +# +# arguments: argument list is built by os.path.walk function call +# arg - None +# dirname - directory location of files +# names - specific files to search in directory +# +# returns: none +# +def processUni(args,dirname,names) : + global options + #Remove excludedDirectory + if options['ExcludeOption']: + for eachExDir in options['exPathList']: + for dir in names: + if os.path.join(dirname,dir) =3D=3D eachExDir: + names.remove(dir) + + for entry in names: + FullPath =3D dirname+os.sep+entry + if fnmatch.fnmatch(FullPath,'*.uni'): + parsefile(dirname,entry) + if fnmatch.fnmatch(FullPath,'*.vf*'): + searchVfiFile(FullPath) + if fnmatch.fnmatch(FullPath,'*.sd'): + searchVfiFile(FullPath) + if fnmatch.fnmatch(FullPath,'*.sdi'): + searchVfiFile(FullPath) + if fnmatch.fnmatch(FullPath,'*.hfr'): + searchVfiFile(FullPath) + return + +#********************************************************************** +# description: Compose a dictionary of all strings that may need UQIs assi= gned +# to them and key is the string +# +# arguments: filename - name of file to search for strings +# +# returns: none +# + +#separate regexes for readability +stringGroups =3D re.compile('^[=20 +\t]*(?:oneof|numeric|checkbox|orderedlist)[=20 +\t]+varid.+?(?:endoneof|endnumeric|endcheckbox|endorderedlist);',=20 +re.DOTALL|re.M).findall stringVarIds =3D re.compile('[=20 +\t]*(?:oneof|numeric|checkbox|orderedlist)[ \t]+varid[ \t]*=3D[=20 +\t]*([a-zA-Z_0-9]+\.[a-zA-Z_0-9]+)').findall +stringTokens =3D re.compile('prompt[ \t]*=3D[ \t]*STRING_TOKEN[=20 +\t]*\(([a-zA-Z_0-9]+)\)').findall + +def searchVfiFile(filename) : + global options, GlobalVarId, stringGroups, stringVarIds,=20 +stringTokens, questionError + try: + with open(filename, 'r') as vfiFile: + databuffer=3DvfiFile.read() + + # Finds specified lines in file + vfiStringGroup =3D stringGroups(databuffer) + + # Searches for prompts within specified lines + for eachGroup in vfiStringGroup: + for eachString in stringTokens(eachGroup): + # Ignore strings defined as STRING_TOKEN(0), STRING_TOKEN(STR_EMPT= Y) or STRING_TOKEN(STR_NULL) + if (eachString !=3D "0") and (eachString !=3D "STR_EMPTY") and (ea= chString !=3D "STR_NULL"): + if eachString not in GlobalVarId: + GlobalVarId[eachString]=3DstringVarIds(eachGroup) + else: + if (GlobalVarId[eachString][0] !=3D stringVarIds(eachGroup)[0]= ): + if options['QuestionOption']: + if options['QuestionOption'] =3D=3D "e": + questionError =3D True + print("ERROR:"), + if options['QuestionOption'] =3D=3D "w": + print("WARNING:"), + print("%s referred by different HII questions(%s and=20 + %s)" %(eachString, GlobalVarId[eachString][0],=20 + stringVarIds(eachGroup)[0])) + except: + print("Error opening file at %s for reading." %filename) + +if __name__ =3D=3D '__main__' : + sys.exit(main(sys.argv)) -- 2.14.1.windows.1