From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: liming.gao@intel.com) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by groups.io with SMTP; Tue, 25 Jun 2019 17:52:01 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Jun 2019 17:52:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,418,1557212400"; d="scan'208";a="313256417" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by orsmga004.jf.intel.com with ESMTP; 25 Jun 2019 17:52:00 -0700 Received: from fmsmsx114.amr.corp.intel.com (10.18.116.8) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 25 Jun 2019 17:51:59 -0700 Received: from shsmsx154.ccr.corp.intel.com (10.239.6.54) by FMSMSX114.amr.corp.intel.com (10.18.116.8) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 25 Jun 2019 17:51:58 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.185]) by SHSMSX154.ccr.corp.intel.com ([169.254.7.156]) with mapi id 14.03.0439.000; Wed, 26 Jun 2019 08:51:56 +0800 From: "Liming Gao" To: "Fan, ZhijuX" , "devel@edk2.groups.io" CC: "Feng, Bob C" , Ard Biesheuvel , Leif Lindholm , "Kinney, Michael D" Subject: Re: [edk2-platform patch V3] Platform/Intel:Add UniTool into edk2-platforms/Platform/Intel/Tools Thread-Topic: [edk2-platform patch V3] Platform/Intel:Add UniTool into edk2-platforms/Platform/Intel/Tools Thread-Index: AdUrRbnIOdF+gtfTSqqvZuUVV7l2JQAcWl9w Date: Wed, 26 Jun 2019 00:51:55 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E48D4DF@SHSMSX104.ccr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: liming.gao@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Zhiju: Please verify this script for edk2-platforms\Platform\Intel\Vlv2TbltDevic= ePkg\PlatformSetupDxe directory with -b and -u option. Seemly, the generate= d uni file is wrong.=20 And, please verify this tool -h option, its help message is not same to t= he one in script.=20 Last on readme.txt, please remove the section of update the UniTool.exe u= tility. Now, we suggest to directly use UniTool.py.=20 Thanks Liming >-----Original Message----- >From: Fan, ZhijuX >Sent: Tuesday, June 25, 2019 7:04 PM >To: devel@edk2.groups.io >Cc: Gao, Liming ; Feng, Bob C >; Ard Biesheuvel ; Leif >Lindholm ; Kinney, Michael D > >Subject: [edk2-platform patch V3] Platform/Intel:Add UniTool into edk2- >platforms/Platform/Intel/Tools > >BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D1855 > >UniTool is one python script to generate UQI (Universal Question >Identifier) unicode string for HII question PROMPT string. UQI >string can be used to identify each HII question. >The scripts function will sync up UQI definitions with uni files >based on vfi/vfr/hfr/sd/sdi in the tree. > >This script can be run in both Py2 and Py3. > >Cc: Liming Gao >Cc: Bob Feng >Cc: Ard Biesheuvel >Cc: Leif Lindholm >Cc: Michael D Kinney >Signed-off-by: Zhiju.Fan >--- >Add README.txt to scripts > > Platform/Intel/Tools/UniTool/README.txt | 46 +++ > Platform/Intel/Tools/UniTool/UniTool.py | 499 >++++++++++++++++++++++++++++++++ > 2 files changed, 545 insertions(+) > create mode 100644 Platform/Intel/Tools/UniTool/README.txt > create mode 100644 Platform/Intel/Tools/UniTool/UniTool.py > >diff --git a/Platform/Intel/Tools/UniTool/README.txt >b/Platform/Intel/Tools/UniTool/README.txt >new file mode 100644 >index 0000000000..3e59a6c75c >--- /dev/null >+++ b/Platform/Intel/Tools/UniTool/README.txt >@@ -0,0 +1,46 @@ >+To update the UniTool.exe utility >+----------------- >+- Must have python and cx_freeze installed with the default location. >+- After modify the UniTool.py script as desired, launch the runit.cmd scr= ipt >+- New version generated in UniTool.exe >+ >+How to use UniTool >+----------------- >+The usage of the tool is: >+UniTool.py [-b] [-u] [-l] [-x] [-h] [-d 'rootDirectory1'] [-d 'rootDirect= ory2'] [-d >'rootDirectory3']... [-q e|w] >+ 'rootDirectory0' 'uqiFile'|'uqiFileDirectory' ['excludedDirect= ory1'] >['excludedDirectory2'] ['excludedDirectory3']... >+ >+Function will sync up UQI definitions with uni files based on vfi/vfr/hfr= /sd/sdi >in the tree. >+ >+Required Arguments: >+ 'rootdirectory0' path to root directory >+ 'uqiFileDirectory' path to UQI file(UqiList.uni) >+ 'uqiFile' UQI file >+ >+Options: >+ -h Show this help >+ -b Build option returns error if any new UQI needs a= ssigning >+ based on vfi/vfr/hfr/sd/sdi when no -u option is = specified >+ -u Create new UQIs that does not already exist in uq= iFile for >+ any string requiring a UQI based on vfi/vfr/hfr/s= d/sdi >+ NOTE: 'uqiFile' cannot be readonly! >+ -l Language deletion option (keeps only English and = uqi) >+ moves all UQIs to 'uqiFile' >+ NOTE: Uni files cannot be readonly! >+ -x Exclude 'rootDirectory'/'excludedDirectory1' & >+ 'rootDirectory'/'excludedDirectory2'... from UQI = list build >+ NOTE: Cannot be the same as rootDirectory >+ -d Add multiple root directories to process >+ -q Print warning(w) or return error(e) if different = HII questions >+ are referring same string token >+ >+Return error if any duplicated UQI string or value in UQI list or if no d= efinition >+for any string referred by HII question when -b or -u is specified >+ >+NOTE: Options must be specified before parameters >+ >+Notice >+----------------- >+- "S0000" style will be used if uqiFile needs to be new created. >+ Use the same uqi style if uqiFile is existed. For example, >+ if the exist UqiFile use "\x0000" style, "\x0000" will be used. >diff --git a/Platform/Intel/Tools/UniTool/UniTool.py >b/Platform/Intel/Tools/UniTool/UniTool.py >new file mode 100644 >index 0000000000..df2b1036ce >--- /dev/null >+++ b/Platform/Intel/Tools/UniTool/UniTool.py >@@ -0,0 +1,499 @@ >+## @file >+# generate UQI (Universal Question Identifier) unicode string for HII >question PROMPT string. UQI string can be used to >+# identify each HII question. >+# >+# Copyright (c) 2019, Intel Corporation. All rights reserved. >+# >+# SPDX-License-Identifier: BSD-2-Clause-Patent >+# >+ >+import re >+import sys >+import os >+import getopt >+import codecs >+import fnmatch >+import logging >+import argparse >+ >+# global variable declarations >+QuestionError =3D False >+UqiList =3D re.compile('^#string[ \t]+([A-Z_0- >9]+)[ \t]+#language[ \t]+uqi[ \t\r\n]+"(?:[x\S]{1,2})([0-9a-fA-F]{4,5})"', >+ re.M).findall >+AllUqis =3D {} >+StringDict =3D {} >+GlobalVarId =3D {} >+Options =3D {} >+ >+# Version message >+__prog__ =3D 'UniTool' >+__description__ =3D 'generate UQI unicode string for HII question PROMPT >string.' >+__copyright__ =3D 'Copyright (c) 2019, Intel Corporation. All rights >reserved.
' >+__version__ =3D '%s Version %s' % (__prog__, '0.1 ') >+_Usage =3D "Syntax: %s [-b] [-u] [-l] [-x] [-h] [-d 'rootDirectory1'] [-= d >'rootDirectory2'] [-d 'rootDirectory3']... \n[-q e|w]" \ >+ "'rootDirectory0' 'uqiFile'|'uqiFileDirectory' ['excludedDirecto= ry1'] >['excludedDirectory2'] ['excludedDirectory3']...\n" \ >+ % (os.path.basename(sys.argv[0])) >+_Epilog =3D """\n\nFunction will sync up UQI definitions with uni files b= ased on >vfi/vfr/hfr/sd/sdi in the tree.\n >+ Required Arguments: >+ 'rootdirectory0' path to root directory >+ 'uqiFileDirectory' path to UQI file(UqiList.uni) >+ 'uqiFile' UQI file >+ >+ Return error if any duplicated UQI string or value in UQI list o= r if no >definition >+ for any string referred by HII question when -b or -u is specifi= ed >+ >+ NOTE: Options must be specified before parameters >+ """ >+ >+# >*********************************************************** >*********** >+# description: Get uni file encoding >+# >+# arguments: Filename - name of uni file >+# >+# returns: utf-8 or utf-16 >+# >+def GetUniFileEncoding(Filename): >+ # >+ # Detect Byte Order Mark at beginning of file. Default to UTF-8 >+ # >+ Encoding =3D 'utf-8' >+ >+ # >+ # Read file >+ # >+ try: >+ with open(Filename, mode=3D'rb') as UniFile: >+ FileIn =3D UniFile.read() >+ except: >+ return Encoding >+ >+ if (FileIn.startswith(codecs.BOM_UTF16_BE) or >FileIn.startswith(codecs.BOM_UTF16_LE)): >+ Encoding =3D 'utf-16' >+ >+ return Encoding >+ >+ >+# rewrite function os.path.walk >+def Walk(Top, Func, Arg): >+ try: >+ Names =3D os.listdir(Top) >+ except os.error: >+ return >+ Func(Arg, Top, Names) >+ for Name in Names: >+ Name =3D os.path.join(Top, Name) >+ if os.path.isdir(Name): >+ Walk(Name, Func, Arg) >+ >+ >+# >*********************************************************** >*********** >+# description: Parses commandline arguments and options >+# Calls function processUni to build dictionary of strings >+# Calls other functions according to user specified options >+# >+# arguments: argv - contains all input from command line >+# - must contain path to root directory >+# - may contain options -h, -u, -l, -b or -x before pat= h >+# >+# returns: none >+# >+def main(): >+ ##### Read input arguments and options >+ global AllUqis, UqiList, QuestionError >+ parser =3D argparse.ArgumentParser(prog=3D__prog__, >+ description=3D__description__ + __co= pyright__, >+ usage=3D_Usage, >+ epilog=3D_Epilog, >+ conflict_handler=3D'resolve') >+ parser.add_argument('Path', nargs=3D'+', >+ help=3D'the path for files to be converted.It cou= ld be directory or >file path.') >+ parser.add_argument('-v', '--version', action=3D'version', >version=3D__version__, >+ help=3D"show program's version number and exit") >+ parser.add_argument('-b', '--build', action=3D'store_true', dest=3D'B= uildOption', >+ help=3D"Build option returns error if any new UQI= needs assigning >" \ >+ "based on vfi/vfr/hfr/sd/sdi when no -u opti= on is specified") >+ parser.add_argument('-u', '--updata', action=3D'store_true', >dest=3D'UpdateUQIs', >+ help=3D"Create new UQIs that does not already exi= st in uqiFile for" >\ >+ "any string requiring a UQI based on vfi/vfr= /hfr/sd/sdi" \ >+ "NOTE: 'uqiFile' cannot be readonly!") >+ parser.add_argument('-l', '--lang', action=3D'store_true', dest=3D'La= ngOption', >+ help=3D"Language deletion option (keeps only Engl= ish and uqi)" \ >+ "moves all UQIs to 'uqiFile', NOTE: Uni file= s cannot be >readonly!") >+ parser.add_argument('-x', '--exclude', action=3D'store_true', >dest=3D'ExcludeOption', >+ help=3D"Exclude 'rootDirectory'/'excludedDirector= y1' &" \ >+ "'rootDirectory'/'excludedDirectory2'... fro= m UQI list build") >+ parser.add_argument('-d', '--dir', action=3D'append', metavar=3D'FILE= DIR', >dest=3D'DirName', >+ help=3D"Add multiple root directories to process"= ) >+ parser.add_argument('-q', '--question', dest=3D'Question', choices=3D= ['w', 'e'], >+ help=3D"Print warning(w) or return error(e) if di= fferent HII >questions" \ >+ "are referring same string token") >+ Opts =3D parser.parse_args() >+ Destname =3D '' >+ DirNameList =3D [] >+ ExDirList =3D [] >+ if Opts.Path: >+ DirNameList.append(Opts.Path[0]) >+ Destname =3D Opts.Path[1] >+ ExDirList =3D Opts.Path[2:] >+ if Opts.DirName: >+ DirNameList.extend(Opts.DirName) >+ QuestionOption =3D Opts.Question >+ ExcludeOption =3D Opts.ExcludeOption >+ BuildOption =3D Opts.BuildOption >+ UpdateUQIs =3D Opts.UpdateUQIs >+ LangOption =3D Opts.LangOption >+ ExPathList =3D [] >+ >+ if ExDirList: >+ try: >+ for EachExDir in ExDirList: >+ for EachRootDir in DirNameList: >+ if EachExDir =3D=3D EachRootDir: >+ print("\nERROR: excludedDirectory is same as root= Directory\n") >+ return >+ ExPathList.append(EachRootDir + os.sep + EachExDir) >+ except: >+ print(_Usage) >+ return >+ >+ global Options >+ Options =3D {'Destname': Destname, 'DirNameList': DirNameList, 'ExPat= hList': >ExPathList, 'BuildOption': BuildOption, >+ 'UpdateUQIs': UpdateUQIs, 'LangOption': LangOption, >'ExcludeOption': ExcludeOption, >+ 'QuestionOption': QuestionOption} >+ print("UQI file: %s" % Destname) >+ for EachDirName in DirNameList: >+ Walk(EachDirName, processUni, None) >+ if QuestionError: >+ return >+ if os.path.isdir(Options['Destname']): >+ DestFileName =3D Options['Destname'] + os.sep + 'UqiList.uni' >+ else: >+ DestFileName =3D Options['Destname'] >+ if os.path.exists(DestFileName) and (DestFileName not in >list(AllUqis.keys())): >+ try: >+ Encoding =3D GetUniFileEncoding(DestFileName) >+ with codecs.open(DestFileName, 'r+', Encoding) as destFile: >+ DestFileBuffer =3D destFile.read() >+ except IOError as e: >+ print("ERROR: " + e.args[1]) >+ return >+ AllUqis[DestFileName] =3D UqiList(DestFileBuffer) >+ if BuildOption: >+ ReturnVal =3D newUqi() >+ if (ReturnVal =3D=3D 1): >+ print('Please fix UQI ERROR(s) above before proceeding.') >+ else: >+ print("No UQI issues detected\n") >+ return >+ >+ >+# >*********************************************************** >*********** >+# description: newUqi collects a list of all currently used uqi values in= the tree >+# Halt build if any duplicated string or value in UQI list. >+# If -u option was specified, creates new UQIs that does not >+# already exist in uqiFile for any string requiring a UQI. >+# >+# arguments: none >+# >+# returns: 0 on success >+# 1 on error - this should cause the build to halt >+# >+ >+Syntax =3D "S" >+SyntaxRE =3D re.compile('#string[ \t]+[A-Z_0- >9]+[ \t]+#language[ \t]+uqi[ \t\r\n]+"([x\S]{1,2}).*', re.DOTALL).findall >+ >+ >+def newUqi(): >+ global Options, GlobalVarId, AllUqis, Syntax, SyntaxRE >+ UqiRange =3D [] >+ UqiStringList =3D [] >+ CreateUQI =3D [] >+ ReturnVal =3D 0 >+ BaseNumSpaces =3D 47 # Used to line up the UQI values in the resulti= ng >uqiFile >+ >+ # Look for duplication in the current UQIs and collect current range = of UQIs >+ for path in AllUqis.keys(): >+ for UqiString in AllUqis[path]: # path contains the path and Fil= ename of >each uni file >+ # Checks for duplicated strings in UQI list >+ for TempString in UqiStringList: >+ if TempString =3D=3D UqiString[0]: >+ print("ERROR: UQI string %s was assigned more than on= ce and will >cause corruption!" % UqiString[0]) >+ print("Delete one occurrence of the string and rerun = tool.") >+ ReturnVal =3D 1 # halt build >+ >+ UqiStringList.append(UqiString[0]) >+ >+ # Checks for duplicated UQI values in UQI list >+ if int(UqiString[1], 16) in UqiRange: >+ print("ERROR: UQI value %04x was assigned more than once = and will >cause corruption!" % int(UqiString[1], >+ = 16)) >+ print("Delete one occurrance of the UQI and rerun tool to= create >alternate value.") >+ ReturnVal =3D 1 # halt build >+ UqiRange.append(int(UqiString[1], 16)) >+ >+ for StringValue in GlobalVarId.keys(): >+ StringFound =3D False >+ for path in StringDict.keys(): >+ for UniString in StringDict[path]: # path contains the path = and >Filename of each uni file >+ if (StringValue =3D=3D UniString): >+ StringFound =3D True >+ break >+ if not StringFound: >+ print("ERROR: No definition for %s referred by HII question" = % >(StringValue)) >+ ReturnVal =3D 1 # halt build >+ >+ # Require a UQI for any string in vfr/vfi files >+ for StringValue in GlobalVarId.keys(): >+ # Ignore strings defined as STRING_TOKEN(0) >+ if (StringValue !=3D "0"): >+ # Check if this string already exists in the UQI list >+ if (StringValue not in UqiStringList) and (StringValue not in= CreateUQI): >+ CreateUQI.append(StringValue) >+ if not Options['UpdateUQIs']: >+ print("ERROR: No UQI for %s referred by HII question"= % >(StringValue)) >+ ReturnVal =3D 1 # halt build after printing all erro= r messages >+ >+ if (ReturnVal =3D=3D 1): >+ return ReturnVal >+ >+ # Update uqiFile with necessary UQIs >+ if Options['UpdateUQIs'] and CreateUQI: >+ if os.path.isdir(Options['Destname']): >+ DestFileName =3D Options['Destname'] + os.sep + 'UqiList.uni' >+ else: >+ DestFileName =3D Options['Destname'] >+ try: >+ Encoding =3D GetUniFileEncoding(DestFileName) >+ with codecs.open(DestFileName, 'r+', Encoding) as OutputFile: >+ PlatformUQI =3D OutputFile.read() >+ except IOError as e: >+ print("ERROR: " + e.args[1]) >+ if (e.args[0] =3D=3D 2): >+ try: >+ with codecs.open(DestFileName, 'w', Encoding) as Outp= utFile: >+ print(DestFileName + " did not exist. Creating n= ew file.") >+ PlatformUQI =3D '' >+ except: >+ print("Error creating " + DestFileName + ".") >+ return 1 >+ if (e.args[1] =3D=3D "Permission denied"): >+ print( >+ "\n%s is Readonly. You must uncheck the ReadOnly att= ibute to run >the -u option.\n" % DestFileName) >+ return 1 >+ >+ # Determines and sets the UQI number format >+ # TODO: there is probably a more elegant way to do this... >+ SyntaxL =3D SyntaxRE(PlatformUQI) >+ if len(SyntaxL) !=3D 0: >+ Syntax =3D SyntaxL[0] >+ >+ # script is reading the file in and writing it back instead of ap= pending >because the codecs module >+ # automatically adds a BOM wherever you start writing. This cause= d build >failure. >+ UqiRange.sort() >+ if (UqiRange =3D=3D []): >+ NextUqi =3D 0 >+ else: >+ NextUqi =3D UqiRange[len(UqiRange) - 1] + 1 >+ >+ for StringValue in CreateUQI: >+ print("%s will be assigned a new UQI value" % StringValue) >+ UqiRange.append(NextUqi) >+ # >+ # Lines up the UQI values in the resulting uqiFile >+ # >+ Spaces =3D " " * (BaseNumSpaces - len(StringValue)) >+ PlatformUQI +=3D '#string %s%s #language uqi \"%s%04x\"\r\n' = % >(StringValue, Spaces, Syntax, NextUqi) >+ print("#string %s%s #language uqi \"%s%04X\"" % (StringValue= , >Spaces, Syntax, NextUqi)) >+ NextUqi +=3D 1 >+ >+ with codecs.open(DestFileName, 'r+', Encoding) as OutputFile: >+ OutputFile.seek(0) >+ OutputFile.write(PlatformUQI) >+ >+ return 0 >+ >+ >+# >*********************************************************** >*********** >+# description: Parses each uni file to collect dictionary of strings >+# Removes additional languages and overwrites current uni fi= les >+# if -l option was specified >+# >+# arguments: path - directory location of file including file name >+# Filename - name of file to be modified >+# >+# returns: error string if failure occurred; >+# none if completed sucessfully >+# >+# the following are global so that parsefile is quicker >+ >+FindUniString =3D re.compile( >+ '^#string[ \t]+([A-Z_0-9]+)(?:[ \t\r\n]+#language[ \t]+[a-zA-Z- >]{2,5}[ \t\r\n]+".*"[ \t]*[\r]?[\n]?)*', >+ re.M).findall >+ >+OtherLang =3D re.compile( >+ '^#string[ \t]+[A-Z_0-9]+(?:[ \t\r\n]+#language[ \t]+[a-zA-Z- >]{2,5}[ \t\r\n]+".*"[ \t]*[\r]?[\n]?)*', re.M).findall >+EachLang =3D re.compile('[ \t\r\n]+#language[ \t]+([a-zA-Z- >]{2,5})[ \t\r\n]+".*"[ \t]*[\r]?[\n]?').findall >+ >+UqiStrings =3D re.compile('^#string[ \t]+[A-Z_0- >9]+[ \t]+#language[ \t]+uqi[ \t\r\n]+".*"[ \t]*[\r]?[\n]?', re.M) >+ >+ >+def parsefile(path, Filename): >+ global Options, StringDict, AllUqis, UqiList, FindUniString, OtherLan= g, >EachLang, UqiStrings >+ >+ FullPath =3D path + os.sep + Filename >+ >+ try: >+ UniEncoding =3D GetUniFileEncoding(FullPath) >+ with codecs.open(FullPath, 'r', UniEncoding) as UniFile: >+ Databuffer =3D UniFile.read() >+ except: >+ print("Error opening " + FullPath + " for reading.") >+ return >+ WriteFile =3D False >+ >+ if os.path.isdir(Options['Destname']): >+ DestFileName =3D Options['Destname'] + os.sep + 'UqiList.uni' >+ else: >+ DestFileName =3D Options['Destname'] >+ >+ if Options['LangOption']: >+ try: >+ UqiEncoding =3D GetUniFileEncoding(DestFileName) >+ with codecs.open(DestFileName, 'r+', UqiEncoding) as OutputFi= le: >+ PlatformUQI =3D OutputFile.read() >+ except IOError as e: >+ print("ERROR: " + e.args[1]) >+ if (e.args[0] =3D=3D 2): >+ try: >+ with codecs.open(DestFileName, 'w', UqiEncoding) as O= utputFile: >+ print(DestFileName + " did not exist. Creating n= ew file.") >+ PlatformUQI =3D '' >+ except: >+ print("Error creating " + DestFileName + ".") >+ return >+ else: >+ print("Error opening " + DestFileName + " for appending."= ) >+ return >+ >+ if (Filename !=3D DestFileName.split(os.sep)[-1]): >+ Uqis =3D re.findall(UqiStrings, Databuffer) >+ if Uqis: >+ for Uqi in Uqis: >+ PlatformUQI +=3D Uqi >+ with codecs.open(DestFileName, 'r+', UqiEncoding) as Outp= utFile: >+ OutputFile.seek(0) >+ OutputFile.write(PlatformUQI) >+ Databuffer =3D re.sub(UqiStrings, '', Databuffer) >+ if Uqis: >+ WriteFile =3D True >+ print("Deleted uqis from %s" % FullPath) >+ stringlist =3D OtherLang(Databuffer) >+ for stringfound in stringlist: >+ ThisString =3D EachLang(stringfound) >+ for LanguageFound in ThisString: >+ if ((LanguageFound !=3D 'en') and (LanguageFound !=3D= 'en-US') and >(LanguageFound !=3D 'eng') and ( >+ LanguageFound !=3D 'uqi')): >+ Databuffer =3D re.sub(re.escape(stringfound), '',= Databuffer) >+ WriteFile =3D True >+ print("Deleted %s from %s" % (LanguageFound, Full= Path)) >+ if (Filename !=3D DestFileName.split(os.sep)[-1]): >+ # adding strings to dictionary >+ StringDict[r'%s' % FullPath] =3D FindUniString(Databuffer) >+ # adding UQIs to dictionary >+ AllUqis[r'%s' % FullPath] =3D UqiList(Databuffer) >+ >+ if WriteFile: >+ try: >+ with codecs.open(FullPath, 'w', UniEncoding) as UniFile: >+ UniFile.write(Databuffer) >+ except: >+ print("Error opening " + FullPath + " for writing.") >+ return >+ >+ >+# >*********************************************************** >*********** >+# description: Searches tree for uni files >+# Calls parsefile to collect dictionary of strings in each u= ni file >+# Calls searchVfiFile for each vfi or vfr file found >+# >+# arguments: argument list is built by os.path.walk function call >+# arg - None >+# dirname - directory location of files >+# names - specific files to search in directory >+# >+# returns: none >+# >+def processUni(args, dirname, names): >+ global Options >+ # Remove excludedDirectory >+ if Options['ExcludeOption']: >+ for EachExDir in Options['ExPathList']: >+ for dir in names: >+ if os.path.join(dirname, dir) =3D=3D EachExDir: >+ names.remove(dir) >+ >+ for entry in names: >+ FullPath =3D dirname + os.sep + entry >+ if fnmatch.fnmatch(FullPath, '*.uni'): >+ parsefile(dirname, entry) >+ if fnmatch.fnmatch(FullPath, '*.vf*'): >+ searchVfiFile(FullPath) >+ if fnmatch.fnmatch(FullPath, '*.sd'): >+ searchVfiFile(FullPath) >+ if fnmatch.fnmatch(FullPath, '*.sdi'): >+ searchVfiFile(FullPath) >+ if fnmatch.fnmatch(FullPath, '*.hfr'): >+ searchVfiFile(FullPath) >+ return >+ >+ >+# >*********************************************************** >*********** >+# description: Compose a dictionary of all strings that may need UQIs >assigned >+# to them and key is the string >+# >+# arguments: Filename - name of file to search for strings >+# >+# returns: none >+# >+ >+# separate regexes for readability >+StringGroups =3D re.compile( >+ >'^[ \t]*(?:oneof|numeric|checkbox|orderedlist)[ \t]+varid.+?(?:endoneof|e >ndnumeric|endcheckbox|endorderedlist);', >+ re.DOTALL | re.M).findall >+StringVarIds =3D re.compile( >+ '[ \t]*(?:oneof|numeric|checkbox|orderedlist)[ \t]+varid[ \t]*=3D[ \t= ]*([a- >zA-Z_0-9]+\.[a-zA-Z_0-9]+)').findall >+StringTokens =3D re.compile('prompt[ \t]*=3D[ \t]*STRING_TOKEN[ \t]*\(([a= -zA- >Z_0-9]+)\)').findall >+ >+ >+def searchVfiFile(Filename): >+ global Options, GlobalVarId, StringGroups, StringVarIds, StringTokens= , >QuestionError >+ try: >+ with open(Filename, 'r') as VfiFile: >+ Databuffer =3D VfiFile.read() >+ >+ # Finds specified lines in file >+ VfiStringGroup =3D StringGroups(Databuffer) >+ >+ # Searches for prompts within specified lines >+ for EachGroup in VfiStringGroup: >+ for EachString in StringTokens(EachGroup): >+ # Ignore strings defined as STRING_TOKEN(0), >STRING_TOKEN(STR_EMPTY) or STRING_TOKEN(STR_NULL) >+ if (EachString !=3D "0") and (EachString !=3D "STR_EMPTY"= ) and >(EachString !=3D "STR_NULL"): >+ if EachString not in GlobalVarId: >+ GlobalVarId[EachString] =3D StringVarIds(EachGrou= p) >+ else: >+ if (GlobalVarId[EachString][0] !=3D StringVarIds(= EachGroup)[0]): >+ if Options['QuestionOption']: >+ if Options['QuestionOption'] =3D=3D "e": >+ QuestionError =3D True >+ print("ERROR:"), >+ if Options['QuestionOption'] =3D=3D "w": >+ print("WARNING:"), >+ print("%s referred by different HII quest= ions(%s and %s)" % ( >+ EachString, GlobalVarId[EachString][0= ], >StringVarIds(EachGroup)[0])) >+ except: >+ print("Error opening file at %s for reading." % Filename) >+ >+ >+if __name__ =3D=3D '__main__': >+ sys.exit(main()) >-- >2.14.1.windows.1