From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.20; helo=mga02.intel.com; envelope-from=bob.c.feng@intel.com; receiver=edk2-devel@lists.01.org Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 36FB4210FBEE9 for ; Mon, 28 Jan 2019 18:06:21 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 Jan 2019 18:06:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,535,1539673200"; d="scan'208";a="295229712" Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga005.jf.intel.com with ESMTP; 28 Jan 2019 18:06:19 -0800 From: "Feng, Bob C" To: edk2-devel@lists.01.org Date: Tue, 29 Jan 2019 10:05:37 +0800 Message-Id: <20190129020610.14300-1-bob.c.feng@intel.com> X-Mailer: git-send-email 2.20.1.windows.1 MIME-Version: 1.0 Subject: [Patch v2 00/33] BaseTools python3 migration patch set X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2019 02:06:22 -0000 Content-Transfer-Encoding: quoted-printable BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D55=0D =0D V2:=0D The python files under CParser4 folder of ECC/Eot tool=0D are generated by antlr4 and forpython3 usage. =0D They have python3 specific syntax, for example=0D the data type declaration for the arguments of a function. That=0D is not compitable with python2. this patch is to remove these syntax.=0D =0D The version2 patch set is commit to https://github.com/BobCF/edk2.git branc= h py3basetools_v2=0D =0D V1:=0D This patch set is to enable python3 on BaseTools. Basetools code will be=0D compatible with both python3 and python2.=0D =0D We will have two envs PYTHON3_ENABLE and PYTHON_COMMAND. The behavior can b= e =0D combined as the below to support this usage.=0D If user wants the specific python interpreter, he only needs to set PYTHON_= COMMAND env.=0D If PYTHON3_ENABLE is set, PYTHON_COMMAND will be set to the found one by ed= k2 scripts based on PYTHON3_ENABLE value. =0D If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is set, then PYTHON_COMMAN= D will be used to run python script. No version check here. =0D If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is not set, PYTHON_COMMAND= will be set to the high version python installed in OS. =0D =0D This patch set is verified by basic testing on Ovmf, MinKabylake and MinPur= ley platform with Python3.7.1 and =0D minimal testing on Ovmf, MinKabylake and MinPurley with Python2.7.15. =0D =0D After this change, we will focus on the Python3 validation.=0D =0D You can also review and try the patch set at https://github.com/BobCF/edk2.= git branch py3basetools=0D Feng, Bob C (9): BaseTools: use OrderedDict instead of sdict BaseTools: Make sure AllPcdList valid. BaseTools:File open failed for VPD MapFile BaseTools:Fixed Rsa issue and a set define issue. BaseTools:ord() don't match in py2 and py3 BaseTools: the list and iterator translation BaseTools: Handle the bytes and str difference BaseTools: ECC tool Python3 adaption BaseTools: Eot tool Python3 adaption Liming Gao (1): BaseTools: Update PYTHON env to PYTHON_COMMAND Yunhua Feng (3): BaseTools: nametuple not have verbose parameter in python3 BaseTools: Remove unnecessary super function BaseTools: replace long by int Zhiju Fan (3): BaseTools:TestTools character encoding issue BaseTools:Double carriage return inserted from Trim.py on Python3 BaseTools:There is extra blank line in datalog Zhijux Fan (17): BaseTool:Rename xrange() to range() BaseTools:use iterate list to replace the itertools BaseTools: Rename iteritems to items BaseTools: replace get_bytes_le() to bytes_le BaseTools:Solve the data sorting problem use python3 BaseTools: Update argparse arguments since it not have version now BaseTools:Similar to octal data rectification BaseTools/UPT:merge UPT Tool use Python2 and Python3 BaseTools: update Test scripts support python3 BaseTools/Scripts: Porting PackageDocumentTools code to use Python3 Basetools: It went wrong when use os.linesep BaseTools:Fv BaseAddress must set If it not set BaseTools: change the Division Operator BaseTools: Similar to octal data rectification BaseTools: Update windows and linux run scripts file to use Python3 BaseTools:Update build tool to print python version information BaseTools:Linux Python highest version check. BaseTools/Bin/CYGWIN_NT-5.1-i686/Ecc = | 6 +- BaseTools/Bin/CYGWIN_NT-5.1-i686/GenDepex = | 6 +- BaseTools/Bin/CYGWIN_NT-5.1-i686/GenFds = | 6 +- BaseTools/Bin/CYGWIN_NT-5.1-i686/TargetTool = | 6 +- BaseTools/Bin/CYGWIN_NT-5.1-i686/Trim = | 6 +- BaseTools/Bin/CYGWIN_NT-5.1-i686/build = | 6 +- BaseTools/BinWrappers/PosixLike/BPDG = | 6 +- BaseTools/BinWrappers/PosixLike/Ecc = | 6 +- BaseTools/BinWrappers/PosixLike/GenDepex = | 6 +- BaseTools/BinWrappers/PosixLike/GenFds = | 6 +- BaseTools/BinWrappers/PosixLike/GenPatchPcdTable = | 6 +- BaseTools/BinWrappers/PosixLike/GenerateCapsule = | 6 +- BaseTools/BinWrappers/PosixLike/PatchPcdValue = | 6 +- BaseTools/BinWrappers/PosixLike/Pkcs7Sign = | 6 +- BaseTools/BinWrappers/PosixLike/Rsa2048Sha256GenerateKeys = | 6 +- BaseTools/BinWrappers/PosixLike/Rsa2048Sha256Sign = | 6 +- BaseTools/BinWrappers/PosixLike/TargetTool = | 6 +- BaseTools/BinWrappers/PosixLike/Trim = | 6 +- BaseTools/BinWrappers/PosixLike/UPT = | 6 +- BaseTools/BinWrappers/PosixLike/build = | 6 +- BaseTools/BinWrappers/WindowsLike/BPDG.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/Ecc.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/GenDepex.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/GenFds.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/GenPatchPcdTable.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/GenerateCapsule.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/PatchPcdValue.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/Pkcs7Sign.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/Rsa2048Sha256GenerateKeys.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/Rsa2048Sha256Sign.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/TargetTool.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/Trim.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/UPT.bat = | 2 +- BaseTools/BinWrappers/WindowsLike/build.bat = | 2 +- BaseTools/Makefile = | 12 +- BaseTools/Scripts/ConvertFceToStructurePcd.py = | 2 +- BaseTools/Scripts/PackageDocumentTools/packagedoc_cli.py = | 6 +- BaseTools/Scripts/PackageDocumentTools/packagedocapp.pyw = | 14 +-- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/basemodel/doxyge= n.py | 4 +- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/baseo= bject.py | 16 +-- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/dec.p= y | 4 +- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/doxyg= engen.py | 12 +- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/doxyg= engen_spec.py | 12 +- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/dsc.p= y | 4 +- BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/edk2/model/inf.p= y | 4 +- BaseTools/Source/C/Makefile = | 12 +- BaseTools/Source/Python/AutoGen/AutoGen.py = | 90 +++++++------ BaseTools/Source/Python/AutoGen/GenC.py = | 18 +-- BaseTools/Source/Python/AutoGen/GenMake.py = | 40 +++--- BaseTools/Source/Python/AutoGen/GenPcdDb.py = | 55 ++++---- BaseTools/Source/Python/AutoGen/GenVar.py = | 34 ++--- BaseTools/Source/Python/AutoGen/InfSectionParser.py = | 2 +- BaseTools/Source/Python/AutoGen/StrGather.py = | 9 +- BaseTools/Source/Python/AutoGen/UniClassObject.py = | 4 +- BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py = | 6 +- BaseTools/Source/Python/BPDG/GenVpd.py = | 32 ++--- BaseTools/Source/Python/Common/Expression.py = | 9 +- BaseTools/Source/Python/Common/LongFilePathOs.py = | 3 +- BaseTools/Source/Python/Common/LongFilePathSupport.py = | 12 -- BaseTools/Source/Python/Common/Misc.py = | 192 +++++++--------------------- BaseTools/Source/Python/Common/StringUtils.py = | 16 +-- BaseTools/Source/Python/Common/VpdInfoFile.py = | 10 +- BaseTools/Source/Python/Ecc/{ =3D> CParser3}/CLexer.py = | 0 BaseTools/Source/Python/Ecc/{ =3D> CParser3}/CParser.py = | 0 BaseTools/Source/Python/Ecc/CParser3/__init__.py = | 0 BaseTools/Source/Python/Ecc/CParser4/C.g4 = | 637 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++++++++++++++++++ BaseTools/Source/Python/Ecc/CParser4/CLexer.py = | 632 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++ BaseTools/Source/Python/Ecc/CParser4/CListener.py = | 815 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ BaseTools/Source/Python/Ecc/CParser4/CParser.py = | 6279 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++++ BaseTools/Source/Python/Ecc/CParser4/__init__.py = | 0 BaseTools/Source/Python/Ecc/Check.py = | 4 +- BaseTools/Source/Python/Ecc/CodeFragmentCollector.py = | 20 +-- BaseTools/Source/Python/Ecc/Configuration.py = | 3 - BaseTools/Source/Python/Ecc/EccMain.py = | 2 +- BaseTools/Source/Python/Ecc/EccToolError.py = | 4 +- BaseTools/Source/Python/Ecc/FileProfile.py = | 2 +- BaseTools/Source/Python/Ecc/MetaDataParser.py = | 2 +- BaseTools/Source/Python/Ecc/c.py = | 6 +- BaseTools/Source/Python/Ecc/config.ini = | 2 - BaseTools/Source/Python/Eot/{ =3D> CParser3}/CLexer.py = | 0 BaseTools/Source/Python/Eot/{ =3D> CParser3}/CParser.py = | 0 BaseTools/Source/Python/Eot/CParser3/__init__.py = | 0 BaseTools/Source/Python/Eot/CParser4/CLexer.py = | 633 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++ BaseTools/Source/Python/Eot/CParser4/CListener.py = | 814 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ BaseTools/Source/Python/Eot/CParser4/CParser.py = | 6279 ++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++++ BaseTools/Source/Python/Eot/CParser4/__init__.py = | 0 BaseTools/Source/Python/Eot/CodeFragmentCollector.py = | 22 ++-- BaseTools/Source/Python/GenFds/AprioriSection.py = | 4 +- BaseTools/Source/Python/GenFds/Capsule.py = | 15 ++- BaseTools/Source/Python/GenFds/CapsuleData.py = | 4 +- BaseTools/Source/Python/GenFds/DataSection.py = | 4 +- BaseTools/Source/Python/GenFds/EfiSection.py = | 4 +- BaseTools/Source/Python/GenFds/Fd.py = | 4 +- BaseTools/Source/Python/GenFds/FdfParser.py = | 22 ++-- BaseTools/Source/Python/GenFds/FfsFileStatement.py = | 16 +-- BaseTools/Source/Python/GenFds/FfsInfStatement.py = | 22 ++-- BaseTools/Source/Python/GenFds/Fv.py = | 52 ++++---- BaseTools/Source/Python/GenFds/FvImageSection.py = | 22 ++-- BaseTools/Source/Python/GenFds/GenFds.py = | 37 +++--- BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py = | 4 +- BaseTools/Source/Python/GenFds/Region.py = | 10 +- BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py = | 4 +- BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py = | 15 +-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py = | 17 +-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py = | 20 +-- BaseTools/Source/Python/Trim/Trim.py = | 20 ++- BaseTools/Source/Python/UPT/Core/IpiDb.py = | 4 +- BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py = | 6 +- BaseTools/Source/Python/UPT/Library/CommentGenerating.py = | 6 +- BaseTools/Source/Python/UPT/Library/CommentParsing.py = | 10 +- BaseTools/Source/Python/UPT/Library/Misc.py = | 190 +++++----------------------- BaseTools/Source/Python/UPT/Library/ParserValidate.py = | 2 +- BaseTools/Source/Python/UPT/Library/Parsing.py = | 2 +- BaseTools/Source/Python/UPT/Library/StringUtils.py = | 40 +++--- BaseTools/Source/Python/UPT/Library/UniClassObject.py = | 6 +- BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py = | 2 +- BaseTools/Source/Python/UPT/Logger/StringTable.py = | 2 +- BaseTools/Source/Python/UPT/Parser/DecParser.py = | 8 +- BaseTools/Source/Python/UPT/Parser/DecParserMisc.py = | 30 +---- BaseTools/Source/Python/UPT/Parser/InfAsBuiltProcess.py = | 4 +- BaseTools/Source/Python/UPT/Parser/InfParser.py = | 4 +- BaseTools/Source/Python/UPT/Parser/InfSectionParser.py = | 4 +- BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py = | 2 +- BaseTools/Source/Python/UPT/UPT.py = | 1 + BaseTools/Source/Python/UPT/Xml/IniToXml.py = | 2 +- BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py = | 2 +- BaseTools/Source/Python/Workspace/BuildClassObject.py = | 6 +- BaseTools/Source/Python/Workspace/DscBuildData.py = | 34 +++-- BaseTools/Source/Python/Workspace/InfBuildData.py = | 4 +- BaseTools/Source/Python/Workspace/MetaDataTable.py = | 2 +- BaseTools/Source/Python/Workspace/MetaFileParser.py = | 4 +- BaseTools/Source/Python/build/BuildReport.py = | 38 ++++-- BaseTools/Source/Python/build/build.py = | 71 ++++++----- BaseTools/Tests/CToolsTests.py = | 2 +- BaseTools/Tests/CheckUnicodeSourceFiles.py = | 6 +- BaseTools/Tests/GNUmakefile = | 2 +- BaseTools/Tests/PythonTest.py = | 15 +++ BaseTools/Tests/TestTools.py = | 14 ++- BaseTools/toolsetup.bat = | 84 +++++++++++-- edksetup.sh = | 77 +++++++++++- 140 files changed, 16982 insertions(+), 938 deletions(-) rename BaseTools/Source/Python/Ecc/{ =3D> CParser3}/CLexer.py (100%) rename BaseTools/Source/Python/Ecc/{ =3D> CParser3}/CParser.py (100%) create mode 100644 BaseTools/Source/Python/Ecc/CParser3/__init__.py create mode 100644 BaseTools/Source/Python/Ecc/CParser4/C.g4 create mode 100644 BaseTools/Source/Python/Ecc/CParser4/CLexer.py create mode 100644 BaseTools/Source/Python/Ecc/CParser4/CListener.py create mode 100644 BaseTools/Source/Python/Ecc/CParser4/CParser.py create mode 100644 BaseTools/Source/Python/Ecc/CParser4/__init__.py rename BaseTools/Source/Python/Eot/{ =3D> CParser3}/CLexer.py (100%) rename BaseTools/Source/Python/Eot/{ =3D> CParser3}/CParser.py (100%) create mode 100644 BaseTools/Source/Python/Eot/CParser3/__init__.py create mode 100644 BaseTools/Source/Python/Eot/CParser4/CLexer.py create mode 100644 BaseTools/Source/Python/Eot/CParser4/CListener.py create mode 100644 BaseTools/Source/Python/Eot/CParser4/CParser.py create mode 100644 BaseTools/Source/Python/Eot/CParser4/__init__.py create mode 100644 BaseTools/Tests/PythonTest.py --=20 2.20.1.windows.1