From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=195.135.221.5; helo=smtp.nue.novell.com; envelope-from=glin@suse.com; receiver=edk2-devel@lists.01.org Received: from smtp.nue.novell.com (smtp.nue.novell.com [195.135.221.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 3FAA5211D5052 for ; Wed, 20 Jun 2018 21:44:15 -0700 (PDT) Received: from localhost.localdomain (unknown.telstraglobal.net [134.159.103.118]) by smtp.nue.novell.com with ESMTP (NOT encrypted); Thu, 21 Jun 2018 06:44:10 +0200 From: Gary Lin To: edk2-devel@lists.01.org Cc: Yonghong Zhu , Liming Gao Date: Thu, 21 Jun 2018 12:43:26 +0800 Message-Id: <20180621044346.28495-1-glin@suse.com> X-Mailer: git-send-email 2.17.1 Subject: [PATCH v3 00/20] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jun 2018 04:44:17 -0000 v3 changes: - Rebase to the current git HEAD (2e1083038d9aa74fcaa2db8158fdee7c8b4af3bb) - Fix a typo in BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/basemodel/ini.py - Remove the patch for reduce() since it's not used anymore v2 changes: - Rebase to the current git HEAD (821807bcefb9a36e598d71a8004fae5aab2052a0) - Apply "futurize -f libfuturize.fixes.fix_absolute_import" and refactor some python scripts to break the circular imports. This patch series is also available in https://github.com/lcp/edk2/tree/python3-futurize-v3 Since python2 will be EOL in 2020, we start to evaluate the impact of the python2 removal. As expected, OMVF building failed the test. It's actually a task noted in the wiki page: https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Python3-Support Maybe it's time to convert the python scripts gradually. This patchset doesn't make the python scripts in BaseTools compatible with python3 immediately. It aims to do the trivial and safe conversion and replacement to make some statements compatible with both python2 and python3, so we can deal with the difficult cases later. With the help of "futurize" from python-future, it's easier to refactor the statements. This patchset is basically equivalent to "futurize -1"(*) plus "StringIO.StringIO => io.BytesIO" and several patches to break the circular imports. The patchset was tested with the following command: $ ./OvmfPkg/build.sh -D SECURE_BOOT_ENABLE \ -D NETWORK_IP6_ENABLE \ -D HTTP_BOOT_ENABLE \ -D TLS_ENABLE \ -D TPM2_ENABLE The firmware file was built successfully and I didn't notice any error so far. (*) http://python-future.org/automatic_conversion.html#stage-1-safe-fixes Contributed-under: TianoCore Contribution Agreement 1.1 Cc: Yonghong Zhu Cc: Liming Gao Signed-off-by: Gary Lin Gary Lin (20): BaseTools: Fix a typo in ini.py BaseTools: Refactor python except statements BaseTools: Refactor python print statements BaseTools: Remove the old python "not-equal" BaseTools: Use the python3-range functions BaseTools: Remove tuple parameter in python scripts BaseTools: Remove the deprecated hash_key() BaseTools: Replace StandardError with Expression BaseTools: Remove types.TypeType BaseTools: Refactor python raise statement BaseTools: Adjust the spaces around commas and colons BaseTools: Migrate to the new octal literal BaseTools: Unify long int and int in python scripts BaseTools: Fix old python2 idioms BaseTools: Replace StringIO.StringIO with io.BytesIO BaseTools: Treat GenFds.py and build.py as python modules BaseTools: Adopt absolute import for python scripts BaseTools: Move OverrideAttribs to OptRomInfStatement.py BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py BaseTools: Move ImageBinDict to GenFdsGlobalVariable.py .../Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py | 5 +- BaseTools/BinWrappers/PosixLike/GenFds | 2 +- BaseTools/BinWrappers/PosixLike/build | 2 +- BaseTools/Scripts/BinToPcd.py | 10 +- BaseTools/Scripts/ConvertMasmToNasm.py | 1 + BaseTools/Scripts/ConvertUni.py | 5 - BaseTools/Scripts/FormatDosFiles.py | 6 +- BaseTools/Scripts/MemoryProfileSymbolGen.py | 21 +- .../PackageDocumentTools/packagedoc_cli.py | 53 ++-- .../plugins/EdkPlugins/basemodel/doxygen.py | 14 +- .../plugins/EdkPlugins/basemodel/efibinary.py | 30 +- .../plugins/EdkPlugins/basemodel/ini.py | 8 +- .../EdkPlugins/edk2/model/baseobject.py | 6 +- .../EdkPlugins/edk2/model/doxygengen.py | 5 +- .../EdkPlugins/edk2/model/doxygengen_spec.py | 5 +- .../plugins/EdkPlugins/edk2/model/inf.py | 8 +- BaseTools/Scripts/PatchCheck.py | 7 +- BaseTools/Scripts/RunMakefile.py | 2 +- .../Scripts/SmiHandlerProfileSymbolGen.py | 19 +- BaseTools/Scripts/UpdateBuildVersions.py | 18 +- BaseTools/Source/Python/AutoGen/AutoGen.py | 93 +++--- .../Source/Python/AutoGen/BuildEngine.py | 38 +-- BaseTools/Source/Python/AutoGen/GenC.py | 80 +++--- BaseTools/Source/Python/AutoGen/GenDepex.py | 8 +- BaseTools/Source/Python/AutoGen/GenMake.py | 11 +- BaseTools/Source/Python/AutoGen/GenPcdDb.py | 148 +++++----- BaseTools/Source/Python/AutoGen/GenVar.py | 163 +++++------ .../Source/Python/AutoGen/IdfClassObject.py | 4 +- .../Source/Python/AutoGen/InfSectionParser.py | 1 + BaseTools/Source/Python/AutoGen/StrGather.py | 12 +- .../Source/Python/AutoGen/UniClassObject.py | 18 +- .../Python/AutoGen/ValidCheckingInfoObject.py | 5 +- BaseTools/Source/Python/BPDG/BPDG.py | 8 +- BaseTools/Source/Python/BPDG/GenVpd.py | 28 +- BaseTools/Source/Python/Common/DataType.py | 4 +- BaseTools/Source/Python/Common/Database.py | 9 +- BaseTools/Source/Python/Common/EdkLogger.py | 3 +- BaseTools/Source/Python/Common/Expression.py | 84 +++--- .../Source/Python/Common/LongFilePathOs.py | 5 +- BaseTools/Source/Python/Common/Misc.py | 61 ++-- BaseTools/Source/Python/Common/Parsing.py | 6 +- .../Source/Python/Common/RangeExpression.py | 34 +-- BaseTools/Source/Python/Common/StringUtils.py | 16 +- .../Python/Common/TargetTxtClassObject.py | 16 +- .../Python/Common/ToolDefClassObject.py | 16 +- BaseTools/Source/Python/Common/VpdInfoFile.py | 23 +- BaseTools/Source/Python/Ecc/CParser.py | 178 ++++++------ BaseTools/Source/Python/Ecc/Check.py | 24 +- .../Python/Ecc/CodeFragmentCollector.py | 82 +++--- BaseTools/Source/Python/Ecc/Configuration.py | 5 +- BaseTools/Source/Python/Ecc/Database.py | 7 +- BaseTools/Source/Python/Ecc/Ecc.py | 25 +- BaseTools/Source/Python/Ecc/Exception.py | 6 +- BaseTools/Source/Python/Ecc/FileProfile.py | 5 +- BaseTools/Source/Python/Ecc/MetaDataParser.py | 8 +- .../Ecc/MetaFileWorkspace/MetaDataTable.py | 5 +- .../Ecc/MetaFileWorkspace/MetaFileParser.py | 46 +-- .../Ecc/MetaFileWorkspace/MetaFileTable.py | 5 +- .../Source/Python/Ecc/Xml/XmlRoutines.py | 9 +- BaseTools/Source/Python/Ecc/c.py | 28 +- BaseTools/Source/Python/Eot/CParser.py | 178 ++++++------ .../Python/Eot/CodeFragmentCollector.py | 72 ++--- BaseTools/Source/Python/Eot/Eot.py | 22 +- BaseTools/Source/Python/Eot/FileProfile.py | 3 +- BaseTools/Source/Python/Eot/InfParserLite.py | 13 +- BaseTools/Source/Python/Eot/Parser.py | 5 +- BaseTools/Source/Python/Eot/Report.py | 3 +- BaseTools/Source/Python/Eot/c.py | 32 ++- .../Source/Python/GenFds/AprioriSection.py | 12 +- BaseTools/Source/Python/GenFds/Capsule.py | 24 +- BaseTools/Source/Python/GenFds/CapsuleData.py | 11 +- .../Source/Python/GenFds/CompressSection.py | 7 +- BaseTools/Source/Python/GenFds/DataSection.py | 7 +- .../Source/Python/GenFds/DepexSection.py | 7 +- BaseTools/Source/Python/GenFds/EfiSection.py | 13 +- BaseTools/Source/Python/GenFds/Fd.py | 33 +-- BaseTools/Source/Python/GenFds/FdfParser.py | 102 +++---- .../Source/Python/GenFds/FfsFileStatement.py | 16 +- .../Source/Python/GenFds/FfsInfStatement.py | 37 +-- BaseTools/Source/Python/GenFds/Fv.py | 32 ++- .../Source/Python/GenFds/FvImageSection.py | 15 +- BaseTools/Source/Python/GenFds/GenFds.py | 123 ++------ .../Python/GenFds/GenFdsGlobalVariable.py | 107 ++++++- BaseTools/Source/Python/GenFds/GuidSection.py | 11 +- .../Python/GenFds/OptRomFileStatement.py | 3 +- .../Python/GenFds/OptRomInfStatement.py | 28 +- BaseTools/Source/Python/GenFds/OptionRom.py | 23 +- BaseTools/Source/Python/GenFds/Region.py | 17 +- .../Source/Python/GenFds/RuleComplexFile.py | 3 +- .../Source/Python/GenFds/RuleSimpleFile.py | 3 +- BaseTools/Source/Python/GenFds/Section.py | 3 +- BaseTools/Source/Python/GenFds/UiSection.py | 7 +- BaseTools/Source/Python/GenFds/VerSection.py | 7 +- BaseTools/Source/Python/GenFds/Vtf.py | 3 +- .../GenPatchPcdTable/GenPatchPcdTable.py | 9 +- .../Python/PatchPcdValue/PatchPcdValue.py | 1 + .../Source/Python/Pkcs7Sign/Pkcs7Sign.py | 32 ++- .../Rsa2048Sha256GenerateKeys.py | 26 +- .../Rsa2048Sha256Sign/Rsa2048Sha256Sign.py | 36 +-- .../Source/Python/Table/TableDataModel.py | 3 +- BaseTools/Source/Python/Table/TableDec.py | 3 +- BaseTools/Source/Python/Table/TableDsc.py | 3 +- .../Source/Python/Table/TableEotReport.py | 5 +- BaseTools/Source/Python/Table/TableFdf.py | 3 +- BaseTools/Source/Python/Table/TableFile.py | 3 +- .../Source/Python/Table/TableFunction.py | 3 +- .../Source/Python/Table/TableIdentifier.py | 5 +- BaseTools/Source/Python/Table/TableInf.py | 3 +- BaseTools/Source/Python/Table/TablePcd.py | 5 +- BaseTools/Source/Python/Table/TableQuery.py | 3 +- BaseTools/Source/Python/Table/TableReport.py | 3 +- .../Source/Python/TargetTool/TargetTool.py | 39 +-- BaseTools/Source/Python/Trim/Trim.py | 25 +- .../Source/Python/UPT/Core/DependencyRules.py | 12 +- .../UPT/Core/DistributionPackageClass.py | 4 +- BaseTools/Source/Python/UPT/Core/FileHook.py | 2 +- BaseTools/Source/Python/UPT/Core/IpiDb.py | 6 +- .../Source/Python/UPT/Core/PackageFile.py | 12 +- .../Python/UPT/GenMetaFile/GenDecFile.py | 15 +- .../Python/UPT/GenMetaFile/GenInfFile.py | 42 ++- BaseTools/Source/Python/UPT/InstallPkg.py | 2 +- BaseTools/Source/Python/UPT/InventoryWs.py | 2 +- .../Python/UPT/Library/CommentParsing.py | 5 +- .../Python/UPT/Library/ExpressionValidate.py | 11 +- BaseTools/Source/Python/UPT/Library/Misc.py | 11 +- .../Python/UPT/Library/ParserValidate.py | 2 +- .../Source/Python/UPT/Library/Parsing.py | 6 +- .../Source/Python/UPT/Library/StringUtils.py | 5 +- .../Python/UPT/Library/UniClassObject.py | 20 +- .../Python/UPT/Library/Xml/XmlRoutines.py | 4 +- BaseTools/Source/Python/UPT/MkPkg.py | 2 +- .../UPT/Object/Parser/InfBinaryObject.py | 6 +- .../UPT/Object/Parser/InfDefineObject.py | 2 +- .../Python/UPT/Object/Parser/InfGuidObject.py | 4 +- .../Object/Parser/InfLibraryClassesObject.py | 2 +- .../Python/UPT/Object/Parser/InfMisc.py | 4 +- .../UPT/Object/Parser/InfPackagesObject.py | 4 +- .../Python/UPT/Object/Parser/InfPcdObject.py | 4 +- .../Python/UPT/Object/Parser/InfPpiObject.py | 4 +- .../UPT/Object/Parser/InfProtocolObject.py | 2 +- .../UPT/Object/Parser/InfSoucesObject.py | 3 +- .../Object/Parser/InfUserExtensionObject.py | 4 +- .../Source/Python/UPT/Parser/DecParserMisc.py | 1 + .../Python/UPT/Parser/InfSectionParser.py | 3 +- .../Python/UPT/PomAdapter/DecPomAlignment.py | 57 ++-- .../Python/UPT/PomAdapter/InfPomAlignment.py | 3 +- .../UPT/PomAdapter/InfPomAlignmentMisc.py | 3 +- BaseTools/Source/Python/UPT/ReplacePkg.py | 2 +- BaseTools/Source/Python/UPT/RmPkg.py | 2 +- BaseTools/Source/Python/UPT/TestInstall.py | 4 +- BaseTools/Source/Python/UPT/UPT.py | 9 +- .../Python/UPT/UnitTest/DecParserTest.py | 5 +- .../UPT/UnitTest/InfBinarySectionTest.py | 10 +- BaseTools/Source/Python/UPT/Xml/CommonXml.py | 2 +- BaseTools/Source/Python/UPT/Xml/IniToXml.py | 1 + BaseTools/Source/Python/UPT/Xml/XmlParser.py | 25 +- .../Source/Python/UPT/Xml/XmlParserMisc.py | 3 +- .../Python/Workspace/BuildClassObject.py | 16 +- .../Source/Python/Workspace/DecBuildData.py | 22 +- .../Source/Python/Workspace/DscBuildData.py | 269 +++++++++--------- .../Source/Python/Workspace/InfBuildData.py | 6 +- .../Source/Python/Workspace/MetaFileParser.py | 75 ++--- .../Source/Python/Workspace/MetaFileTable.py | 15 +- .../Python/Workspace/WorkspaceCommon.py | 5 +- .../Python/Workspace/WorkspaceDatabase.py | 7 +- BaseTools/Source/Python/build/BuildReport.py | 21 +- BaseTools/Source/Python/build/build.py | 42 +-- BaseTools/Tests/CheckPythonSyntax.py | 2 +- BaseTools/Tests/TestTools.py | 13 +- BaseTools/Tests/TianoCompress.py | 6 +- BaseTools/gcc/mingw-gcc-build.py | 112 ++++---- 171 files changed, 1927 insertions(+), 1762 deletions(-) -- 2.17.1