From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=195.135.221.5; helo=smtp.nue.novell.com; envelope-from=glin@suse.com; receiver=edk2-devel@lists.01.org Received: from smtp.nue.novell.com (smtp.nue.novell.com [195.135.221.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id F114A2239365A for ; Thu, 1 Feb 2018 00:31:03 -0800 (PST) Received: from localhost.localdomain (unknown.telstraglobal.net [134.159.103.118]) by smtp.nue.novell.com with ESMTP (NOT encrypted); Thu, 01 Feb 2018 09:36:37 +0100 From: Gary Lin To: edk2-devel@lists.01.org Cc: Yonghong Zhu , Liming Gao Date: Thu, 1 Feb 2018 16:35:48 +0800 Message-Id: <20180201083608.16036-1-glin@suse.com> X-Mailer: git-send-email 2.16.1 Subject: [PATCH v2 00/20] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2018 08:31:04 -0000 v2 changes: - Rebase to the current git HEAD (821807bcefb9a36e598d71a8004fae5aab2052a0) - Apply "futurize -f libfuturize.fixes.fix_absolute_import" and refactor some python scripts to break the circular imports. This patch series is also available in https://github.com/lcp/edk2/tree/python3-futurize-v2 Since python2 will be EOL in 2020, we start to evaluate the impact of the python2 removal. As expected, OMVF building failed the test. It's actually a task noted in the wiki page: https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Python3-Support Maybe it's time to convert the python scripts gradully. This patchset doesn't make the python scripts in BaseTools compatible with python3 immediately. It aims to do the trivial and safe conversion and replacement to make some statements compatible with both python2 and python3, so we can deal with the difficult cases later. With the help of "futurize" from python-future, it's easier to refactor the statements. This patchset is basically equivalent to "futurize -1" plus "StringIO.StringIO => io.BytesIO". For the "io.BytesIO" change, it MIGHT introduce slow down to the build time since io.BytesIO is slower than StringIO.StringIO in python2(*). For a quick test, I built OVMF with the following command based on 8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f: $ rm -rf Build && make -C BaseTools/ clean $ time ./OvmfPkg/build.sh -D SECURE_BOOT_ENABLE \ -D NETWORK_IP6_ENABLE \ -D HTTP_BOOT_ENABLE \ -D TLS_ENABLE Before io.BytesIO: Build total time: 00:03:56 real 4m22.991s user 3m55.874s sys 0m27.250s After io.BytesIO: Build total time: 00:03:57 real 4m23.953s user 3m57.526s sys 0m27.192s The difference is only 1 second, and I would say the impact is subtle. The next step will be fixing relative import and maybe applying more futurize fixes. We won't get there soon but at least we are moving... (*) https://stackoverflow.com/questions/37462075/confusing-about-stringio-cstringio-and-byteio Contributed-under: TianoCore Contribution Agreement 1.1 Cc: Yonghong Zhu Cc: Liming Gao Signed-off-by: Gary Lin Gary Lin (20): BaseTools: Refactor python except statements BaseTools: Refactor python print statements BaseTools: Remove the old python "not-equal" BaseTools: Use the python3-range functions BaseTools: Remove tuple parameter in python scripts BaseTools: Remove the deprecated hash_key() BaseTools: Import reduce() from functools BaseTools: Replace StandardError with Expression BaseTools: Remove types.TypeType BaseTools: Refactor python raise statement BaseTools: Adjust the spaces around commas and colons BaseTools: Migrate to the new octal literal BaseTools: Unify long int and int in python scripts BaseTools: Adjust old python2 idioms BaseTools: Replace StringIO.StringIO with io.BytesIO BaseTools: Treat GenFds.py and build.py as python modules BaseTools: Adopt absolute import for python scripts BaseTools: Move OverrideAttribs to OptRomInfStatement.py BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py BaseTools: Move ImageBinDict to GenFdsGlobalVariable.py BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py | 5 +- BaseTools/BinWrappers/PosixLike/GenFds | 2 +- BaseTools/BinWrappers/PosixLike/build | 2 +- BaseTools/Scripts/BinToPcd.py | 46 +++-- BaseTools/Scripts/ConvertMasmToNasm.py | 1 + BaseTools/Scripts/ConvertUni.py | 5 - BaseTools/Scripts/MemoryProfileSymbolGen.py | 22 +- BaseTools/Scripts/PatchCheck.py | 7 +- BaseTools/Scripts/RunMakefile.py | 2 +- BaseTools/Scripts/SmiHandlerProfileSymbolGen.py | 20 +- BaseTools/Scripts/UpdateBuildVersions.py | 18 +- BaseTools/Source/Python/AutoGen/AutoGen.py | 98 ++++----- BaseTools/Source/Python/AutoGen/BuildEngine.py | 38 ++-- BaseTools/Source/Python/AutoGen/GenC.py | 12 +- BaseTools/Source/Python/AutoGen/GenDepex.py | 8 +- BaseTools/Source/Python/AutoGen/GenMake.py | 11 +- BaseTools/Source/Python/AutoGen/GenPcdDb.py | 149 +++++++------- BaseTools/Source/Python/AutoGen/GenVar.py | 166 +++++++-------- BaseTools/Source/Python/AutoGen/IdfClassObject.py | 1 - BaseTools/Source/Python/AutoGen/InfSectionParser.py | 1 + BaseTools/Source/Python/AutoGen/StrGather.py | 8 +- BaseTools/Source/Python/AutoGen/UniClassObject.py | 18 +- BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py | 10 +- BaseTools/Source/Python/BPDG/BPDG.py | 8 +- BaseTools/Source/Python/BPDG/GenVpd.py | 28 +-- BaseTools/Source/Python/Common/DataType.py | 4 +- BaseTools/Source/Python/Common/Database.py | 8 +- BaseTools/Source/Python/Common/DecClassObject.py | 56 ++--- BaseTools/Source/Python/Common/Dictionary.py | 14 +- BaseTools/Source/Python/Common/DscClassObject.py | 91 +++++---- BaseTools/Source/Python/Common/EdkIIWorkspace.py | 28 +-- BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py | 152 +++++++------- BaseTools/Source/Python/Common/EdkLogger.py | 3 +- BaseTools/Source/Python/Common/Expression.py | 86 ++++---- BaseTools/Source/Python/Common/FdfClassObject.py | 6 +- BaseTools/Source/Python/Common/FdfParserLite.py | 47 ++--- BaseTools/Source/Python/Common/InfClassObject.py | 134 ++++++------ BaseTools/Source/Python/Common/LongFilePathOs.py | 5 +- BaseTools/Source/Python/Common/MigrationUtilities.py | 4 +- BaseTools/Source/Python/Common/Misc.py | 79 ++++---- BaseTools/Source/Python/Common/Parsing.py | 6 +- BaseTools/Source/Python/Common/RangeExpression.py | 32 +-- BaseTools/Source/Python/Common/String.py | 16 +- BaseTools/Source/Python/Common/TargetTxtClassObject.py | 24 ++- BaseTools/Source/Python/Common/ToolDefClassObject.py | 12 +- BaseTools/Source/Python/Common/VpdInfoFile.py | 23 ++- BaseTools/Source/Python/CommonDataClass/ModuleClass.py | 3 +- BaseTools/Source/Python/CommonDataClass/PackageClass.py | 3 +- BaseTools/Source/Python/CommonDataClass/PlatformClass.py | 3 +- BaseTools/Source/Python/Ecc/CParser.py | 178 ++++++++-------- BaseTools/Source/Python/Ecc/Check.py | 10 +- BaseTools/Source/Python/Ecc/CodeFragmentCollector.py | 82 ++++---- BaseTools/Source/Python/Ecc/Configuration.py | 5 +- BaseTools/Source/Python/Ecc/Database.py | 7 +- BaseTools/Source/Python/Ecc/Ecc.py | 25 +-- BaseTools/Source/Python/Ecc/Exception.py | 6 +- BaseTools/Source/Python/Ecc/FileProfile.py | 5 +- BaseTools/Source/Python/Ecc/MetaDataParser.py | 8 +- BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py | 5 +- BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py | 44 ++-- BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileTable.py | 5 +- BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py | 9 +- BaseTools/Source/Python/Ecc/c.py | 28 +-- BaseTools/Source/Python/Eot/CParser.py | 178 ++++++++-------- BaseTools/Source/Python/Eot/CodeFragmentCollector.py | 72 +++---- BaseTools/Source/Python/Eot/Eot.py | 15 +- BaseTools/Source/Python/Eot/FileProfile.py | 3 +- BaseTools/Source/Python/Eot/FvImage.py | 28 +-- BaseTools/Source/Python/Eot/InfParserLite.py | 13 +- BaseTools/Source/Python/Eot/Parser.py | 5 +- BaseTools/Source/Python/Eot/Report.py | 3 +- BaseTools/Source/Python/Eot/c.py | 32 +-- BaseTools/Source/Python/GenFds/AprioriSection.py | 12 +- BaseTools/Source/Python/GenFds/Capsule.py | 22 +- BaseTools/Source/Python/GenFds/CapsuleData.py | 11 +- BaseTools/Source/Python/GenFds/CompressSection.py | 7 +- BaseTools/Source/Python/GenFds/DataSection.py | 7 +- BaseTools/Source/Python/GenFds/DepexSection.py | 7 +- BaseTools/Source/Python/GenFds/EfiSection.py | 13 +- BaseTools/Source/Python/GenFds/Fd.py | 32 +-- BaseTools/Source/Python/GenFds/FdfParser.py | 100 ++++----- BaseTools/Source/Python/GenFds/FfsFileStatement.py | 16 +- BaseTools/Source/Python/GenFds/FfsInfStatement.py | 35 ++-- BaseTools/Source/Python/GenFds/Fv.py | 34 ++-- BaseTools/Source/Python/GenFds/FvImageSection.py | 15 +- BaseTools/Source/Python/GenFds/GenFds.py | 126 ++---------- BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 108 +++++++++- BaseTools/Source/Python/GenFds/GuidSection.py | 11 +- BaseTools/Source/Python/GenFds/OptRomFileStatement.py | 3 +- BaseTools/Source/Python/GenFds/OptRomInfStatement.py | 30 ++- BaseTools/Source/Python/GenFds/OptionRom.py | 23 +-- BaseTools/Source/Python/GenFds/Region.py | 17 +- BaseTools/Source/Python/GenFds/RuleComplexFile.py | 3 +- BaseTools/Source/Python/GenFds/RuleSimpleFile.py | 3 +- BaseTools/Source/Python/GenFds/Section.py | 3 +- BaseTools/Source/Python/GenFds/UiSection.py | 7 +- BaseTools/Source/Python/GenFds/VerSection.py | 7 +- BaseTools/Source/Python/GenFds/Vtf.py | 3 +- BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py | 9 +- BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py | 1 + BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py | 32 +-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py | 30 +-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py | 36 ++-- BaseTools/Source/Python/Table/TableDataModel.py | 3 +- BaseTools/Source/Python/Table/TableDec.py | 3 +- BaseTools/Source/Python/Table/TableDsc.py | 3 +- BaseTools/Source/Python/Table/TableEotReport.py | 5 +- BaseTools/Source/Python/Table/TableFdf.py | 3 +- BaseTools/Source/Python/Table/TableFile.py | 3 +- BaseTools/Source/Python/Table/TableFunction.py | 3 +- BaseTools/Source/Python/Table/TableIdentifier.py | 5 +- BaseTools/Source/Python/Table/TableInf.py | 3 +- BaseTools/Source/Python/Table/TablePcd.py | 5 +- BaseTools/Source/Python/Table/TableQuery.py | 3 +- BaseTools/Source/Python/Table/TableReport.py | 3 +- BaseTools/Source/Python/TargetTool/TargetTool.py | 39 ++-- BaseTools/Source/Python/Trim/Trim.py | 25 +-- BaseTools/Source/Python/UPT/Core/DependencyRules.py | 12 +- BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py | 4 +- BaseTools/Source/Python/UPT/Core/FileHook.py | 2 +- BaseTools/Source/Python/UPT/Core/IpiDb.py | 6 +- BaseTools/Source/Python/UPT/Core/PackageFile.py | 12 +- BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py | 15 +- BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py | 42 ++-- BaseTools/Source/Python/UPT/InstallPkg.py | 2 +- BaseTools/Source/Python/UPT/InventoryWs.py | 2 +- BaseTools/Source/Python/UPT/Library/CommentParsing.py | 5 +- BaseTools/Source/Python/UPT/Library/ExpressionValidate.py | 11 +- BaseTools/Source/Python/UPT/Library/Misc.py | 11 +- BaseTools/Source/Python/UPT/Library/ParserValidate.py | 2 +- BaseTools/Source/Python/UPT/Library/Parsing.py | 6 +- BaseTools/Source/Python/UPT/Library/String.py | 5 +- BaseTools/Source/Python/UPT/Library/UniClassObject.py | 20 +- BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py | 4 +- BaseTools/Source/Python/UPT/MkPkg.py | 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py | 6 +- BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py | 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py | 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py | 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py | 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py | 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py | 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py | 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py | 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py | 3 +- BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py | 4 +- BaseTools/Source/Python/UPT/Parser/DecParserMisc.py | 1 + BaseTools/Source/Python/UPT/Parser/InfSectionParser.py | 3 +- BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py | 57 +++--- BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py | 3 +- BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py | 3 +- BaseTools/Source/Python/UPT/ReplacePkg.py | 2 +- BaseTools/Source/Python/UPT/RmPkg.py | 2 +- BaseTools/Source/Python/UPT/TestInstall.py | 4 +- BaseTools/Source/Python/UPT/UPT.py | 9 +- BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py | 5 +- BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py | 10 +- BaseTools/Source/Python/UPT/Xml/CommonXml.py | 2 +- BaseTools/Source/Python/UPT/Xml/IniToXml.py | 1 + BaseTools/Source/Python/UPT/Xml/XmlParser.py | 25 +-- BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py | 3 +- BaseTools/Source/Python/Workspace/BuildClassObject.py | 2 +- BaseTools/Source/Python/Workspace/DecBuildData.py | 14 +- BaseTools/Source/Python/Workspace/DscBuildData.py | 213 ++++++++++---------- BaseTools/Source/Python/Workspace/InfBuildData.py | 6 +- BaseTools/Source/Python/Workspace/MetaFileParser.py | 75 +++---- BaseTools/Source/Python/Workspace/MetaFileTable.py | 15 +- BaseTools/Source/Python/Workspace/WorkspaceCommon.py | 5 +- BaseTools/Source/Python/Workspace/WorkspaceDatabase.py | 7 +- BaseTools/Source/Python/build/BuildReport.py | 19 +- BaseTools/Source/Python/build/build.py | 38 ++-- BaseTools/Tests/CheckPythonSyntax.py | 2 +- BaseTools/Tests/TestTools.py | 13 +- BaseTools/Tests/TianoCompress.py | 6 +- BaseTools/gcc/mingw-gcc-build.py | 112 +++++----- 175 files changed, 2092 insertions(+), 1927 deletions(-) -- 2.16.1