From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=195.135.221.5; helo=smtp.nue.novell.com; envelope-from=glin@suse.com; receiver=edk2-devel@lists.01.org Received: from smtp.nue.novell.com (smtp.nue.novell.com [195.135.221.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 6C8FF21E25698 for ; Thu, 25 Jan 2018 17:56:21 -0800 (PST) Received: from emea4-mta.ukb.novell.com ([10.120.13.87]) by smtp.nue.novell.com with ESMTP (TLS encrypted); Fri, 26 Jan 2018 03:01:49 +0100 Received: from GaryWorkstation (nwb-a10-snat.microfocus.com [10.120.13.201]) by emea4-mta.ukb.novell.com with ESMTP (TLS encrypted); Fri, 26 Jan 2018 02:01:31 +0000 Date: Fri, 26 Jan 2018 10:01:26 +0800 From: Gary Lin To: "Zhu, Yonghong" Cc: "edk2-devel@lists.01.org" , "Gao, Liming" Message-ID: <20180126020126.xdpzi4wbz3iejvwd@GaryWorkstation> References: <20180119044316.4713-1-glin@suse.com> MIME-Version: 1.0 In-Reply-To: User-Agent: NeoMutt/20170912 (1.9.0) Subject: Re: [PATCH 00/15] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jan 2018 01:56:23 -0000 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Jan 25, 2018 at 01:37:39PM +0000, Zhu, Yonghong wrote: > Hi Gary, > > Thanks for your patches. I am still in evaluating these changes and do some verification for it. > I still need some more days to give you comment. Thanks. I forgot to mention my branch in the cover letter. https://github.com/lcp/edk2/tree/python3-futurize It would be easier to review/apply patches from a git branch. Thanks, Gary Lin > > Best Regards, > Zhu Yonghong > > > -----Original Message----- > From: Gary Lin [mailto:glin@suse.com] > Sent: Friday, January 19, 2018 12:43 PM > To: edk2-devel@lists.01.org > Cc: Zhu, Yonghong ; Gao, Liming > Subject: [PATCH 00/15] BaseTools: One step toward python3 > > Since python2 will be EOF in 2020, we start to evaluate the impact of the python2 removal. As expected, OMVF building failed the test. It's actually a task noted in the wiki page: > > https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Python3-Support > > Maybe it's time to convert the python scripts gradully. > > This patchset doesn't make the python scripts in BaseTools compatible with python3 immediately. It aims to do the trivial and safe conversion and replacement to make some statements compatible with both python2 and python3, so we can deal with the difficult cases later. > > With the help of "futurize" from python-future, it's easier to refactor the statements. This patchset is basically equivalent to "futurize -1" > plus "StringIO.StringIO => io.BytesIO" and minus "fix_absolute_import". > The reason to skip "fix_absolute_import" is that python2 failed to find some modules after converting to absolute import, and it might take time to figure out a proper fix. > > For the "io.BytesIO" change, it MIGHT introduce slow down to the build time since io.BytesIO is slower than StringIO.StringIO in python2(*). > For a quick test, I built OVMF with the following command based on > 8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f: > > $ rm -rf Build && make -C BaseTools/ clean $ time ./OvmfPkg/build.sh -D SECURE_BOOT_ENABLE \ > -D NETWORK_IP6_ENABLE \ > -D HTTP_BOOT_ENABLE \ > -D TLS_ENABLE > > Before io.BytesIO: > > Build total time: 00:03:56 > real 4m22.991s > user 3m55.874s > sys 0m27.250s > > After io.BytesIO: > > Build total time: 00:03:57 > real 4m23.953s > user 3m57.526s > sys 0m27.192s > > The difference is only 1 second, and I would say the impact is subtle. > > The next step will be fixing relative import and maybe applying more futurize fixes. We won't get there soon but at least we are moving... > > (*) https://stackoverflow.com/questions/37462075/confusing-about-stringio-cstringio-and-byteio > > Contributed-under: TianoCore Contribution Agreement 1.1 > Cc: Yonghong Zhu > Cc: Liming Gao > Signed-off-by: Gary Lin > > Gary Lin (15): > BaseTools: Refactor python except statements > BaseTools: Refactor python print statements > BaseTools: Remove the old python "not-equal" > BaseTools: Use the python3-range functions > BaseTools: Remove tuple parameter in python scripts > BaseTools: Remove the deprecated hash_key() > BaseTools: Import reduce() from functools > BaseTools: Replace StandardError with Expression > BaseTools: Remove types.TypeType > BaseTools: Refactor python raise statement > BaseTools: Adjust the spaces around commas and colons > BaseTools: Migrate to the new octal literal > BaseTools: Unify long int and int in python scripts > BaseTools: Adjust old python2 idioms > BaseTools: Replace StringIO.StringIO with io.BytesIO > > BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py | 5 +- > BaseTools/Scripts/BinToPcd.py | 46 +++--- > BaseTools/Scripts/ConvertMasmToNasm.py | 1 + > BaseTools/Scripts/ConvertUni.py | 5 - > BaseTools/Scripts/MemoryProfileSymbolGen.py | 22 +-- > BaseTools/Scripts/PatchCheck.py | 7 +- > BaseTools/Scripts/RunMakefile.py | 2 +- > BaseTools/Scripts/SmiHandlerProfileSymbolGen.py | 20 +-- > BaseTools/Scripts/UpdateBuildVersions.py | 18 +- > BaseTools/Source/Python/AutoGen/AutoGen.py | 91 +++++----- > BaseTools/Source/Python/AutoGen/BuildEngine.py | 38 +++-- > BaseTools/Source/Python/AutoGen/GenC.py | 5 +- > BaseTools/Source/Python/AutoGen/GenDepex.py | 8 +- > BaseTools/Source/Python/AutoGen/GenMake.py | 8 +- > BaseTools/Source/Python/AutoGen/GenPcdDb.py | 142 ++++++++-------- > BaseTools/Source/Python/AutoGen/GenVar.py | 165 +++++++++---------- > BaseTools/Source/Python/AutoGen/IdfClassObject.py | 1 - > BaseTools/Source/Python/AutoGen/InfSectionParser.py | 1 + > BaseTools/Source/Python/AutoGen/StrGather.py | 5 +- > BaseTools/Source/Python/AutoGen/UniClassObject.py | 18 +- > BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py | 10 +- > BaseTools/Source/Python/BPDG/BPDG.py | 3 +- > BaseTools/Source/Python/BPDG/GenVpd.py | 25 +-- > BaseTools/Source/Python/Common/DataType.py | 4 +- > BaseTools/Source/Python/Common/DecClassObject.py | 39 ++--- > BaseTools/Source/Python/Common/Dictionary.py | 9 +- > BaseTools/Source/Python/Common/DscClassObject.py | 70 ++++---- > BaseTools/Source/Python/Common/EdkIIWorkspace.py | 25 +-- > BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py | 133 +++++++-------- > BaseTools/Source/Python/Common/Expression.py | 81 ++++----- > BaseTools/Source/Python/Common/FdfClassObject.py | 1 + > BaseTools/Source/Python/Common/FdfParserLite.py | 47 +++--- > BaseTools/Source/Python/Common/InfClassObject.py | 113 ++++++------- > BaseTools/Source/Python/Common/LongFilePathOs.py | 2 +- > BaseTools/Source/Python/Common/MigrationUtilities.py | 1 + > BaseTools/Source/Python/Common/Misc.py | 70 ++++---- > BaseTools/Source/Python/Common/Parsing.py | 1 + > BaseTools/Source/Python/Common/RangeExpression.py | 32 ++-- > BaseTools/Source/Python/Common/String.py | 7 +- > BaseTools/Source/Python/Common/TargetTxtClassObject.py | 15 +- > BaseTools/Source/Python/Common/ToolDefClassObject.py | 3 +- > BaseTools/Source/Python/Common/VpdInfoFile.py | 23 +-- > BaseTools/Source/Python/Ecc/CParser.py | 173 ++++++++++---------- > BaseTools/Source/Python/Ecc/Check.py | 1 + > BaseTools/Source/Python/Ecc/CodeFragmentCollector.py | 69 ++++---- > BaseTools/Source/Python/Ecc/Configuration.py | 5 +- > BaseTools/Source/Python/Ecc/Exception.py | 3 +- > BaseTools/Source/Python/Ecc/MetaDataParser.py | 3 +- > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py | 5 +- > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py | 41 ++--- > BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py | 9 +- > BaseTools/Source/Python/Ecc/c.py | 15 +- > BaseTools/Source/Python/Eot/CParser.py | 173 ++++++++++---------- > BaseTools/Source/Python/Eot/CodeFragmentCollector.py | 61 +++---- > BaseTools/Source/Python/Eot/FvImage.py | 17 +- > BaseTools/Source/Python/Eot/InfParserLite.py | 8 +- > BaseTools/Source/Python/Eot/Parser.py | 2 +- > BaseTools/Source/Python/Eot/c.py | 23 +-- > BaseTools/Source/Python/GenFds/AprioriSection.py | 7 +- > BaseTools/Source/Python/GenFds/Capsule.py | 10 +- > BaseTools/Source/Python/GenFds/CapsuleData.py | 6 +- > BaseTools/Source/Python/GenFds/EfiSection.py | 6 +- > BaseTools/Source/Python/GenFds/Fd.py | 12 +- > BaseTools/Source/Python/GenFds/FdfParser.py | 43 ++--- > BaseTools/Source/Python/GenFds/FfsFileStatement.py | 5 +- > BaseTools/Source/Python/GenFds/FfsInfStatement.py | 16 +- > BaseTools/Source/Python/GenFds/Fv.py | 13 +- > BaseTools/Source/Python/GenFds/FvImageSection.py | 8 +- > BaseTools/Source/Python/GenFds/GenFds.py | 20 ++- > BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 10 +- > BaseTools/Source/Python/GenFds/OptionRom.py | 3 - > BaseTools/Source/Python/GenFds/Region.py | 14 +- > BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py | 9 +- > BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py | 1 + > BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py | 32 ++-- > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py | 30 ++-- > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py | 36 ++-- > BaseTools/Source/Python/TargetTool/TargetTool.py | 39 ++--- > BaseTools/Source/Python/Trim/Trim.py | 25 +-- > BaseTools/Source/Python/UPT/Core/DependencyRules.py | 12 +- > BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py | 4 +- > BaseTools/Source/Python/UPT/Core/FileHook.py | 2 +- > BaseTools/Source/Python/UPT/Core/IpiDb.py | 6 +- > BaseTools/Source/Python/UPT/Core/PackageFile.py | 12 +- > BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py | 15 +- > BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py | 42 ++--- > BaseTools/Source/Python/UPT/InstallPkg.py | 2 +- > BaseTools/Source/Python/UPT/InventoryWs.py | 2 +- > BaseTools/Source/Python/UPT/Library/CommentParsing.py | 5 +- > BaseTools/Source/Python/UPT/Library/ExpressionValidate.py | 11 +- > BaseTools/Source/Python/UPT/Library/Misc.py | 11 +- > BaseTools/Source/Python/UPT/Library/ParserValidate.py | 2 +- > BaseTools/Source/Python/UPT/Library/Parsing.py | 3 +- > BaseTools/Source/Python/UPT/Library/String.py | 5 +- > BaseTools/Source/Python/UPT/Library/UniClassObject.py | 20 ++- > BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py | 4 +- > BaseTools/Source/Python/UPT/MkPkg.py | 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py | 6 +- > BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py | 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py | 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py | 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py | 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py | 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py | 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py | 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py | 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py | 3 +- > BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py | 4 +- > BaseTools/Source/Python/UPT/Parser/DecParserMisc.py | 1 + > BaseTools/Source/Python/UPT/Parser/InfSectionParser.py | 3 +- > BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py | 57 +++---- > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py | 3 +- > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py | 3 +- > BaseTools/Source/Python/UPT/ReplacePkg.py | 2 +- > BaseTools/Source/Python/UPT/RmPkg.py | 2 +- > BaseTools/Source/Python/UPT/TestInstall.py | 4 +- > BaseTools/Source/Python/UPT/UPT.py | 9 +- > BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py | 5 +- > BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py | 10 +- > BaseTools/Source/Python/UPT/Xml/CommonXml.py | 2 +- > BaseTools/Source/Python/UPT/Xml/IniToXml.py | 1 + > BaseTools/Source/Python/UPT/Xml/XmlParser.py | 25 +-- > BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py | 3 +- > BaseTools/Source/Python/Workspace/BuildClassObject.py | 2 +- > BaseTools/Source/Python/Workspace/DecBuildData.py | 14 +- > BaseTools/Source/Python/Workspace/DscBuildData.py | 142 ++++++++-------- > BaseTools/Source/Python/Workspace/InfBuildData.py | 3 +- > BaseTools/Source/Python/Workspace/MetaFileParser.py | 74 +++++---- > BaseTools/Source/Python/Workspace/MetaFileTable.py | 10 +- > BaseTools/Source/Python/Workspace/WorkspaceCommon.py | 2 +- > BaseTools/Source/Python/build/BuildReport.py | 17 +- > BaseTools/Source/Python/build/build.py | 35 ++-- > BaseTools/Tests/CheckPythonSyntax.py | 2 +- > BaseTools/Tests/TestTools.py | 13 +- > BaseTools/Tests/TianoCompress.py | 6 +- > BaseTools/gcc/mingw-gcc-build.py | 112 ++++++------- > 136 files changed, 1559 insertions(+), 1477 deletions(-) > > -- > 2.15.1 > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel >