From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=195.135.221.5; helo=smtp.nue.novell.com; envelope-from=glin@suse.com; receiver=edk2-devel@lists.01.org Received: from smtp.nue.novell.com (smtp.nue.novell.com [195.135.221.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id F067A2097F55D for ; Wed, 20 Jun 2018 01:08:36 -0700 (PDT) Received: from emea4-mta.ukb.novell.com ([10.120.13.87]) by smtp.nue.novell.com with ESMTP (TLS encrypted); Wed, 20 Jun 2018 10:08:34 +0200 Received: from GaryWorkstation (nwb-a10-snat.microfocus.com [10.120.13.202]) by emea4-mta.ukb.novell.com with ESMTP (TLS encrypted); Wed, 20 Jun 2018 09:08:25 +0100 Date: Wed, 20 Jun 2018 16:08:20 +0800 From: Gary Lin To: "Zhu, Yonghong" Cc: Paolo Bonzini , "edk2-devel@lists.01.org" , "Gao, Liming" Message-ID: <20180620080820.zsoyncgqtp33uyu5@GaryWorkstation> References: <20180201083608.16036-1-glin@suse.com> MIME-Version: 1.0 In-Reply-To: User-Agent: NeoMutt/20170912 (1.9.0) Subject: Re: [PATCH v2 00/20] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jun 2018 08:08:37 -0000 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Jun 20, 2018 at 07:29:40AM +0000, Zhu, Yonghong wrote: > Hi Paolo and Gary, > > The patches are good and helpful. But it is out of date, so could you help to recreate the patches ? > Sure. I'll re-run the futurize scripts since BaseTools has been changed a lot recently and it might take a while. Cheers, Gary Lin > Best Regards, > Zhu Yonghong > > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Paolo Bonzini > Sent: Wednesday, June 20, 2018 2:22 PM > To: Gary Lin ; edk2-devel@lists.01.org > Cc: Gao, Liming > Subject: Re: [edk2] [PATCH v2 00/20] BaseTools: One step toward python3 > > On 01/02/2018 09:35, Gary Lin wrote: > > v2 changes: > > - Rebase to the current git HEAD (821807bcefb9a36e598d71a8004fae5aab2052a0) > > - Apply "futurize -f libfuturize.fixes.fix_absolute_import" and > > refactor some python scripts to break the circular imports. > > > > This patch series is also available in > > https://github.com/lcp/edk2/tree/python3-futurize-v2 > > > > Since python2 will be EOL in 2020, we start to evaluate the impact of > > the python2 removal. As expected, OMVF building failed the test. It's > > actually a task noted in the wiki page: > > > > https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools- > > Python3-Support > > > > Maybe it's time to convert the python scripts gradully. > > I cannot find any answer to this series. Is there any reason why it wasn't considered? > > Thanks, > > Paolo > > > This patchset doesn't make the python scripts in BaseTools compatible > > with python3 immediately. It aims to do the trivial and safe > > conversion and replacement to make some statements compatible with > > both python2 and python3, so we can deal with the difficult cases later. > > > > With the help of "futurize" from python-future, it's easier to > > refactor the statements. This patchset is basically equivalent to "futurize -1" > > plus "StringIO.StringIO => io.BytesIO". > > > > For the "io.BytesIO" change, it MIGHT introduce slow down to the build > > time since io.BytesIO is slower than StringIO.StringIO in python2(*). > > For a quick test, I built OVMF with the following command based on > > 8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f: > > > > $ rm -rf Build && make -C BaseTools/ clean $ time ./OvmfPkg/build.sh > > -D SECURE_BOOT_ENABLE \ > > -D NETWORK_IP6_ENABLE \ > > -D HTTP_BOOT_ENABLE \ > > -D TLS_ENABLE > > > > Before io.BytesIO: > > > > Build total time: 00:03:56 > > real 4m22.991s > > user 3m55.874s > > sys 0m27.250s > > > > After io.BytesIO: > > > > Build total time: 00:03:57 > > real 4m23.953s > > user 3m57.526s > > sys 0m27.192s > > > > The difference is only 1 second, and I would say the impact is subtle. > > > > The next step will be fixing relative import and maybe applying more > > futurize fixes. We won't get there soon but at least we are moving... > > > > (*) > > https://stackoverflow.com/questions/37462075/confusing-about-stringio- > > cstringio-and-byteio > > > > Contributed-under: TianoCore Contribution Agreement 1.1 > > Cc: Yonghong Zhu > > Cc: Liming Gao > > Signed-off-by: Gary Lin > > > > > > Gary Lin (20): > > BaseTools: Refactor python except statements > > BaseTools: Refactor python print statements > > BaseTools: Remove the old python "not-equal" > > BaseTools: Use the python3-range functions > > BaseTools: Remove tuple parameter in python scripts > > BaseTools: Remove the deprecated hash_key() > > BaseTools: Import reduce() from functools > > BaseTools: Replace StandardError with Expression > > BaseTools: Remove types.TypeType > > BaseTools: Refactor python raise statement > > BaseTools: Adjust the spaces around commas and colons > > BaseTools: Migrate to the new octal literal > > BaseTools: Unify long int and int in python scripts > > BaseTools: Adjust old python2 idioms > > BaseTools: Replace StringIO.StringIO with io.BytesIO > > BaseTools: Treat GenFds.py and build.py as python modules > > BaseTools: Adopt absolute import for python scripts > > BaseTools: Move OverrideAttribs to OptRomInfStatement.py > > BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py > > BaseTools: Move ImageBinDict to GenFdsGlobalVariable.py > > > > BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py | 5 +- > > BaseTools/BinWrappers/PosixLike/GenFds | 2 +- > > BaseTools/BinWrappers/PosixLike/build | 2 +- > > BaseTools/Scripts/BinToPcd.py | 46 +++-- > > BaseTools/Scripts/ConvertMasmToNasm.py | 1 + > > BaseTools/Scripts/ConvertUni.py | 5 - > > BaseTools/Scripts/MemoryProfileSymbolGen.py | 22 +- > > BaseTools/Scripts/PatchCheck.py | 7 +- > > BaseTools/Scripts/RunMakefile.py | 2 +- > > BaseTools/Scripts/SmiHandlerProfileSymbolGen.py | 20 +- > > BaseTools/Scripts/UpdateBuildVersions.py | 18 +- > > BaseTools/Source/Python/AutoGen/AutoGen.py | 98 ++++----- > > BaseTools/Source/Python/AutoGen/BuildEngine.py | 38 ++-- > > BaseTools/Source/Python/AutoGen/GenC.py | 12 +- > > BaseTools/Source/Python/AutoGen/GenDepex.py | 8 +- > > BaseTools/Source/Python/AutoGen/GenMake.py | 11 +- > > BaseTools/Source/Python/AutoGen/GenPcdDb.py | 149 +++++++------- > > BaseTools/Source/Python/AutoGen/GenVar.py | 166 +++++++-------- > > BaseTools/Source/Python/AutoGen/IdfClassObject.py | 1 - > > BaseTools/Source/Python/AutoGen/InfSectionParser.py | 1 + > > BaseTools/Source/Python/AutoGen/StrGather.py | 8 +- > > BaseTools/Source/Python/AutoGen/UniClassObject.py | 18 +- > > BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py | 10 +- > > BaseTools/Source/Python/BPDG/BPDG.py | 8 +- > > BaseTools/Source/Python/BPDG/GenVpd.py | 28 +-- > > BaseTools/Source/Python/Common/DataType.py | 4 +- > > BaseTools/Source/Python/Common/Database.py | 8 +- > > BaseTools/Source/Python/Common/DecClassObject.py | 56 ++--- > > BaseTools/Source/Python/Common/Dictionary.py | 14 +- > > BaseTools/Source/Python/Common/DscClassObject.py | 91 +++++---- > > BaseTools/Source/Python/Common/EdkIIWorkspace.py | 28 +-- > > BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py | 152 +++++++------- > > BaseTools/Source/Python/Common/EdkLogger.py | 3 +- > > BaseTools/Source/Python/Common/Expression.py | 86 ++++---- > > BaseTools/Source/Python/Common/FdfClassObject.py | 6 +- > > BaseTools/Source/Python/Common/FdfParserLite.py | 47 ++--- > > BaseTools/Source/Python/Common/InfClassObject.py | 134 ++++++------ > > BaseTools/Source/Python/Common/LongFilePathOs.py | 5 +- > > BaseTools/Source/Python/Common/MigrationUtilities.py | 4 +- > > BaseTools/Source/Python/Common/Misc.py | 79 ++++---- > > BaseTools/Source/Python/Common/Parsing.py | 6 +- > > BaseTools/Source/Python/Common/RangeExpression.py | 32 +-- > > BaseTools/Source/Python/Common/String.py | 16 +- > > BaseTools/Source/Python/Common/TargetTxtClassObject.py | 24 ++- > > BaseTools/Source/Python/Common/ToolDefClassObject.py | 12 +- > > BaseTools/Source/Python/Common/VpdInfoFile.py | 23 ++- > > BaseTools/Source/Python/CommonDataClass/ModuleClass.py | 3 +- > > BaseTools/Source/Python/CommonDataClass/PackageClass.py | 3 +- > > BaseTools/Source/Python/CommonDataClass/PlatformClass.py | 3 +- > > BaseTools/Source/Python/Ecc/CParser.py | 178 ++++++++-------- > > BaseTools/Source/Python/Ecc/Check.py | 10 +- > > BaseTools/Source/Python/Ecc/CodeFragmentCollector.py | 82 ++++---- > > BaseTools/Source/Python/Ecc/Configuration.py | 5 +- > > BaseTools/Source/Python/Ecc/Database.py | 7 +- > > BaseTools/Source/Python/Ecc/Ecc.py | 25 +-- > > BaseTools/Source/Python/Ecc/Exception.py | 6 +- > > BaseTools/Source/Python/Ecc/FileProfile.py | 5 +- > > BaseTools/Source/Python/Ecc/MetaDataParser.py | 8 +- > > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py | 5 +- > > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py | 44 ++-- > > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileTable.py | 5 +- > > BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py | 9 +- > > BaseTools/Source/Python/Ecc/c.py | 28 +-- > > BaseTools/Source/Python/Eot/CParser.py | 178 ++++++++-------- > > BaseTools/Source/Python/Eot/CodeFragmentCollector.py | 72 +++---- > > BaseTools/Source/Python/Eot/Eot.py | 15 +- > > BaseTools/Source/Python/Eot/FileProfile.py | 3 +- > > BaseTools/Source/Python/Eot/FvImage.py | 28 +-- > > BaseTools/Source/Python/Eot/InfParserLite.py | 13 +- > > BaseTools/Source/Python/Eot/Parser.py | 5 +- > > BaseTools/Source/Python/Eot/Report.py | 3 +- > > BaseTools/Source/Python/Eot/c.py | 32 +-- > > BaseTools/Source/Python/GenFds/AprioriSection.py | 12 +- > > BaseTools/Source/Python/GenFds/Capsule.py | 22 +- > > BaseTools/Source/Python/GenFds/CapsuleData.py | 11 +- > > BaseTools/Source/Python/GenFds/CompressSection.py | 7 +- > > BaseTools/Source/Python/GenFds/DataSection.py | 7 +- > > BaseTools/Source/Python/GenFds/DepexSection.py | 7 +- > > BaseTools/Source/Python/GenFds/EfiSection.py | 13 +- > > BaseTools/Source/Python/GenFds/Fd.py | 32 +-- > > BaseTools/Source/Python/GenFds/FdfParser.py | 100 ++++----- > > BaseTools/Source/Python/GenFds/FfsFileStatement.py | 16 +- > > BaseTools/Source/Python/GenFds/FfsInfStatement.py | 35 ++-- > > BaseTools/Source/Python/GenFds/Fv.py | 34 ++-- > > BaseTools/Source/Python/GenFds/FvImageSection.py | 15 +- > > BaseTools/Source/Python/GenFds/GenFds.py | 126 ++---------- > > BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | 108 +++++++++- > > BaseTools/Source/Python/GenFds/GuidSection.py | 11 +- > > BaseTools/Source/Python/GenFds/OptRomFileStatement.py | 3 +- > > BaseTools/Source/Python/GenFds/OptRomInfStatement.py | 30 ++- > > BaseTools/Source/Python/GenFds/OptionRom.py | 23 +-- > > BaseTools/Source/Python/GenFds/Region.py | 17 +- > > BaseTools/Source/Python/GenFds/RuleComplexFile.py | 3 +- > > BaseTools/Source/Python/GenFds/RuleSimpleFile.py | 3 +- > > BaseTools/Source/Python/GenFds/Section.py | 3 +- > > BaseTools/Source/Python/GenFds/UiSection.py | 7 +- > > BaseTools/Source/Python/GenFds/VerSection.py | 7 +- > > BaseTools/Source/Python/GenFds/Vtf.py | 3 +- > > BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py | 9 +- > > BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py | 1 + > > BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py | 32 +-- > > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py | 30 +-- > > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py | 36 ++-- > > BaseTools/Source/Python/Table/TableDataModel.py | 3 +- > > BaseTools/Source/Python/Table/TableDec.py | 3 +- > > BaseTools/Source/Python/Table/TableDsc.py | 3 +- > > BaseTools/Source/Python/Table/TableEotReport.py | 5 +- > > BaseTools/Source/Python/Table/TableFdf.py | 3 +- > > BaseTools/Source/Python/Table/TableFile.py | 3 +- > > BaseTools/Source/Python/Table/TableFunction.py | 3 +- > > BaseTools/Source/Python/Table/TableIdentifier.py | 5 +- > > BaseTools/Source/Python/Table/TableInf.py | 3 +- > > BaseTools/Source/Python/Table/TablePcd.py | 5 +- > > BaseTools/Source/Python/Table/TableQuery.py | 3 +- > > BaseTools/Source/Python/Table/TableReport.py | 3 +- > > BaseTools/Source/Python/TargetTool/TargetTool.py | 39 ++-- > > BaseTools/Source/Python/Trim/Trim.py | 25 +-- > > BaseTools/Source/Python/UPT/Core/DependencyRules.py | 12 +- > > BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py | 4 +- > > BaseTools/Source/Python/UPT/Core/FileHook.py | 2 +- > > BaseTools/Source/Python/UPT/Core/IpiDb.py | 6 +- > > BaseTools/Source/Python/UPT/Core/PackageFile.py | 12 +- > > BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py | 15 +- > > BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py | 42 ++-- > > BaseTools/Source/Python/UPT/InstallPkg.py | 2 +- > > BaseTools/Source/Python/UPT/InventoryWs.py | 2 +- > > BaseTools/Source/Python/UPT/Library/CommentParsing.py | 5 +- > > BaseTools/Source/Python/UPT/Library/ExpressionValidate.py | 11 +- > > BaseTools/Source/Python/UPT/Library/Misc.py | 11 +- > > BaseTools/Source/Python/UPT/Library/ParserValidate.py | 2 +- > > BaseTools/Source/Python/UPT/Library/Parsing.py | 6 +- > > BaseTools/Source/Python/UPT/Library/String.py | 5 +- > > BaseTools/Source/Python/UPT/Library/UniClassObject.py | 20 +- > > BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py | 4 +- > > BaseTools/Source/Python/UPT/MkPkg.py | 2 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py | 6 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py | 2 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py | 4 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py | 2 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py | 4 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py | 4 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py | 4 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py | 4 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py | 2 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py | 3 +- > > BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py | 4 +- > > BaseTools/Source/Python/UPT/Parser/DecParserMisc.py | 1 + > > BaseTools/Source/Python/UPT/Parser/InfSectionParser.py | 3 +- > > BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py | 57 +++--- > > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py | 3 +- > > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py | 3 +- > > BaseTools/Source/Python/UPT/ReplacePkg.py | 2 +- > > BaseTools/Source/Python/UPT/RmPkg.py | 2 +- > > BaseTools/Source/Python/UPT/TestInstall.py | 4 +- > > BaseTools/Source/Python/UPT/UPT.py | 9 +- > > BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py | 5 +- > > BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py | 10 +- > > BaseTools/Source/Python/UPT/Xml/CommonXml.py | 2 +- > > BaseTools/Source/Python/UPT/Xml/IniToXml.py | 1 + > > BaseTools/Source/Python/UPT/Xml/XmlParser.py | 25 +-- > > BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py | 3 +- > > BaseTools/Source/Python/Workspace/BuildClassObject.py | 2 +- > > BaseTools/Source/Python/Workspace/DecBuildData.py | 14 +- > > BaseTools/Source/Python/Workspace/DscBuildData.py | 213 ++++++++++---------- > > BaseTools/Source/Python/Workspace/InfBuildData.py | 6 +- > > BaseTools/Source/Python/Workspace/MetaFileParser.py | 75 +++---- > > BaseTools/Source/Python/Workspace/MetaFileTable.py | 15 +- > > BaseTools/Source/Python/Workspace/WorkspaceCommon.py | 5 +- > > BaseTools/Source/Python/Workspace/WorkspaceDatabase.py | 7 +- > > BaseTools/Source/Python/build/BuildReport.py | 19 +- > > BaseTools/Source/Python/build/build.py | 38 ++-- > > BaseTools/Tests/CheckPythonSyntax.py | 2 +- > > BaseTools/Tests/TestTools.py | 13 +- > > BaseTools/Tests/TianoCompress.py | 6 +- > > BaseTools/gcc/mingw-gcc-build.py | 112 +++++----- > > 175 files changed, 2092 insertions(+), 1927 deletions(-) > > > > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel >