From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.43; helo=mga05.intel.com; envelope-from=yonghong.zhu@intel.com; receiver=edk2-devel@lists.01.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id BBA4A2119BAC3 for ; Wed, 20 Jun 2018 00:29:49 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jun 2018 00:29:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,246,1526367600"; d="scan'208";a="58746283" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by FMSMGA003.fm.intel.com with ESMTP; 20 Jun 2018 00:29:48 -0700 Received: from fmsmsx151.amr.corp.intel.com (10.18.125.4) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 20 Jun 2018 00:29:48 -0700 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX151.amr.corp.intel.com (10.18.125.4) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 20 Jun 2018 00:29:42 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.51]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.116]) with mapi id 14.03.0319.002; Wed, 20 Jun 2018 15:29:40 +0800 From: "Zhu, Yonghong" To: Paolo Bonzini , Gary Lin , "edk2-devel@lists.01.org" CC: "Gao, Liming" , "Zhu, Yonghong" Thread-Topic: [edk2] [PATCH v2 00/20] BaseTools: One step toward python3 Thread-Index: AQHUCF8YL+51TbCWzE+opQobVAtgJKRouQcA Date: Wed, 20 Jun 2018 07:29:40 +0000 Message-ID: References: <20180201083608.16036-1-glin@suse.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYzgwNjAyYTMtZTRhNy00Zjg3LTg4NmMtZGJlYmMwMmFjODVjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoielBJRjZmRU5WUVljOGNaeXg3YmxIdXRIZ09XblZscGYra0dMdGlQTkUxS2k3cUtoRGJTK2ZoVXljZmVBUEVzVCJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.200.100 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH v2 00/20] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jun 2018 07:29:50 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Paolo and Gary, The patches are good and helpful. But it is out of date, so could you help = to recreate the patches ? Best Regards, Zhu Yonghong=20 -----Original Message----- From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Paol= o Bonzini Sent: Wednesday, June 20, 2018 2:22 PM To: Gary Lin ; edk2-devel@lists.01.org Cc: Gao, Liming Subject: Re: [edk2] [PATCH v2 00/20] BaseTools: One step toward python3 On 01/02/2018 09:35, Gary Lin wrote: > v2 changes: > - Rebase to the current git HEAD (821807bcefb9a36e598d71a8004fae5aab205= 2a0) > - Apply "futurize -f libfuturize.fixes.fix_absolute_import" and > refactor some python scripts to break the circular imports. >=20 > This patch series is also available in > https://github.com/lcp/edk2/tree/python3-futurize-v2 >=20 > Since python2 will be EOL in 2020, we start to evaluate the impact of=20 > the python2 removal. As expected, OMVF building failed the test. It's=20 > actually a task noted in the wiki page: >=20 > https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools- > Python3-Support >=20 > Maybe it's time to convert the python scripts gradully. I cannot find any answer to this series. Is there any reason why it wasn't= considered? Thanks, Paolo > This patchset doesn't make the python scripts in BaseTools compatible=20 > with python3 immediately. It aims to do the trivial and safe=20 > conversion and replacement to make some statements compatible with=20 > both python2 and python3, so we can deal with the difficult cases later. >=20 > With the help of "futurize" from python-future, it's easier to=20 > refactor the statements. This patchset is basically equivalent to "futuri= ze -1" > plus "StringIO.StringIO =3D> io.BytesIO". >=20 > For the "io.BytesIO" change, it MIGHT introduce slow down to the build=20 > time since io.BytesIO is slower than StringIO.StringIO in python2(*). > For a quick test, I built OVMF with the following command based on > 8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f: >=20 > $ rm -rf Build && make -C BaseTools/ clean $ time ./OvmfPkg/build.sh=20 > -D SECURE_BOOT_ENABLE \ > -D NETWORK_IP6_ENABLE \ > -D HTTP_BOOT_ENABLE \ > -D TLS_ENABLE >=20 > Before io.BytesIO: >=20 > Build total time: 00:03:56 > real 4m22.991s > user 3m55.874s > sys 0m27.250s >=20 > After io.BytesIO: >=20 > Build total time: 00:03:57 > real 4m23.953s > user 3m57.526s > sys 0m27.192s >=20 > The difference is only 1 second, and I would say the impact is subtle.=20 >=20 > The next step will be fixing relative import and maybe applying more=20 > futurize fixes. We won't get there soon but at least we are moving... >=20 > (*)=20 > https://stackoverflow.com/questions/37462075/confusing-about-stringio- > cstringio-and-byteio >=20 > Contributed-under: TianoCore Contribution Agreement 1.1 > Cc: Yonghong Zhu > Cc: Liming Gao > Signed-off-by: Gary Lin >=20 >=20 > Gary Lin (20): > BaseTools: Refactor python except statements > BaseTools: Refactor python print statements > BaseTools: Remove the old python "not-equal" > BaseTools: Use the python3-range functions > BaseTools: Remove tuple parameter in python scripts > BaseTools: Remove the deprecated hash_key() > BaseTools: Import reduce() from functools > BaseTools: Replace StandardError with Expression > BaseTools: Remove types.TypeType > BaseTools: Refactor python raise statement > BaseTools: Adjust the spaces around commas and colons > BaseTools: Migrate to the new octal literal > BaseTools: Unify long int and int in python scripts > BaseTools: Adjust old python2 idioms > BaseTools: Replace StringIO.StringIO with io.BytesIO > BaseTools: Treat GenFds.py and build.py as python modules > BaseTools: Adopt absolute import for python scripts > BaseTools: Move OverrideAttribs to OptRomInfStatement.py > BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py > BaseTools: Move ImageBinDict to GenFdsGlobalVariable.py >=20 > BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py |= 5 +- > BaseTools/BinWrappers/PosixLike/GenFds |= 2 +- > BaseTools/BinWrappers/PosixLike/build |= 2 +- > BaseTools/Scripts/BinToPcd.py |= 46 +++-- > BaseTools/Scripts/ConvertMasmToNasm.py |= 1 + > BaseTools/Scripts/ConvertUni.py |= 5 - > BaseTools/Scripts/MemoryProfileSymbolGen.py |= 22 +- > BaseTools/Scripts/PatchCheck.py |= 7 +- > BaseTools/Scripts/RunMakefile.py |= 2 +- > BaseTools/Scripts/SmiHandlerProfileSymbolGen.py |= 20 +- > BaseTools/Scripts/UpdateBuildVersions.py |= 18 +- > BaseTools/Source/Python/AutoGen/AutoGen.py |= 98 ++++----- > BaseTools/Source/Python/AutoGen/BuildEngine.py |= 38 ++-- > BaseTools/Source/Python/AutoGen/GenC.py |= 12 +- > BaseTools/Source/Python/AutoGen/GenDepex.py |= 8 +- > BaseTools/Source/Python/AutoGen/GenMake.py |= 11 +- > BaseTools/Source/Python/AutoGen/GenPcdDb.py |= 149 +++++++------- > BaseTools/Source/Python/AutoGen/GenVar.py |= 166 +++++++-------- > BaseTools/Source/Python/AutoGen/IdfClassObject.py |= 1 - > BaseTools/Source/Python/AutoGen/InfSectionParser.py |= 1 + > BaseTools/Source/Python/AutoGen/StrGather.py |= 8 +- > BaseTools/Source/Python/AutoGen/UniClassObject.py |= 18 +- > BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py |= 10 +- > BaseTools/Source/Python/BPDG/BPDG.py |= 8 +- > BaseTools/Source/Python/BPDG/GenVpd.py |= 28 +-- > BaseTools/Source/Python/Common/DataType.py |= 4 +- > BaseTools/Source/Python/Common/Database.py |= 8 +- > BaseTools/Source/Python/Common/DecClassObject.py |= 56 ++--- > BaseTools/Source/Python/Common/Dictionary.py |= 14 +- > BaseTools/Source/Python/Common/DscClassObject.py |= 91 +++++---- > BaseTools/Source/Python/Common/EdkIIWorkspace.py |= 28 +-- > BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py |= 152 +++++++------- > BaseTools/Source/Python/Common/EdkLogger.py |= 3 +- > BaseTools/Source/Python/Common/Expression.py |= 86 ++++---- > BaseTools/Source/Python/Common/FdfClassObject.py |= 6 +- > BaseTools/Source/Python/Common/FdfParserLite.py |= 47 ++--- > BaseTools/Source/Python/Common/InfClassObject.py |= 134 ++++++------ > BaseTools/Source/Python/Common/LongFilePathOs.py |= 5 +- > BaseTools/Source/Python/Common/MigrationUtilities.py |= 4 +- > BaseTools/Source/Python/Common/Misc.py |= 79 ++++---- > BaseTools/Source/Python/Common/Parsing.py |= 6 +- > BaseTools/Source/Python/Common/RangeExpression.py |= 32 +-- > BaseTools/Source/Python/Common/String.py |= 16 +- > BaseTools/Source/Python/Common/TargetTxtClassObject.py |= 24 ++- > BaseTools/Source/Python/Common/ToolDefClassObject.py |= 12 +- > BaseTools/Source/Python/Common/VpdInfoFile.py |= 23 ++- > BaseTools/Source/Python/CommonDataClass/ModuleClass.py |= 3 +- > BaseTools/Source/Python/CommonDataClass/PackageClass.py |= 3 +- > BaseTools/Source/Python/CommonDataClass/PlatformClass.py |= 3 +- > BaseTools/Source/Python/Ecc/CParser.py |= 178 ++++++++-------- > BaseTools/Source/Python/Ecc/Check.py |= 10 +- > BaseTools/Source/Python/Ecc/CodeFragmentCollector.py |= 82 ++++---- > BaseTools/Source/Python/Ecc/Configuration.py |= 5 +- > BaseTools/Source/Python/Ecc/Database.py |= 7 +- > BaseTools/Source/Python/Ecc/Ecc.py |= 25 +-- > BaseTools/Source/Python/Ecc/Exception.py |= 6 +- > BaseTools/Source/Python/Ecc/FileProfile.py |= 5 +- > BaseTools/Source/Python/Ecc/MetaDataParser.py |= 8 +- > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py |= 5 +- > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py |= 44 ++-- > BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileTable.py |= 5 +- > BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py |= 9 +- > BaseTools/Source/Python/Ecc/c.py |= 28 +-- > BaseTools/Source/Python/Eot/CParser.py |= 178 ++++++++-------- > BaseTools/Source/Python/Eot/CodeFragmentCollector.py |= 72 +++---- > BaseTools/Source/Python/Eot/Eot.py |= 15 +- > BaseTools/Source/Python/Eot/FileProfile.py |= 3 +- > BaseTools/Source/Python/Eot/FvImage.py |= 28 +-- > BaseTools/Source/Python/Eot/InfParserLite.py |= 13 +- > BaseTools/Source/Python/Eot/Parser.py |= 5 +- > BaseTools/Source/Python/Eot/Report.py |= 3 +- > BaseTools/Source/Python/Eot/c.py |= 32 +-- > BaseTools/Source/Python/GenFds/AprioriSection.py |= 12 +- > BaseTools/Source/Python/GenFds/Capsule.py |= 22 +- > BaseTools/Source/Python/GenFds/CapsuleData.py |= 11 +- > BaseTools/Source/Python/GenFds/CompressSection.py |= 7 +- > BaseTools/Source/Python/GenFds/DataSection.py |= 7 +- > BaseTools/Source/Python/GenFds/DepexSection.py |= 7 +- > BaseTools/Source/Python/GenFds/EfiSection.py |= 13 +- > BaseTools/Source/Python/GenFds/Fd.py |= 32 +-- > BaseTools/Source/Python/GenFds/FdfParser.py |= 100 ++++----- > BaseTools/Source/Python/GenFds/FfsFileStatement.py |= 16 +- > BaseTools/Source/Python/GenFds/FfsInfStatement.py |= 35 ++-- > BaseTools/Source/Python/GenFds/Fv.py |= 34 ++-- > BaseTools/Source/Python/GenFds/FvImageSection.py |= 15 +- > BaseTools/Source/Python/GenFds/GenFds.py |= 126 ++---------- > BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py |= 108 +++++++++- > BaseTools/Source/Python/GenFds/GuidSection.py |= 11 +- > BaseTools/Source/Python/GenFds/OptRomFileStatement.py |= 3 +- > BaseTools/Source/Python/GenFds/OptRomInfStatement.py |= 30 ++- > BaseTools/Source/Python/GenFds/OptionRom.py |= 23 +-- > BaseTools/Source/Python/GenFds/Region.py |= 17 +- > BaseTools/Source/Python/GenFds/RuleComplexFile.py |= 3 +- > BaseTools/Source/Python/GenFds/RuleSimpleFile.py |= 3 +- > BaseTools/Source/Python/GenFds/Section.py |= 3 +- > BaseTools/Source/Python/GenFds/UiSection.py |= 7 +- > BaseTools/Source/Python/GenFds/VerSection.py |= 7 +- > BaseTools/Source/Python/GenFds/Vtf.py |= 3 +- > BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py |= 9 +- > BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py |= 1 + > BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py |= 32 +-- > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py |= 30 +-- > BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py |= 36 ++-- > BaseTools/Source/Python/Table/TableDataModel.py |= 3 +- > BaseTools/Source/Python/Table/TableDec.py |= 3 +- > BaseTools/Source/Python/Table/TableDsc.py |= 3 +- > BaseTools/Source/Python/Table/TableEotReport.py |= 5 +- > BaseTools/Source/Python/Table/TableFdf.py |= 3 +- > BaseTools/Source/Python/Table/TableFile.py |= 3 +- > BaseTools/Source/Python/Table/TableFunction.py |= 3 +- > BaseTools/Source/Python/Table/TableIdentifier.py |= 5 +- > BaseTools/Source/Python/Table/TableInf.py |= 3 +- > BaseTools/Source/Python/Table/TablePcd.py |= 5 +- > BaseTools/Source/Python/Table/TableQuery.py |= 3 +- > BaseTools/Source/Python/Table/TableReport.py |= 3 +- > BaseTools/Source/Python/TargetTool/TargetTool.py |= 39 ++-- > BaseTools/Source/Python/Trim/Trim.py |= 25 +-- > BaseTools/Source/Python/UPT/Core/DependencyRules.py |= 12 +- > BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py |= 4 +- > BaseTools/Source/Python/UPT/Core/FileHook.py |= 2 +- > BaseTools/Source/Python/UPT/Core/IpiDb.py |= 6 +- > BaseTools/Source/Python/UPT/Core/PackageFile.py |= 12 +- > BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py |= 15 +- > BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py |= 42 ++-- > BaseTools/Source/Python/UPT/InstallPkg.py |= 2 +- > BaseTools/Source/Python/UPT/InventoryWs.py |= 2 +- > BaseTools/Source/Python/UPT/Library/CommentParsing.py |= 5 +- > BaseTools/Source/Python/UPT/Library/ExpressionValidate.py |= 11 +- > BaseTools/Source/Python/UPT/Library/Misc.py |= 11 +- > BaseTools/Source/Python/UPT/Library/ParserValidate.py |= 2 +- > BaseTools/Source/Python/UPT/Library/Parsing.py |= 6 +- > BaseTools/Source/Python/UPT/Library/String.py |= 5 +- > BaseTools/Source/Python/UPT/Library/UniClassObject.py |= 20 +- > BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py |= 4 +- > BaseTools/Source/Python/UPT/MkPkg.py |= 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py |= 6 +- > BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py |= 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py |= 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py |= 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py |= 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py |= 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py |= 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py |= 4 +- > BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py |= 2 +- > BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py |= 3 +- > BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py |= 4 +- > BaseTools/Source/Python/UPT/Parser/DecParserMisc.py |= 1 + > BaseTools/Source/Python/UPT/Parser/InfSectionParser.py |= 3 +- > BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py |= 57 +++--- > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py |= 3 +- > BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py |= 3 +- > BaseTools/Source/Python/UPT/ReplacePkg.py |= 2 +- > BaseTools/Source/Python/UPT/RmPkg.py |= 2 +- > BaseTools/Source/Python/UPT/TestInstall.py |= 4 +- > BaseTools/Source/Python/UPT/UPT.py |= 9 +- > BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py |= 5 +- > BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py |= 10 +- > BaseTools/Source/Python/UPT/Xml/CommonXml.py |= 2 +- > BaseTools/Source/Python/UPT/Xml/IniToXml.py |= 1 + > BaseTools/Source/Python/UPT/Xml/XmlParser.py |= 25 +-- > BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py |= 3 +- > BaseTools/Source/Python/Workspace/BuildClassObject.py |= 2 +- > BaseTools/Source/Python/Workspace/DecBuildData.py |= 14 +- > BaseTools/Source/Python/Workspace/DscBuildData.py |= 213 ++++++++++---------- > BaseTools/Source/Python/Workspace/InfBuildData.py |= 6 +- > BaseTools/Source/Python/Workspace/MetaFileParser.py |= 75 +++---- > BaseTools/Source/Python/Workspace/MetaFileTable.py |= 15 +- > BaseTools/Source/Python/Workspace/WorkspaceCommon.py |= 5 +- > BaseTools/Source/Python/Workspace/WorkspaceDatabase.py |= 7 +- > BaseTools/Source/Python/build/BuildReport.py |= 19 +- > BaseTools/Source/Python/build/build.py |= 38 ++-- > BaseTools/Tests/CheckPythonSyntax.py |= 2 +- > BaseTools/Tests/TestTools.py |= 13 +- > BaseTools/Tests/TianoCompress.py |= 6 +- > BaseTools/gcc/mingw-gcc-build.py |= 112 +++++----- > 175 files changed, 2092 insertions(+), 1927 deletions(-) >=20 _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel