From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.43; helo=mga05.intel.com; envelope-from=yonghong.zhu@intel.com; receiver=edk2-devel@lists.01.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 06ED221E25688 for ; Thu, 25 Jan 2018 05:32:13 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Jan 2018 05:37:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,412,1511856000"; d="scan'208";a="198585512" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga005.fm.intel.com with ESMTP; 25 Jan 2018 05:37:42 -0800 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 25 Jan 2018 05:37:42 -0800 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by FMSMSX112.amr.corp.intel.com (10.18.116.6) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 25 Jan 2018 05:37:42 -0800 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.213]) by shsmsx102.ccr.corp.intel.com ([169.254.2.189]) with mapi id 14.03.0319.002; Thu, 25 Jan 2018 21:37:40 +0800 From: "Zhu, Yonghong" To: Gary Lin , "edk2-devel@lists.01.org" CC: "Gao, Liming" , "Zhu, Yonghong" Thread-Topic: [PATCH 00/15] BaseTools: One step toward python3 Thread-Index: AQHTkOAWHlLL4wwU9kqhIQHcdKJHbKOEoRfA Date: Thu, 25 Jan 2018 13:37:39 +0000 Message-ID: References: <20180119044316.4713-1-glin@suse.com> In-Reply-To: <20180119044316.4713-1-glin@suse.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH 00/15] BaseTools: One step toward python3 X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jan 2018 13:32:14 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Gary, Thanks for your patches. I am still in evaluating these changes and do some= verification for it. I still need some more days to give you comment. Thanks. Best Regards, Zhu Yonghong -----Original Message----- From: Gary Lin [mailto:glin@suse.com]=20 Sent: Friday, January 19, 2018 12:43 PM To: edk2-devel@lists.01.org Cc: Zhu, Yonghong ; Gao, Liming Subject: [PATCH 00/15] BaseTools: One step toward python3 Since python2 will be EOF in 2020, we start to evaluate the impact of the p= ython2 removal. As expected, OMVF building failed the test. It's actually a= task noted in the wiki page: https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Pytho= n3-Support Maybe it's time to convert the python scripts gradully. This patchset doesn't make the python scripts in BaseTools compatible with = python3 immediately. It aims to do the trivial and safe conversion and repl= acement to make some statements compatible with both python2 and python3, s= o we can deal with the difficult cases later. With the help of "futurize" from python-future, it's easier to refactor the= statements. This patchset is basically equivalent to "futurize -1" plus "StringIO.StringIO =3D> io.BytesIO" and minus "fix_absolute_import". The reason to skip "fix_absolute_import" is that python2 failed to find som= e modules after converting to absolute import, and it might take time to fi= gure out a proper fix. For the "io.BytesIO" change, it MIGHT introduce slow down to the build time= since io.BytesIO is slower than StringIO.StringIO in python2(*). For a quick test, I built OVMF with the following command based on 8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f: $ rm -rf Build && make -C BaseTools/ clean $ time ./OvmfPkg/build.sh -D SEC= URE_BOOT_ENABLE \ -D NETWORK_IP6_ENABLE \ -D HTTP_BOOT_ENABLE \ -D TLS_ENABLE Before io.BytesIO: Build total time: 00:03:56 real 4m22.991s user 3m55.874s sys 0m27.250s After io.BytesIO: Build total time: 00:03:57 real 4m23.953s user 3m57.526s sys 0m27.192s The difference is only 1 second, and I would say the impact is subtle.=20 The next step will be fixing relative import and maybe applying more futuri= ze fixes. We won't get there soon but at least we are moving...=20 (*) https://stackoverflow.com/questions/37462075/confusing-about-stringio-c= stringio-and-byteio Contributed-under: TianoCore Contribution Agreement 1.1 Cc: Yonghong Zhu Cc: Liming Gao Signed-off-by: Gary Lin Gary Lin (15): BaseTools: Refactor python except statements BaseTools: Refactor python print statements BaseTools: Remove the old python "not-equal" BaseTools: Use the python3-range functions BaseTools: Remove tuple parameter in python scripts BaseTools: Remove the deprecated hash_key() BaseTools: Import reduce() from functools BaseTools: Replace StandardError with Expression BaseTools: Remove types.TypeType BaseTools: Refactor python raise statement BaseTools: Adjust the spaces around commas and colons BaseTools: Migrate to the new octal literal BaseTools: Unify long int and int in python scripts BaseTools: Adjust old python2 idioms BaseTools: Replace StringIO.StringIO with io.BytesIO BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py | = 5 +- BaseTools/Scripts/BinToPcd.py | = 46 +++--- BaseTools/Scripts/ConvertMasmToNasm.py | = 1 + BaseTools/Scripts/ConvertUni.py | = 5 - BaseTools/Scripts/MemoryProfileSymbolGen.py | = 22 +-- BaseTools/Scripts/PatchCheck.py | = 7 +- BaseTools/Scripts/RunMakefile.py | = 2 +- BaseTools/Scripts/SmiHandlerProfileSymbolGen.py | = 20 +-- BaseTools/Scripts/UpdateBuildVersions.py | = 18 +- BaseTools/Source/Python/AutoGen/AutoGen.py | = 91 +++++----- BaseTools/Source/Python/AutoGen/BuildEngine.py | = 38 +++-- BaseTools/Source/Python/AutoGen/GenC.py | = 5 +- BaseTools/Source/Python/AutoGen/GenDepex.py | = 8 +- BaseTools/Source/Python/AutoGen/GenMake.py | = 8 +- BaseTools/Source/Python/AutoGen/GenPcdDb.py | 1= 42 ++++++++-------- BaseTools/Source/Python/AutoGen/GenVar.py | 1= 65 +++++++++---------- BaseTools/Source/Python/AutoGen/IdfClassObject.py | = 1 - BaseTools/Source/Python/AutoGen/InfSectionParser.py | = 1 + BaseTools/Source/Python/AutoGen/StrGather.py | = 5 +- BaseTools/Source/Python/AutoGen/UniClassObject.py | = 18 +- BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py | = 10 +- BaseTools/Source/Python/BPDG/BPDG.py | = 3 +- BaseTools/Source/Python/BPDG/GenVpd.py | = 25 +-- BaseTools/Source/Python/Common/DataType.py | = 4 +- BaseTools/Source/Python/Common/DecClassObject.py | = 39 ++--- BaseTools/Source/Python/Common/Dictionary.py | = 9 +- BaseTools/Source/Python/Common/DscClassObject.py | = 70 ++++---- BaseTools/Source/Python/Common/EdkIIWorkspace.py | = 25 +-- BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py | 1= 33 +++++++-------- BaseTools/Source/Python/Common/Expression.py | = 81 ++++----- BaseTools/Source/Python/Common/FdfClassObject.py | = 1 + BaseTools/Source/Python/Common/FdfParserLite.py | = 47 +++--- BaseTools/Source/Python/Common/InfClassObject.py | 1= 13 ++++++------- BaseTools/Source/Python/Common/LongFilePathOs.py | = 2 +- BaseTools/Source/Python/Common/MigrationUtilities.py | = 1 + BaseTools/Source/Python/Common/Misc.py | = 70 ++++---- BaseTools/Source/Python/Common/Parsing.py | = 1 + BaseTools/Source/Python/Common/RangeExpression.py | = 32 ++-- BaseTools/Source/Python/Common/String.py | = 7 +- BaseTools/Source/Python/Common/TargetTxtClassObject.py | = 15 +- BaseTools/Source/Python/Common/ToolDefClassObject.py | = 3 +- BaseTools/Source/Python/Common/VpdInfoFile.py | = 23 +-- BaseTools/Source/Python/Ecc/CParser.py | 1= 73 ++++++++++---------- BaseTools/Source/Python/Ecc/Check.py | = 1 + BaseTools/Source/Python/Ecc/CodeFragmentCollector.py | = 69 ++++---- BaseTools/Source/Python/Ecc/Configuration.py | = 5 +- BaseTools/Source/Python/Ecc/Exception.py | = 3 +- BaseTools/Source/Python/Ecc/MetaDataParser.py | = 3 +- BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py | = 5 +- BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py | = 41 ++--- BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py | = 9 +- BaseTools/Source/Python/Ecc/c.py | = 15 +- BaseTools/Source/Python/Eot/CParser.py | 1= 73 ++++++++++---------- BaseTools/Source/Python/Eot/CodeFragmentCollector.py | = 61 +++---- BaseTools/Source/Python/Eot/FvImage.py | = 17 +- BaseTools/Source/Python/Eot/InfParserLite.py | = 8 +- BaseTools/Source/Python/Eot/Parser.py | = 2 +- BaseTools/Source/Python/Eot/c.py | = 23 +-- BaseTools/Source/Python/GenFds/AprioriSection.py | = 7 +- BaseTools/Source/Python/GenFds/Capsule.py | = 10 +- BaseTools/Source/Python/GenFds/CapsuleData.py | = 6 +- BaseTools/Source/Python/GenFds/EfiSection.py | = 6 +- BaseTools/Source/Python/GenFds/Fd.py | = 12 +- BaseTools/Source/Python/GenFds/FdfParser.py | = 43 ++--- BaseTools/Source/Python/GenFds/FfsFileStatement.py | = 5 +- BaseTools/Source/Python/GenFds/FfsInfStatement.py | = 16 +- BaseTools/Source/Python/GenFds/Fv.py | = 13 +- BaseTools/Source/Python/GenFds/FvImageSection.py | = 8 +- BaseTools/Source/Python/GenFds/GenFds.py | = 20 ++- BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py | = 10 +- BaseTools/Source/Python/GenFds/OptionRom.py | = 3 - BaseTools/Source/Python/GenFds/Region.py | = 14 +- BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py | = 9 +- BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py | = 1 + BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py | = 32 ++-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py | = 30 ++-- BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py | = 36 ++-- BaseTools/Source/Python/TargetTool/TargetTool.py | = 39 ++--- BaseTools/Source/Python/Trim/Trim.py | = 25 +-- BaseTools/Source/Python/UPT/Core/DependencyRules.py | = 12 +- BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py | = 4 +- BaseTools/Source/Python/UPT/Core/FileHook.py | = 2 +- BaseTools/Source/Python/UPT/Core/IpiDb.py | = 6 +- BaseTools/Source/Python/UPT/Core/PackageFile.py | = 12 +- BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py | = 15 +- BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py | = 42 ++--- BaseTools/Source/Python/UPT/InstallPkg.py | = 2 +- BaseTools/Source/Python/UPT/InventoryWs.py | = 2 +- BaseTools/Source/Python/UPT/Library/CommentParsing.py | = 5 +- BaseTools/Source/Python/UPT/Library/ExpressionValidate.py | = 11 +- BaseTools/Source/Python/UPT/Library/Misc.py | = 11 +- BaseTools/Source/Python/UPT/Library/ParserValidate.py | = 2 +- BaseTools/Source/Python/UPT/Library/Parsing.py | = 3 +- BaseTools/Source/Python/UPT/Library/String.py | = 5 +- BaseTools/Source/Python/UPT/Library/UniClassObject.py | = 20 ++- BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py | = 4 +- BaseTools/Source/Python/UPT/MkPkg.py | = 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py | = 6 +- BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py | = 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py | = 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py | = 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py | = 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py | = 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py | = 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py | = 4 +- BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py | = 2 +- BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py | = 3 +- BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py | = 4 +- BaseTools/Source/Python/UPT/Parser/DecParserMisc.py | = 1 + BaseTools/Source/Python/UPT/Parser/InfSectionParser.py | = 3 +- BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py | = 57 +++---- BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py | = 3 +- BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py | = 3 +- BaseTools/Source/Python/UPT/ReplacePkg.py | = 2 +- BaseTools/Source/Python/UPT/RmPkg.py | = 2 +- BaseTools/Source/Python/UPT/TestInstall.py | = 4 +- BaseTools/Source/Python/UPT/UPT.py | = 9 +- BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py | = 5 +- BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py | = 10 +- BaseTools/Source/Python/UPT/Xml/CommonXml.py | = 2 +- BaseTools/Source/Python/UPT/Xml/IniToXml.py | = 1 + BaseTools/Source/Python/UPT/Xml/XmlParser.py | = 25 +-- BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py | = 3 +- BaseTools/Source/Python/Workspace/BuildClassObject.py | = 2 +- BaseTools/Source/Python/Workspace/DecBuildData.py | = 14 +- BaseTools/Source/Python/Workspace/DscBuildData.py | 1= 42 ++++++++-------- BaseTools/Source/Python/Workspace/InfBuildData.py | = 3 +- BaseTools/Source/Python/Workspace/MetaFileParser.py | = 74 +++++---- BaseTools/Source/Python/Workspace/MetaFileTable.py | = 10 +- BaseTools/Source/Python/Workspace/WorkspaceCommon.py | = 2 +- BaseTools/Source/Python/build/BuildReport.py | = 17 +- BaseTools/Source/Python/build/build.py | = 35 ++-- BaseTools/Tests/CheckPythonSyntax.py | = 2 +- BaseTools/Tests/TestTools.py | = 13 +- BaseTools/Tests/TianoCompress.py | = 6 +- BaseTools/gcc/mingw-gcc-build.py | 1= 12 ++++++------- 136 files changed, 1559 insertions(+), 1477 deletions(-) -- 2.15.1