public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
* [PATCH v3 00/20] BaseTools: One step toward python3
@ 2018-06-21  4:43 Gary Lin
  2018-06-21  4:43 ` [PATCH v3 01/20] BaseTools: Fix a typo in ini.py Gary Lin
                   ` (19 more replies)
  0 siblings, 20 replies; 37+ messages in thread
From: Gary Lin @ 2018-06-21  4:43 UTC (permalink / raw)
  To: edk2-devel; +Cc: Yonghong Zhu, Liming Gao

v3 changes:
  - Rebase to the current git HEAD (2e1083038d9aa74fcaa2db8158fdee7c8b4af3bb)
  - Fix a typo in BaseTools/Scripts/PackageDocumentTools/plugins/EdkPlugins/basemodel/ini.py
  - Remove the patch for reduce() since it's not used anymore 

v2 changes:
  - Rebase to the current git HEAD (821807bcefb9a36e598d71a8004fae5aab2052a0)
  - Apply "futurize -f libfuturize.fixes.fix_absolute_import" and
    refactor some python scripts to break the circular imports.

This patch series is also available in
https://github.com/lcp/edk2/tree/python3-futurize-v3

Since python2 will be EOL in 2020, we start to evaluate the impact of
the python2 removal. As expected, OMVF building failed the test. It's
actually a task noted in the wiki page:

https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Python3-Support

Maybe it's time to convert the python scripts gradually.

This patchset doesn't make the python scripts in BaseTools compatible
with python3 immediately. It aims to do the trivial and safe conversion
and replacement to make some statements compatible with both python2 and
python3, so we can deal with the difficult cases later.

With the help of "futurize" from python-future, it's easier to refactor
the statements. This patchset is basically equivalent to "futurize -1"(*)
plus "StringIO.StringIO => io.BytesIO" and several patches to break the 
circular imports.

The patchset was tested with the following command:

$ ./OvmfPkg/build.sh -D SECURE_BOOT_ENABLE \
                     -D NETWORK_IP6_ENABLE \
                     -D HTTP_BOOT_ENABLE \
                     -D TLS_ENABLE \
                     -D TPM2_ENABLE

The firmware file was built successfully and I didn't notice any error
so far.

(*) http://python-future.org/automatic_conversion.html#stage-1-safe-fixes

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>

Gary Lin (20):
  BaseTools: Fix a typo in ini.py
  BaseTools: Refactor python except statements
  BaseTools: Refactor python print statements
  BaseTools: Remove the old python "not-equal"
  BaseTools: Use the python3-range functions
  BaseTools: Remove tuple parameter in python scripts
  BaseTools: Remove the deprecated hash_key()
  BaseTools: Replace StandardError with Expression
  BaseTools: Remove types.TypeType
  BaseTools: Refactor python raise statement
  BaseTools: Adjust the spaces around commas and colons
  BaseTools: Migrate to the new octal literal
  BaseTools: Unify long int and int in python scripts
  BaseTools: Fix old python2 idioms
  BaseTools: Replace StringIO.StringIO with io.BytesIO
  BaseTools: Treat GenFds.py and build.py as python modules
  BaseTools: Adopt absolute import for python scripts
  BaseTools: Move OverrideAttribs to OptRomInfStatement.py
  BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py
  BaseTools: Move ImageBinDict to GenFdsGlobalVariable.py

 .../Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py   |   5 +-
 BaseTools/BinWrappers/PosixLike/GenFds        |   2 +-
 BaseTools/BinWrappers/PosixLike/build         |   2 +-
 BaseTools/Scripts/BinToPcd.py                 |  10 +-
 BaseTools/Scripts/ConvertMasmToNasm.py        |   1 +
 BaseTools/Scripts/ConvertUni.py               |   5 -
 BaseTools/Scripts/FormatDosFiles.py           |   6 +-
 BaseTools/Scripts/MemoryProfileSymbolGen.py   |  21 +-
 .../PackageDocumentTools/packagedoc_cli.py    |  53 ++--
 .../plugins/EdkPlugins/basemodel/doxygen.py   |  14 +-
 .../plugins/EdkPlugins/basemodel/efibinary.py |  30 +-
 .../plugins/EdkPlugins/basemodel/ini.py       |   8 +-
 .../EdkPlugins/edk2/model/baseobject.py       |   6 +-
 .../EdkPlugins/edk2/model/doxygengen.py       |   5 +-
 .../EdkPlugins/edk2/model/doxygengen_spec.py  |   5 +-
 .../plugins/EdkPlugins/edk2/model/inf.py      |   8 +-
 BaseTools/Scripts/PatchCheck.py               |   7 +-
 BaseTools/Scripts/RunMakefile.py              |   2 +-
 .../Scripts/SmiHandlerProfileSymbolGen.py     |  19 +-
 BaseTools/Scripts/UpdateBuildVersions.py      |  18 +-
 BaseTools/Source/Python/AutoGen/AutoGen.py    |  93 +++---
 .../Source/Python/AutoGen/BuildEngine.py      |  38 +--
 BaseTools/Source/Python/AutoGen/GenC.py       |  80 +++---
 BaseTools/Source/Python/AutoGen/GenDepex.py   |   8 +-
 BaseTools/Source/Python/AutoGen/GenMake.py    |  11 +-
 BaseTools/Source/Python/AutoGen/GenPcdDb.py   | 148 +++++-----
 BaseTools/Source/Python/AutoGen/GenVar.py     | 163 +++++------
 .../Source/Python/AutoGen/IdfClassObject.py   |   4 +-
 .../Source/Python/AutoGen/InfSectionParser.py |   1 +
 BaseTools/Source/Python/AutoGen/StrGather.py  |  12 +-
 .../Source/Python/AutoGen/UniClassObject.py   |  18 +-
 .../Python/AutoGen/ValidCheckingInfoObject.py |   5 +-
 BaseTools/Source/Python/BPDG/BPDG.py          |   8 +-
 BaseTools/Source/Python/BPDG/GenVpd.py        |  28 +-
 BaseTools/Source/Python/Common/DataType.py    |   4 +-
 BaseTools/Source/Python/Common/Database.py    |   9 +-
 BaseTools/Source/Python/Common/EdkLogger.py   |   3 +-
 BaseTools/Source/Python/Common/Expression.py  |  84 +++---
 .../Source/Python/Common/LongFilePathOs.py    |   5 +-
 BaseTools/Source/Python/Common/Misc.py        |  61 ++--
 BaseTools/Source/Python/Common/Parsing.py     |   6 +-
 .../Source/Python/Common/RangeExpression.py   |  34 +--
 BaseTools/Source/Python/Common/StringUtils.py |  16 +-
 .../Python/Common/TargetTxtClassObject.py     |  16 +-
 .../Python/Common/ToolDefClassObject.py       |  16 +-
 BaseTools/Source/Python/Common/VpdInfoFile.py |  23 +-
 BaseTools/Source/Python/Ecc/CParser.py        | 178 ++++++------
 BaseTools/Source/Python/Ecc/Check.py          |  24 +-
 .../Python/Ecc/CodeFragmentCollector.py       |  82 +++---
 BaseTools/Source/Python/Ecc/Configuration.py  |   5 +-
 BaseTools/Source/Python/Ecc/Database.py       |   7 +-
 BaseTools/Source/Python/Ecc/Ecc.py            |  25 +-
 BaseTools/Source/Python/Ecc/Exception.py      |   6 +-
 BaseTools/Source/Python/Ecc/FileProfile.py    |   5 +-
 BaseTools/Source/Python/Ecc/MetaDataParser.py |   8 +-
 .../Ecc/MetaFileWorkspace/MetaDataTable.py    |   5 +-
 .../Ecc/MetaFileWorkspace/MetaFileParser.py   |  46 +--
 .../Ecc/MetaFileWorkspace/MetaFileTable.py    |   5 +-
 .../Source/Python/Ecc/Xml/XmlRoutines.py      |   9 +-
 BaseTools/Source/Python/Ecc/c.py              |  28 +-
 BaseTools/Source/Python/Eot/CParser.py        | 178 ++++++------
 .../Python/Eot/CodeFragmentCollector.py       |  72 ++---
 BaseTools/Source/Python/Eot/Eot.py            |  22 +-
 BaseTools/Source/Python/Eot/FileProfile.py    |   3 +-
 BaseTools/Source/Python/Eot/InfParserLite.py  |  13 +-
 BaseTools/Source/Python/Eot/Parser.py         |   5 +-
 BaseTools/Source/Python/Eot/Report.py         |   3 +-
 BaseTools/Source/Python/Eot/c.py              |  32 ++-
 .../Source/Python/GenFds/AprioriSection.py    |  12 +-
 BaseTools/Source/Python/GenFds/Capsule.py     |  24 +-
 BaseTools/Source/Python/GenFds/CapsuleData.py |  11 +-
 .../Source/Python/GenFds/CompressSection.py   |   7 +-
 BaseTools/Source/Python/GenFds/DataSection.py |   7 +-
 .../Source/Python/GenFds/DepexSection.py      |   7 +-
 BaseTools/Source/Python/GenFds/EfiSection.py  |  13 +-
 BaseTools/Source/Python/GenFds/Fd.py          |  33 +--
 BaseTools/Source/Python/GenFds/FdfParser.py   | 102 +++----
 .../Source/Python/GenFds/FfsFileStatement.py  |  16 +-
 .../Source/Python/GenFds/FfsInfStatement.py   |  37 +--
 BaseTools/Source/Python/GenFds/Fv.py          |  32 ++-
 .../Source/Python/GenFds/FvImageSection.py    |  15 +-
 BaseTools/Source/Python/GenFds/GenFds.py      | 123 ++------
 .../Python/GenFds/GenFdsGlobalVariable.py     | 107 ++++++-
 BaseTools/Source/Python/GenFds/GuidSection.py |  11 +-
 .../Python/GenFds/OptRomFileStatement.py      |   3 +-
 .../Python/GenFds/OptRomInfStatement.py       |  28 +-
 BaseTools/Source/Python/GenFds/OptionRom.py   |  23 +-
 BaseTools/Source/Python/GenFds/Region.py      |  17 +-
 .../Source/Python/GenFds/RuleComplexFile.py   |   3 +-
 .../Source/Python/GenFds/RuleSimpleFile.py    |   3 +-
 BaseTools/Source/Python/GenFds/Section.py     |   3 +-
 BaseTools/Source/Python/GenFds/UiSection.py   |   7 +-
 BaseTools/Source/Python/GenFds/VerSection.py  |   7 +-
 BaseTools/Source/Python/GenFds/Vtf.py         |   3 +-
 .../GenPatchPcdTable/GenPatchPcdTable.py      |   9 +-
 .../Python/PatchPcdValue/PatchPcdValue.py     |   1 +
 .../Source/Python/Pkcs7Sign/Pkcs7Sign.py      |  32 ++-
 .../Rsa2048Sha256GenerateKeys.py              |  26 +-
 .../Rsa2048Sha256Sign/Rsa2048Sha256Sign.py    |  36 +--
 .../Source/Python/Table/TableDataModel.py     |   3 +-
 BaseTools/Source/Python/Table/TableDec.py     |   3 +-
 BaseTools/Source/Python/Table/TableDsc.py     |   3 +-
 .../Source/Python/Table/TableEotReport.py     |   5 +-
 BaseTools/Source/Python/Table/TableFdf.py     |   3 +-
 BaseTools/Source/Python/Table/TableFile.py    |   3 +-
 .../Source/Python/Table/TableFunction.py      |   3 +-
 .../Source/Python/Table/TableIdentifier.py    |   5 +-
 BaseTools/Source/Python/Table/TableInf.py     |   3 +-
 BaseTools/Source/Python/Table/TablePcd.py     |   5 +-
 BaseTools/Source/Python/Table/TableQuery.py   |   3 +-
 BaseTools/Source/Python/Table/TableReport.py  |   3 +-
 .../Source/Python/TargetTool/TargetTool.py    |  39 +--
 BaseTools/Source/Python/Trim/Trim.py          |  25 +-
 .../Source/Python/UPT/Core/DependencyRules.py |  12 +-
 .../UPT/Core/DistributionPackageClass.py      |   4 +-
 BaseTools/Source/Python/UPT/Core/FileHook.py  |   2 +-
 BaseTools/Source/Python/UPT/Core/IpiDb.py     |   6 +-
 .../Source/Python/UPT/Core/PackageFile.py     |  12 +-
 .../Python/UPT/GenMetaFile/GenDecFile.py      |  15 +-
 .../Python/UPT/GenMetaFile/GenInfFile.py      |  42 ++-
 BaseTools/Source/Python/UPT/InstallPkg.py     |   2 +-
 BaseTools/Source/Python/UPT/InventoryWs.py    |   2 +-
 .../Python/UPT/Library/CommentParsing.py      |   5 +-
 .../Python/UPT/Library/ExpressionValidate.py  |  11 +-
 BaseTools/Source/Python/UPT/Library/Misc.py   |  11 +-
 .../Python/UPT/Library/ParserValidate.py      |   2 +-
 .../Source/Python/UPT/Library/Parsing.py      |   6 +-
 .../Source/Python/UPT/Library/StringUtils.py  |   5 +-
 .../Python/UPT/Library/UniClassObject.py      |  20 +-
 .../Python/UPT/Library/Xml/XmlRoutines.py     |   4 +-
 BaseTools/Source/Python/UPT/MkPkg.py          |   2 +-
 .../UPT/Object/Parser/InfBinaryObject.py      |   6 +-
 .../UPT/Object/Parser/InfDefineObject.py      |   2 +-
 .../Python/UPT/Object/Parser/InfGuidObject.py |   4 +-
 .../Object/Parser/InfLibraryClassesObject.py  |   2 +-
 .../Python/UPT/Object/Parser/InfMisc.py       |   4 +-
 .../UPT/Object/Parser/InfPackagesObject.py    |   4 +-
 .../Python/UPT/Object/Parser/InfPcdObject.py  |   4 +-
 .../Python/UPT/Object/Parser/InfPpiObject.py  |   4 +-
 .../UPT/Object/Parser/InfProtocolObject.py    |   2 +-
 .../UPT/Object/Parser/InfSoucesObject.py      |   3 +-
 .../Object/Parser/InfUserExtensionObject.py   |   4 +-
 .../Source/Python/UPT/Parser/DecParserMisc.py |   1 +
 .../Python/UPT/Parser/InfSectionParser.py     |   3 +-
 .../Python/UPT/PomAdapter/DecPomAlignment.py  |  57 ++--
 .../Python/UPT/PomAdapter/InfPomAlignment.py  |   3 +-
 .../UPT/PomAdapter/InfPomAlignmentMisc.py     |   3 +-
 BaseTools/Source/Python/UPT/ReplacePkg.py     |   2 +-
 BaseTools/Source/Python/UPT/RmPkg.py          |   2 +-
 BaseTools/Source/Python/UPT/TestInstall.py    |   4 +-
 BaseTools/Source/Python/UPT/UPT.py            |   9 +-
 .../Python/UPT/UnitTest/DecParserTest.py      |   5 +-
 .../UPT/UnitTest/InfBinarySectionTest.py      |  10 +-
 BaseTools/Source/Python/UPT/Xml/CommonXml.py  |   2 +-
 BaseTools/Source/Python/UPT/Xml/IniToXml.py   |   1 +
 BaseTools/Source/Python/UPT/Xml/XmlParser.py  |  25 +-
 .../Source/Python/UPT/Xml/XmlParserMisc.py    |   3 +-
 .../Python/Workspace/BuildClassObject.py      |  16 +-
 .../Source/Python/Workspace/DecBuildData.py   |  22 +-
 .../Source/Python/Workspace/DscBuildData.py   | 269 +++++++++---------
 .../Source/Python/Workspace/InfBuildData.py   |   6 +-
 .../Source/Python/Workspace/MetaFileParser.py |  75 ++---
 .../Source/Python/Workspace/MetaFileTable.py  |  15 +-
 .../Python/Workspace/WorkspaceCommon.py       |   5 +-
 .../Python/Workspace/WorkspaceDatabase.py     |   7 +-
 BaseTools/Source/Python/build/BuildReport.py  |  21 +-
 BaseTools/Source/Python/build/build.py        |  42 +--
 BaseTools/Tests/CheckPythonSyntax.py          |   2 +-
 BaseTools/Tests/TestTools.py                  |  13 +-
 BaseTools/Tests/TianoCompress.py              |   6 +-
 BaseTools/gcc/mingw-gcc-build.py              | 112 ++++----
 171 files changed, 1927 insertions(+), 1762 deletions(-)

-- 
2.17.1



^ permalink raw reply	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2018-06-25  6:18 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-06-21  4:43 [PATCH v3 00/20] BaseTools: One step toward python3 Gary Lin
2018-06-21  4:43 ` [PATCH v3 01/20] BaseTools: Fix a typo in ini.py Gary Lin
2018-06-21 15:46   ` Carsey, Jaben
2018-06-21  4:43 ` [PATCH v3 02/20] BaseTools: Refactor python except statements Gary Lin
2018-06-21  6:27   ` Gary Lin
2018-06-21  4:43 ` [PATCH v3 03/20] BaseTools: Refactor python print statements Gary Lin
2018-06-21  4:43 ` [PATCH v3 04/20] BaseTools: Remove the old python "not-equal" Gary Lin
2018-06-21  4:43 ` [PATCH v3 05/20] BaseTools: Use the python3-range functions Gary Lin
2018-06-21 12:40   ` Zhu, Yonghong
2018-06-22  2:04     ` Gary Lin
2018-06-22  5:29       ` Zhu, Yonghong
2018-06-22  8:21         ` Gary Lin
2018-06-22  8:45           ` Gary Lin
2018-06-22  8:53             ` Zhu, Yonghong
2018-06-22  9:13               ` Gary Lin
2018-06-22  9:22                 ` Zhu, Yonghong
2018-06-22  9:24                 ` Zhu, Yonghong
2018-06-22  9:48                   ` Gary Lin
2018-06-22 10:22                     ` Zhu, Yonghong
2018-06-25  3:34                       ` Gary Lin
2018-06-25  5:58                         ` Zhu, Yonghong
2018-06-25  6:18                           ` Gary Lin
2018-06-21  4:43 ` [PATCH v3 06/20] BaseTools: Remove tuple parameter in python scripts Gary Lin
2018-06-21  4:43 ` [PATCH v3 07/20] BaseTools: Remove the deprecated hash_key() Gary Lin
2018-06-21  4:43 ` [PATCH v3 08/20] BaseTools: Replace StandardError with Expression Gary Lin
2018-06-21  4:43 ` [PATCH v3 09/20] BaseTools: Remove types.TypeType Gary Lin
2018-06-21  4:43 ` [PATCH v3 10/20] BaseTools: Refactor python raise statement Gary Lin
2018-06-21  4:43 ` [PATCH v3 11/20] BaseTools: Adjust the spaces around commas and colons Gary Lin
2018-06-21  4:43 ` [PATCH v3 12/20] BaseTools: Migrate to the new octal literal Gary Lin
2018-06-21  4:43 ` [PATCH v3 13/20] BaseTools: Unify long int and int in python scripts Gary Lin
2018-06-21  4:43 ` [PATCH v3 14/20] BaseTools: Fix old python2 idioms Gary Lin
2018-06-21  4:43 ` [PATCH v3 15/20] BaseTools: Replace StringIO.StringIO with io.BytesIO Gary Lin
2018-06-21  4:43 ` [PATCH v3 16/20] BaseTools: Treat GenFds.py and build.py as python modules Gary Lin
2018-06-21  4:43 ` [PATCH v3 17/20] BaseTools: Adopt absolute import for python scripts Gary Lin
2018-06-21  4:43 ` [PATCH v3 18/20] BaseTools: Move OverrideAttribs to OptRomInfStatement.py Gary Lin
2018-06-21  4:43 ` [PATCH v3 19/20] BaseTools: Move FindExtendTool to GenFdsGlobalVariable.py Gary Lin
2018-06-21  4:43 ` [PATCH v3 20/20] BaseTools: Move ImageBinDict " Gary Lin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox