public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
* [PATCH 00/15] BaseTools: One step toward python3
@ 2018-01-19  4:43 Gary Lin
  2018-01-19  4:43 ` [PATCH 01/15] BaseTools: Refactor python except statements Gary Lin
                   ` (15 more replies)
  0 siblings, 16 replies; 18+ messages in thread
From: Gary Lin @ 2018-01-19  4:43 UTC (permalink / raw)
  To: edk2-devel; +Cc: Yonghong Zhu, Liming Gao

Since python2 will be EOF in 2020, we start to evaluate the impact of
the python2 removal. As expected, OMVF building failed the test. It's
actually a task noted in the wiki page:

https://github.com/tianocore/tianocore.github.io/wiki/Tasks-BaseTools-Python3-Support

Maybe it's time to convert the python scripts gradully.

This patchset doesn't make the python scripts in BaseTools compatible
with python3 immediately. It aims to do the trivial and safe conversion
and replacement to make some statements compatible with both python2 and
python3, so we can deal with the difficult cases later.

With the help of "futurize" from python-future, it's easier to refactor
the statements. This patchset is basically equivalent to "futurize -1"
plus "StringIO.StringIO => io.BytesIO" and minus "fix_absolute_import".
The reason to skip "fix_absolute_import" is that python2 failed to find
some modules after converting to absolute import, and it might take time
to figure out a proper fix.

For the "io.BytesIO" change, it MIGHT introduce slow down to the build
time since io.BytesIO is slower than StringIO.StringIO in python2(*).
For a quick test, I built OVMF with the following command based on
8ab0bd2397c9d3922e0c7dbb1aa6f7e08799079f:

$ rm -rf Build && make -C BaseTools/ clean
$ time ./OvmfPkg/build.sh -D SECURE_BOOT_ENABLE \
                          -D NETWORK_IP6_ENABLE \
                          -D HTTP_BOOT_ENABLE \
                          -D TLS_ENABLE

Before io.BytesIO:

  Build total time: 00:03:56
  real    4m22.991s
  user    3m55.874s
  sys     0m27.250s

After io.BytesIO:

  Build total time: 00:03:57
  real    4m23.953s
  user    3m57.526s
  sys     0m27.192s

The difference is only 1 second, and I would say the impact is subtle. 

The next step will be fixing relative import and maybe applying more
futurize fixes. We won't get there soon but at least we are moving... 

(*) https://stackoverflow.com/questions/37462075/confusing-about-stringio-cstringio-and-byteio

Contributed-under: TianoCore Contribution Agreement 1.1
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Gary Lin <glin@suse.com>

Gary Lin (15):
  BaseTools: Refactor python except statements
  BaseTools: Refactor python print statements
  BaseTools: Remove the old python "not-equal"
  BaseTools: Use the python3-range functions
  BaseTools: Remove tuple parameter in python scripts
  BaseTools: Remove the deprecated hash_key()
  BaseTools: Import reduce() from functools
  BaseTools: Replace StandardError with Expression
  BaseTools: Remove types.TypeType
  BaseTools: Refactor python raise statement
  BaseTools: Adjust the spaces around commas and colons
  BaseTools: Migrate to the new octal literal
  BaseTools: Unify long int and int in python scripts
  BaseTools: Adjust old python2 idioms
  BaseTools: Replace StringIO.StringIO with io.BytesIO

 BaseTools/Bin/CYGWIN_NT-5.1-i686/armcc_wrapper.py                      |   5 +-
 BaseTools/Scripts/BinToPcd.py                                          |  46 +++---
 BaseTools/Scripts/ConvertMasmToNasm.py                                 |   1 +
 BaseTools/Scripts/ConvertUni.py                                        |   5 -
 BaseTools/Scripts/MemoryProfileSymbolGen.py                            |  22 +--
 BaseTools/Scripts/PatchCheck.py                                        |   7 +-
 BaseTools/Scripts/RunMakefile.py                                       |   2 +-
 BaseTools/Scripts/SmiHandlerProfileSymbolGen.py                        |  20 +--
 BaseTools/Scripts/UpdateBuildVersions.py                               |  18 +-
 BaseTools/Source/Python/AutoGen/AutoGen.py                             |  91 +++++-----
 BaseTools/Source/Python/AutoGen/BuildEngine.py                         |  38 +++--
 BaseTools/Source/Python/AutoGen/GenC.py                                |   5 +-
 BaseTools/Source/Python/AutoGen/GenDepex.py                            |   8 +-
 BaseTools/Source/Python/AutoGen/GenMake.py                             |   8 +-
 BaseTools/Source/Python/AutoGen/GenPcdDb.py                            | 142 ++++++++--------
 BaseTools/Source/Python/AutoGen/GenVar.py                              | 165 +++++++++----------
 BaseTools/Source/Python/AutoGen/IdfClassObject.py                      |   1 -
 BaseTools/Source/Python/AutoGen/InfSectionParser.py                    |   1 +
 BaseTools/Source/Python/AutoGen/StrGather.py                           |   5 +-
 BaseTools/Source/Python/AutoGen/UniClassObject.py                      |  18 +-
 BaseTools/Source/Python/AutoGen/ValidCheckingInfoObject.py             |  10 +-
 BaseTools/Source/Python/BPDG/BPDG.py                                   |   3 +-
 BaseTools/Source/Python/BPDG/GenVpd.py                                 |  25 +--
 BaseTools/Source/Python/Common/DataType.py                             |   4 +-
 BaseTools/Source/Python/Common/DecClassObject.py                       |  39 ++---
 BaseTools/Source/Python/Common/Dictionary.py                           |   9 +-
 BaseTools/Source/Python/Common/DscClassObject.py                       |  70 ++++----
 BaseTools/Source/Python/Common/EdkIIWorkspace.py                       |  25 +--
 BaseTools/Source/Python/Common/EdkIIWorkspaceBuild.py                  | 133 +++++++--------
 BaseTools/Source/Python/Common/Expression.py                           |  81 ++++-----
 BaseTools/Source/Python/Common/FdfClassObject.py                       |   1 +
 BaseTools/Source/Python/Common/FdfParserLite.py                        |  47 +++---
 BaseTools/Source/Python/Common/InfClassObject.py                       | 113 ++++++-------
 BaseTools/Source/Python/Common/LongFilePathOs.py                       |   2 +-
 BaseTools/Source/Python/Common/MigrationUtilities.py                   |   1 +
 BaseTools/Source/Python/Common/Misc.py                                 |  70 ++++----
 BaseTools/Source/Python/Common/Parsing.py                              |   1 +
 BaseTools/Source/Python/Common/RangeExpression.py                      |  32 ++--
 BaseTools/Source/Python/Common/String.py                               |   7 +-
 BaseTools/Source/Python/Common/TargetTxtClassObject.py                 |  15 +-
 BaseTools/Source/Python/Common/ToolDefClassObject.py                   |   3 +-
 BaseTools/Source/Python/Common/VpdInfoFile.py                          |  23 +--
 BaseTools/Source/Python/Ecc/CParser.py                                 | 173 ++++++++++----------
 BaseTools/Source/Python/Ecc/Check.py                                   |   1 +
 BaseTools/Source/Python/Ecc/CodeFragmentCollector.py                   |  69 ++++----
 BaseTools/Source/Python/Ecc/Configuration.py                           |   5 +-
 BaseTools/Source/Python/Ecc/Exception.py                               |   3 +-
 BaseTools/Source/Python/Ecc/MetaDataParser.py                          |   3 +-
 BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaDataTable.py         |   5 +-
 BaseTools/Source/Python/Ecc/MetaFileWorkspace/MetaFileParser.py        |  41 ++---
 BaseTools/Source/Python/Ecc/Xml/XmlRoutines.py                         |   9 +-
 BaseTools/Source/Python/Ecc/c.py                                       |  15 +-
 BaseTools/Source/Python/Eot/CParser.py                                 | 173 ++++++++++----------
 BaseTools/Source/Python/Eot/CodeFragmentCollector.py                   |  61 +++----
 BaseTools/Source/Python/Eot/FvImage.py                                 |  17 +-
 BaseTools/Source/Python/Eot/InfParserLite.py                           |   8 +-
 BaseTools/Source/Python/Eot/Parser.py                                  |   2 +-
 BaseTools/Source/Python/Eot/c.py                                       |  23 +--
 BaseTools/Source/Python/GenFds/AprioriSection.py                       |   7 +-
 BaseTools/Source/Python/GenFds/Capsule.py                              |  10 +-
 BaseTools/Source/Python/GenFds/CapsuleData.py                          |   6 +-
 BaseTools/Source/Python/GenFds/EfiSection.py                           |   6 +-
 BaseTools/Source/Python/GenFds/Fd.py                                   |  12 +-
 BaseTools/Source/Python/GenFds/FdfParser.py                            |  43 ++---
 BaseTools/Source/Python/GenFds/FfsFileStatement.py                     |   5 +-
 BaseTools/Source/Python/GenFds/FfsInfStatement.py                      |  16 +-
 BaseTools/Source/Python/GenFds/Fv.py                                   |  13 +-
 BaseTools/Source/Python/GenFds/FvImageSection.py                       |   8 +-
 BaseTools/Source/Python/GenFds/GenFds.py                               |  20 ++-
 BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py                 |  10 +-
 BaseTools/Source/Python/GenFds/OptionRom.py                            |   3 -
 BaseTools/Source/Python/GenFds/Region.py                               |  14 +-
 BaseTools/Source/Python/GenPatchPcdTable/GenPatchPcdTable.py           |   9 +-
 BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py                 |   1 +
 BaseTools/Source/Python/Pkcs7Sign/Pkcs7Sign.py                         |  32 ++--
 BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256GenerateKeys.py |  30 ++--
 BaseTools/Source/Python/Rsa2048Sha256Sign/Rsa2048Sha256Sign.py         |  36 ++--
 BaseTools/Source/Python/TargetTool/TargetTool.py                       |  39 ++---
 BaseTools/Source/Python/Trim/Trim.py                                   |  25 +--
 BaseTools/Source/Python/UPT/Core/DependencyRules.py                    |  12 +-
 BaseTools/Source/Python/UPT/Core/DistributionPackageClass.py           |   4 +-
 BaseTools/Source/Python/UPT/Core/FileHook.py                           |   2 +-
 BaseTools/Source/Python/UPT/Core/IpiDb.py                              |   6 +-
 BaseTools/Source/Python/UPT/Core/PackageFile.py                        |  12 +-
 BaseTools/Source/Python/UPT/GenMetaFile/GenDecFile.py                  |  15 +-
 BaseTools/Source/Python/UPT/GenMetaFile/GenInfFile.py                  |  42 ++---
 BaseTools/Source/Python/UPT/InstallPkg.py                              |   2 +-
 BaseTools/Source/Python/UPT/InventoryWs.py                             |   2 +-
 BaseTools/Source/Python/UPT/Library/CommentParsing.py                  |   5 +-
 BaseTools/Source/Python/UPT/Library/ExpressionValidate.py              |  11 +-
 BaseTools/Source/Python/UPT/Library/Misc.py                            |  11 +-
 BaseTools/Source/Python/UPT/Library/ParserValidate.py                  |   2 +-
 BaseTools/Source/Python/UPT/Library/Parsing.py                         |   3 +-
 BaseTools/Source/Python/UPT/Library/String.py                          |   5 +-
 BaseTools/Source/Python/UPT/Library/UniClassObject.py                  |  20 ++-
 BaseTools/Source/Python/UPT/Library/Xml/XmlRoutines.py                 |   4 +-
 BaseTools/Source/Python/UPT/MkPkg.py                                   |   2 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfBinaryObject.py           |   6 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfDefineObject.py           |   2 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfGuidObject.py             |   4 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfLibraryClassesObject.py   |   2 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfMisc.py                   |   4 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfPackagesObject.py         |   4 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfPcdObject.py              |   4 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfPpiObject.py              |   4 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfProtocolObject.py         |   2 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfSoucesObject.py           |   3 +-
 BaseTools/Source/Python/UPT/Object/Parser/InfUserExtensionObject.py    |   4 +-
 BaseTools/Source/Python/UPT/Parser/DecParserMisc.py                    |   1 +
 BaseTools/Source/Python/UPT/Parser/InfSectionParser.py                 |   3 +-
 BaseTools/Source/Python/UPT/PomAdapter/DecPomAlignment.py              |  57 +++----
 BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignment.py              |   3 +-
 BaseTools/Source/Python/UPT/PomAdapter/InfPomAlignmentMisc.py          |   3 +-
 BaseTools/Source/Python/UPT/ReplacePkg.py                              |   2 +-
 BaseTools/Source/Python/UPT/RmPkg.py                                   |   2 +-
 BaseTools/Source/Python/UPT/TestInstall.py                             |   4 +-
 BaseTools/Source/Python/UPT/UPT.py                                     |   9 +-
 BaseTools/Source/Python/UPT/UnitTest/DecParserTest.py                  |   5 +-
 BaseTools/Source/Python/UPT/UnitTest/InfBinarySectionTest.py           |  10 +-
 BaseTools/Source/Python/UPT/Xml/CommonXml.py                           |   2 +-
 BaseTools/Source/Python/UPT/Xml/IniToXml.py                            |   1 +
 BaseTools/Source/Python/UPT/Xml/XmlParser.py                           |  25 +--
 BaseTools/Source/Python/UPT/Xml/XmlParserMisc.py                       |   3 +-
 BaseTools/Source/Python/Workspace/BuildClassObject.py                  |   2 +-
 BaseTools/Source/Python/Workspace/DecBuildData.py                      |  14 +-
 BaseTools/Source/Python/Workspace/DscBuildData.py                      | 142 ++++++++--------
 BaseTools/Source/Python/Workspace/InfBuildData.py                      |   3 +-
 BaseTools/Source/Python/Workspace/MetaFileParser.py                    |  74 +++++----
 BaseTools/Source/Python/Workspace/MetaFileTable.py                     |  10 +-
 BaseTools/Source/Python/Workspace/WorkspaceCommon.py                   |   2 +-
 BaseTools/Source/Python/build/BuildReport.py                           |  17 +-
 BaseTools/Source/Python/build/build.py                                 |  35 ++--
 BaseTools/Tests/CheckPythonSyntax.py                                   |   2 +-
 BaseTools/Tests/TestTools.py                                           |  13 +-
 BaseTools/Tests/TianoCompress.py                                       |   6 +-
 BaseTools/gcc/mingw-gcc-build.py                                       | 112 ++++++-------
 136 files changed, 1559 insertions(+), 1477 deletions(-)

-- 
2.15.1



^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2018-01-26  1:56 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-19  4:43 [PATCH 00/15] BaseTools: One step toward python3 Gary Lin
2018-01-19  4:43 ` [PATCH 01/15] BaseTools: Refactor python except statements Gary Lin
2018-01-19  4:43 ` [PATCH 02/15] BaseTools: Refactor python print statements Gary Lin
2018-01-19  4:43 ` [PATCH 03/15] BaseTools: Remove the old python "not-equal" Gary Lin
2018-01-19  4:43 ` [PATCH 04/15] BaseTools: Use the python3-range functions Gary Lin
2018-01-19  4:43 ` [PATCH 05/15] BaseTools: Remove tuple parameter in python scripts Gary Lin
2018-01-19  4:43 ` [PATCH 06/15] BaseTools: Remove the deprecated hash_key() Gary Lin
2018-01-19  4:43 ` [PATCH 07/15] BaseTools: Import reduce() from functools Gary Lin
2018-01-19  4:43 ` [PATCH 08/15] BaseTools: Replace StandardError with Expression Gary Lin
2018-01-19  4:43 ` [PATCH 09/15] BaseTools: Remove types.TypeType Gary Lin
2018-01-19  4:43 ` [PATCH 10/15] BaseTools: Refactor python raise statement Gary Lin
2018-01-19  4:43 ` [PATCH 11/15] BaseTools: Adjust the spaces around commas and colons Gary Lin
2018-01-19  4:43 ` [PATCH 12/15] BaseTools: Migrate to the new octal literal Gary Lin
2018-01-19  4:43 ` [PATCH 13/15] BaseTools: Unify long int and int in python scripts Gary Lin
2018-01-19  4:43 ` [PATCH 14/15] BaseTools: Adjust old python2 idioms Gary Lin
2018-01-19  4:43 ` [PATCH 15/15] BaseTools: Replace StringIO.StringIO with io.BytesIO Gary Lin
2018-01-25 13:37 ` [PATCH 00/15] BaseTools: One step toward python3 Zhu, Yonghong
2018-01-26  2:01   ` Gary Lin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox