[PATCH 0/2] BaseTools: Python 3.9 fixes

[PATCH 0/2] BaseTools: Python 3.9 fixes

[Cole]

These patches fix two issues running the BaseTools test suite
on python 3.9 from Fedora rawhide. See patches for individual details

Cole Robinson (2):
  BaseTools: fix ucs-2 lookup on python 3.9
  BaseTools: Work around array.array.tostring() removal in python 3.9

 BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
 BaseTools/Source/Python/Common/Misc.py            | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
2.26.2

[PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9

[Cole]

python3.9 changed/fixed codec.register behavior to always replace
hyphen with underscore for passed in codec names:

  https://bugs.python.org/issue37751

So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in
addition to existing 'ucs-2' for back compat.

This fixes test failures on python3.9, example:

======================================================================
FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 375, in PreProcess
    FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 303, in OpenUniFile
    UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 312, in VerifyUcs2Data
    Ucs2Info = codecs.lookup('ucs-2')
LookupError: unknown encoding: ucs-2

Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
 BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index b2895f7e5c..883c2356e0 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
 
 TheUcs2Codec = Ucs2Codec()
 def Ucs2Search(name):
-    if name == 'ucs-2':
+    if name in ['ucs-2', 'ucs_2']:
         return codecs.CodecInfo(
             name=name,
             encode=TheUcs2Codec.encode,
-- 
2.26.2

Re: [edk2-devel] [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9

[Yuwei Chen]

Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Cole
> Sent: Wednesday, August 12, 2020 1:28 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
> Subject: [edk2-devel] [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
> 
> python3.9 changed/fixed codec.register behavior to always replace hyphen
> with underscore for passed in codec names:
> 
>   https://bugs.python.org/issue37751
> 
> So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in addition
> to existing 'ucs-2' for back compat.
> 
> This fixes test failures on python3.9, example:
> 
> ==========================================================
> ============
> FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 375, in PreProcess
>     FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
>   File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 303, in OpenUniFile
>     UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
>   File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 312, in VerifyUcs2Data
>     Ucs2Info = codecs.lookup('ucs-2')
> LookupError: unknown encoding: ucs-2
> 
> Signed-off-by: Cole Robinson <crobinso@redhat.com>
> ---
>  BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> index b2895f7e5c..883c2356e0 100644
> --- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> +++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> @@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
> 
>  TheUcs2Codec = Ucs2Codec()
>  def Ucs2Search(name):
> -    if name == 'ucs-2':
> +    if name in ['ucs-2', 'ucs_2']:
>          return codecs.CodecInfo(
>              name=name,
>              encode=TheUcs2Codec.encode,
> --
> 2.26.2
> 
> 
> 

Re: [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9

[Bob Feng]

Reviewed-by: Bob Feng <bob.c.feng@intel.com>

-----Original Message-----
From: Cole Robinson <crobinso@redhat.com> 
Sent: Wednesday, August 12, 2020 1:28 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
Subject: [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9

python3.9 changed/fixed codec.register behavior to always replace hyphen with underscore for passed in codec names:

  https://bugs.python.org/issue37751

So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in addition to existing 'ucs-2' for back compat.

This fixes test failures on python3.9, example:

======================================================================
FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 375, in PreProcess
    FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 303, in OpenUniFile
    UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
  File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 312, in VerifyUcs2Data
    Ucs2Info = codecs.lookup('ucs-2')
LookupError: unknown encoding: ucs-2

Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
 BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index b2895f7e5c..883c2356e0 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
  TheUcs2Codec = Ucs2Codec() def Ucs2Search(name):-    if name == 'ucs-2':+    if name in ['ucs-2', 'ucs_2']:         return codecs.CodecInfo(             name=name,             encode=TheUcs2Codec.encode,-- 
2.26.2

[PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9

[Cole]

In python3, array.array.tostring() was a compat alias for tobytes().
tostring() was removed in python 3.9.

Convert this to use tolist() which should be valid for all python
versions.

This fixes this build error on python3.9:

(Python 3.9.0b5 on linux) Traceback (most recent call last):
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 593, in Main
    GenerateVfrBinSec(CommandOptions.ModuleName, CommandOptions.DebugDir, CommandOptions.OutputFile)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 449, in GenerateVfrBinSec
    VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName, VfrNameList)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in GetVariableOffset
    return _parseForGCC(lines, efifilepath, varnames)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in _parseForGCC
    efisecs = PeImageClass(efifilepath).SectionHeaderList
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in __init__
    if ByteArray.tostring() != b'PE\0\0':
AttributeError: 'array.array' object has no attribute 'tostring'

Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
 BaseTools/Source/Python/Common/Misc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
index ad55671080..4be7957138 100755
--- a/BaseTools/Source/Python/Common/Misc.py
+++ b/BaseTools/Source/Python/Common/Misc.py
@@ -1635,7 +1635,7 @@ class PeImageClass():
         ByteArray = array.array('B')
         ByteArray.fromfile(PeObject, 4)
         # PE signature should be 'PE\0\0'
-        if ByteArray.tostring() != b'PE\0\0':
+        if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]:
             self.ErrorInfo = self.FileName + ' has no valid PE signature PE00'
             return
 
-- 
2.26.2

Re: [edk2-devel] [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9

[Yuwei Chen]

Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Cole
> Sent: Wednesday, August 12, 2020 1:28 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
> Subject: [edk2-devel] [PATCH 2/2] BaseTools: Work around
> array.array.tostring() removal in python 3.9
> 
> In python3, array.array.tostring() was a compat alias for tobytes().
> tostring() was removed in python 3.9.
> 
> Convert this to use tolist() which should be valid for all python versions.
> 
> This fixes this build error on python3.9:
> 
> (Python 3.9.0b5 on linux) Traceback (most recent call last):
>   File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Tr
> im.py", line 593, in Main
>     GenerateVfrBinSec(CommandOptions.ModuleName,
> CommandOptions.DebugDir, CommandOptions.OutputFile)
>   File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Tr
> im.py", line 449, in GenerateVfrBinSec
>     VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName,
> VfrNameList)
>   File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in
> GetVariableOffset
>     return _parseForGCC(lines, efifilepath, varnames)
>   File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in
> _parseForGCC
>     efisecs = PeImageClass(efifilepath).SectionHeaderList
>   File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in
> __init__
>     if ByteArray.tostring() != b'PE\0\0':
> AttributeError: 'array.array' object has no attribute 'tostring'
> 
> Signed-off-by: Cole Robinson <crobinso@redhat.com>
> ---
>  BaseTools/Source/Python/Common/Misc.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/BaseTools/Source/Python/Common/Misc.py
> b/BaseTools/Source/Python/Common/Misc.py
> index ad55671080..4be7957138 100755
> --- a/BaseTools/Source/Python/Common/Misc.py
> +++ b/BaseTools/Source/Python/Common/Misc.py
> @@ -1635,7 +1635,7 @@ class PeImageClass():
>          ByteArray = array.array('B')
>          ByteArray.fromfile(PeObject, 4)
>          # PE signature should be 'PE\0\0'
> -        if ByteArray.tostring() != b'PE\0\0':
> +        if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]:
>              self.ErrorInfo = self.FileName + ' has no valid PE signature PE00'
>              return
> 
> --
> 2.26.2
> 
> 
> 

Re: [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9

[Bob Feng]

Reviewed-by: Bob Feng <bob.c.feng@intel.com>

-----Original Message-----
From: Cole Robinson <crobinso@redhat.com> 
Sent: Wednesday, August 12, 2020 1:28 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
Subject: [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9

In python3, array.array.tostring() was a compat alias for tobytes().
tostring() was removed in python 3.9.

Convert this to use tolist() which should be valid for all python versions.

This fixes this build error on python3.9:

(Python 3.9.0b5 on linux) Traceback (most recent call last):
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 593, in Main
    GenerateVfrBinSec(CommandOptions.ModuleName, CommandOptions.DebugDir, CommandOptions.OutputFile)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 449, in GenerateVfrBinSec
    VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName, VfrNameList)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in GetVariableOffset
    return _parseForGCC(lines, efifilepath, varnames)
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in _parseForGCC
    efisecs = PeImageClass(efifilepath).SectionHeaderList
  File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in __init__
    if ByteArray.tostring() != b'PE\0\0':
AttributeError: 'array.array' object has no attribute 'tostring'

Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
 BaseTools/Source/Python/Common/Misc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
index ad55671080..4be7957138 100755
--- a/BaseTools/Source/Python/Common/Misc.py
+++ b/BaseTools/Source/Python/Common/Misc.py
@@ -1635,7 +1635,7 @@ class PeImageClass():
         ByteArray = array.array('B')         ByteArray.fromfile(PeObject, 4)         # PE signature should be 'PE\0\0'-        if ByteArray.tostring() != b'PE\0\0':+        if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]:             self.ErrorInfo = self.FileName + ' has no valid PE signature PE00'             return -- 
2.26.2

Re: [edk2-devel] [PATCH 0/2] BaseTools: Python 3.9 fixes

[Laszlo Ersek]

On 08/11/20 19:28, Cole wrote:
> These patches fix two issues running the BaseTools test suite
> on python 3.9 from Fedora rawhide. See patches for individual details
> 
> Cole Robinson (2):
>   BaseTools: fix ucs-2 lookup on python 3.9
>   BaseTools: Work around array.array.tostring() removal in python 3.9
> 
>  BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
>  BaseTools/Source/Python/Common/Misc.py            | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 

Thank you, Cole, for the patches! The series has been merged in commit
range 5dc2699d101d..43bec9ea3d56.

Laszlo