* [PATCH 0/2] BaseTools: Python 3.9 fixes
@ 2020-08-11 17:28 Cole
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Cole @ 2020-08-11 17:28 UTC (permalink / raw)
To: devel; +Cc: bob.c.feng, liming.gao, Cole Robinson
These patches fix two issues running the BaseTools test suite
on python 3.9 from Fedora rawhide. See patches for individual details
Cole Robinson (2):
BaseTools: fix ucs-2 lookup on python 3.9
BaseTools: Work around array.array.tostring() removal in python 3.9
BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
BaseTools/Source/Python/Common/Misc.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
2.26.2
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
2020-08-11 17:28 [PATCH 0/2] BaseTools: Python 3.9 fixes Cole
@ 2020-08-11 17:28 ` Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
2020-08-14 1:20 ` Bob Feng
2020-08-11 17:28 ` [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in " Cole
2020-08-17 17:07 ` [edk2-devel] [PATCH 0/2] BaseTools: Python 3.9 fixes Laszlo Ersek
2 siblings, 2 replies; 8+ messages in thread
From: Cole @ 2020-08-11 17:28 UTC (permalink / raw)
To: devel; +Cc: bob.c.feng, liming.gao, Cole Robinson
python3.9 changed/fixed codec.register behavior to always replace
hyphen with underscore for passed in codec names:
https://bugs.python.org/issue37751
So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in
addition to existing 'ucs-2' for back compat.
This fixes test failures on python3.9, example:
======================================================================
FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 375, in PreProcess
FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 303, in OpenUniFile
UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 312, in VerifyUcs2Data
Ucs2Info = codecs.lookup('ucs-2')
LookupError: unknown encoding: ucs-2
Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index b2895f7e5c..883c2356e0 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
TheUcs2Codec = Ucs2Codec()
def Ucs2Search(name):
- if name == 'ucs-2':
+ if name in ['ucs-2', 'ucs_2']:
return codecs.CodecInfo(
name=name,
encode=TheUcs2Codec.encode,
--
2.26.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9
2020-08-11 17:28 [PATCH 0/2] BaseTools: Python 3.9 fixes Cole
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
@ 2020-08-11 17:28 ` Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
2020-08-14 1:20 ` Bob Feng
2020-08-17 17:07 ` [edk2-devel] [PATCH 0/2] BaseTools: Python 3.9 fixes Laszlo Ersek
2 siblings, 2 replies; 8+ messages in thread
From: Cole @ 2020-08-11 17:28 UTC (permalink / raw)
To: devel; +Cc: bob.c.feng, liming.gao, Cole Robinson
In python3, array.array.tostring() was a compat alias for tobytes().
tostring() was removed in python 3.9.
Convert this to use tolist() which should be valid for all python
versions.
This fixes this build error on python3.9:
(Python 3.9.0b5 on linux) Traceback (most recent call last):
File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 593, in Main
GenerateVfrBinSec(CommandOptions.ModuleName, CommandOptions.DebugDir, CommandOptions.OutputFile)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 449, in GenerateVfrBinSec
VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName, VfrNameList)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in GetVariableOffset
return _parseForGCC(lines, efifilepath, varnames)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in _parseForGCC
efisecs = PeImageClass(efifilepath).SectionHeaderList
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in __init__
if ByteArray.tostring() != b'PE\0\0':
AttributeError: 'array.array' object has no attribute 'tostring'
Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
BaseTools/Source/Python/Common/Misc.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
index ad55671080..4be7957138 100755
--- a/BaseTools/Source/Python/Common/Misc.py
+++ b/BaseTools/Source/Python/Common/Misc.py
@@ -1635,7 +1635,7 @@ class PeImageClass():
ByteArray = array.array('B')
ByteArray.fromfile(PeObject, 4)
# PE signature should be 'PE\0\0'
- if ByteArray.tostring() != b'PE\0\0':
+ if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]:
self.ErrorInfo = self.FileName + ' has no valid PE signature PE00'
return
--
2.26.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [edk2-devel] [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
@ 2020-08-13 0:16 ` Yuwei Chen
2020-08-14 1:20 ` Bob Feng
1 sibling, 0 replies; 8+ messages in thread
From: Yuwei Chen @ 2020-08-13 0:16 UTC (permalink / raw)
To: devel@edk2.groups.io, crobinso@redhat.com; +Cc: Feng, Bob C, Gao, Liming
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Cole
> Sent: Wednesday, August 12, 2020 1:28 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
> Subject: [edk2-devel] [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
>
> python3.9 changed/fixed codec.register behavior to always replace hyphen
> with underscore for passed in codec names:
>
> https://bugs.python.org/issue37751
>
> So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in addition
> to existing 'ucs-2' for back compat.
>
> This fixes test failures on python3.9, example:
>
> ==========================================================
> ============
> FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 375, in PreProcess
> FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
> File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 303, in OpenUniFile
> UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
> File "/builddir/build/BUILD/edk2-edk2-
> stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line
> 312, in VerifyUcs2Data
> Ucs2Info = codecs.lookup('ucs-2')
> LookupError: unknown encoding: ucs-2
>
> Signed-off-by: Cole Robinson <crobinso@redhat.com>
> ---
> BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> index b2895f7e5c..883c2356e0 100644
> --- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> +++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> @@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
>
> TheUcs2Codec = Ucs2Codec()
> def Ucs2Search(name):
> - if name == 'ucs-2':
> + if name in ['ucs-2', 'ucs_2']:
> return codecs.CodecInfo(
> name=name,
> encode=TheUcs2Codec.encode,
> --
> 2.26.2
>
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [edk2-devel] [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9
2020-08-11 17:28 ` [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in " Cole
@ 2020-08-13 0:16 ` Yuwei Chen
2020-08-14 1:20 ` Bob Feng
1 sibling, 0 replies; 8+ messages in thread
From: Yuwei Chen @ 2020-08-13 0:16 UTC (permalink / raw)
To: devel@edk2.groups.io, crobinso@redhat.com; +Cc: Feng, Bob C, Gao, Liming
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Cole
> Sent: Wednesday, August 12, 2020 1:28 AM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
> Subject: [edk2-devel] [PATCH 2/2] BaseTools: Work around
> array.array.tostring() removal in python 3.9
>
> In python3, array.array.tostring() was a compat alias for tobytes().
> tostring() was removed in python 3.9.
>
> Convert this to use tolist() which should be valid for all python versions.
>
> This fixes this build error on python3.9:
>
> (Python 3.9.0b5 on linux) Traceback (most recent call last):
> File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Tr
> im.py", line 593, in Main
> GenerateVfrBinSec(CommandOptions.ModuleName,
> CommandOptions.DebugDir, CommandOptions.OutputFile)
> File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Tr
> im.py", line 449, in GenerateVfrBinSec
> VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName,
> VfrNameList)
> File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in
> GetVariableOffset
> return _parseForGCC(lines, efifilepath, varnames)
> File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in
> _parseForGCC
> efisecs = PeImageClass(efifilepath).SectionHeaderList
> File "/root/edk2/edk2-edk2-
> stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in
> __init__
> if ByteArray.tostring() != b'PE\0\0':
> AttributeError: 'array.array' object has no attribute 'tostring'
>
> Signed-off-by: Cole Robinson <crobinso@redhat.com>
> ---
> BaseTools/Source/Python/Common/Misc.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/BaseTools/Source/Python/Common/Misc.py
> b/BaseTools/Source/Python/Common/Misc.py
> index ad55671080..4be7957138 100755
> --- a/BaseTools/Source/Python/Common/Misc.py
> +++ b/BaseTools/Source/Python/Common/Misc.py
> @@ -1635,7 +1635,7 @@ class PeImageClass():
> ByteArray = array.array('B')
> ByteArray.fromfile(PeObject, 4)
> # PE signature should be 'PE\0\0'
> - if ByteArray.tostring() != b'PE\0\0':
> + if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]:
> self.ErrorInfo = self.FileName + ' has no valid PE signature PE00'
> return
>
> --
> 2.26.2
>
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9
2020-08-11 17:28 ` [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in " Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
@ 2020-08-14 1:20 ` Bob Feng
1 sibling, 0 replies; 8+ messages in thread
From: Bob Feng @ 2020-08-14 1:20 UTC (permalink / raw)
To: Cole Robinson, devel@edk2.groups.io; +Cc: Gao, Liming
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: Cole Robinson <crobinso@redhat.com>
Sent: Wednesday, August 12, 2020 1:28 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
Subject: [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in python 3.9
In python3, array.array.tostring() was a compat alias for tobytes().
tostring() was removed in python 3.9.
Convert this to use tolist() which should be valid for all python versions.
This fixes this build error on python3.9:
(Python 3.9.0b5 on linux) Traceback (most recent call last):
File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 593, in Main
GenerateVfrBinSec(CommandOptions.ModuleName, CommandOptions.DebugDir, CommandOptions.OutputFile)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/BinWrappers/PosixLike/../../Source/Python/Trim/Trim.py", line 449, in GenerateVfrBinSec
VfrUniOffsetList = GetVariableOffset(MapFileName, EfiFileName, VfrNameList)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 88, in GetVariableOffset
return _parseForGCC(lines, efifilepath, varnames)
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 151, in _parseForGCC
efisecs = PeImageClass(efifilepath).SectionHeaderList
File "/root/edk2/edk2-edk2-stable202002/BaseTools/Source/Python/Common/Misc.py", line 1638, in __init__
if ByteArray.tostring() != b'PE\0\0':
AttributeError: 'array.array' object has no attribute 'tostring'
Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
BaseTools/Source/Python/Common/Misc.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
index ad55671080..4be7957138 100755
--- a/BaseTools/Source/Python/Common/Misc.py
+++ b/BaseTools/Source/Python/Common/Misc.py
@@ -1635,7 +1635,7 @@ class PeImageClass():
ByteArray = array.array('B') ByteArray.fromfile(PeObject, 4) # PE signature should be 'PE\0\0'- if ByteArray.tostring() != b'PE\0\0':+ if ByteArray.tolist() != [ord('P'), ord('E'), 0, 0]: self.ErrorInfo = self.FileName + ' has no valid PE signature PE00' return --
2.26.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
@ 2020-08-14 1:20 ` Bob Feng
1 sibling, 0 replies; 8+ messages in thread
From: Bob Feng @ 2020-08-14 1:20 UTC (permalink / raw)
To: Cole Robinson, devel@edk2.groups.io; +Cc: Gao, Liming
Reviewed-by: Bob Feng <bob.c.feng@intel.com>
-----Original Message-----
From: Cole Robinson <crobinso@redhat.com>
Sent: Wednesday, August 12, 2020 1:28 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Cole Robinson <crobinso@redhat.com>
Subject: [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9
python3.9 changed/fixed codec.register behavior to always replace hyphen with underscore for passed in codec names:
https://bugs.python.org/issue37751
So the custom Ucs2Search needs to be adapted to handle 'ucs_2' in addition to existing 'ucs-2' for back compat.
This fixes test failures on python3.9, example:
======================================================================
FAIL: testUtf16InUniFile (CheckUnicodeSourceFiles.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 375, in PreProcess
FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 303, in OpenUniFile
UniFileClassObject.VerifyUcs2Data(FileIn, FileName, Encoding)
File "/builddir/build/BUILD/edk2-edk2-stable202002/BaseTools/Source/Python/AutoGen/UniClassObject.py", line 312, in VerifyUcs2Data
Ucs2Info = codecs.lookup('ucs-2')
LookupError: unknown encoding: ucs-2
Signed-off-by: Cole Robinson <crobinso@redhat.com>
---
BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index b2895f7e5c..883c2356e0 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -152,7 +152,7 @@ class Ucs2Codec(codecs.Codec):
TheUcs2Codec = Ucs2Codec() def Ucs2Search(name):- if name == 'ucs-2':+ if name in ['ucs-2', 'ucs_2']: return codecs.CodecInfo( name=name, encode=TheUcs2Codec.encode,--
2.26.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [edk2-devel] [PATCH 0/2] BaseTools: Python 3.9 fixes
2020-08-11 17:28 [PATCH 0/2] BaseTools: Python 3.9 fixes Cole
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
2020-08-11 17:28 ` [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in " Cole
@ 2020-08-17 17:07 ` Laszlo Ersek
2 siblings, 0 replies; 8+ messages in thread
From: Laszlo Ersek @ 2020-08-17 17:07 UTC (permalink / raw)
To: devel, crobinso; +Cc: bob.c.feng, liming.gao
On 08/11/20 19:28, Cole wrote:
> These patches fix two issues running the BaseTools test suite
> on python 3.9 from Fedora rawhide. See patches for individual details
>
> Cole Robinson (2):
> BaseTools: fix ucs-2 lookup on python 3.9
> BaseTools: Work around array.array.tostring() removal in python 3.9
>
> BaseTools/Source/Python/AutoGen/UniClassObject.py | 2 +-
> BaseTools/Source/Python/Common/Misc.py | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
Thank you, Cole, for the patches! The series has been merged in commit
range 5dc2699d101d..43bec9ea3d56.
Laszlo
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-08-17 17:07 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-08-11 17:28 [PATCH 0/2] BaseTools: Python 3.9 fixes Cole
2020-08-11 17:28 ` [PATCH 1/2] BaseTools: fix ucs-2 lookup on python 3.9 Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
2020-08-14 1:20 ` Bob Feng
2020-08-11 17:28 ` [PATCH 2/2] BaseTools: Work around array.array.tostring() removal in " Cole
2020-08-13 0:16 ` [edk2-devel] " Yuwei Chen
2020-08-14 1:20 ` Bob Feng
2020-08-17 17:07 ` [edk2-devel] [PATCH 0/2] BaseTools: Python 3.9 fixes Laszlo Ersek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox