* [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
@ 2019-12-04 21:38 Philippe Mathieu-Daudé
2019-12-05 18:36 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 3+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-12-04 21:38 UTC (permalink / raw)
To: devel; +Cc: Zhiju Fan, Philippe Mathieu-Daude, Bob Feng, Liming Gao
Under Centos 7.7 we get:
Build environment: Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
[...]
======================================================================
ERROR: testRandomDataCycles (TianoCompress.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles
self.compressionTestCycle(data)
File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle
start = self.ReadTmpFile('input')
File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
data = f.read()
File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
----------------------------------------------------------------------
Fix by specifying the UTF-8 encoding.
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
---
RFC because I'm not sure this is the best way to fix this, but
this is similar to commit 31e3eeb5e3d2d.
---
BaseTools/Tests/TestTools.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
index 1099fd4eeaea..41cdb28b0c8c 100644
--- a/BaseTools/Tests/TestTools.py
+++ b/BaseTools/Tests/TestTools.py
@@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
return open(os.path.join(self.testDir, fileName), mode)
def ReadTmpFile(self, fileName):
- f = open(self.GetTmpFilePath(fileName), 'r')
+ f = codecs.open(self.GetTmpFilePath(fileName), 'r', encoding='utf-8')
data = f.read()
f.close()
return data
--
2.21.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
2019-12-04 21:38 [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools Philippe Mathieu-Daudé
@ 2019-12-05 18:36 ` Philippe Mathieu-Daudé
2019-12-05 20:09 ` [edk2-devel] " Laszlo Ersek
0 siblings, 1 reply; 3+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-12-05 18:36 UTC (permalink / raw)
To: devel; +Cc: Zhiju Fan, Bob Feng, Liming Gao
On 12/4/19 10:38 PM, Philippe Mathieu-Daude wrote:
> Under Centos 7.7 we get:
>
> Build environment: Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
> [...]
> ======================================================================
> ERROR: testRandomDataCycles (TianoCompress.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles
> self.compressionTestCycle(data)
> File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle
> start = self.ReadTmpFile('input')
> File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
> data = f.read()
> File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
>
> ----------------------------------------------------------------------
>
> Fix by specifying the UTF-8 encoding.
>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
> ---
> RFC because I'm not sure this is the best way to fix this, but
> this is similar to commit 31e3eeb5e3d2d.
> ---
> BaseTools/Tests/TestTools.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
> index 1099fd4eeaea..41cdb28b0c8c 100644
> --- a/BaseTools/Tests/TestTools.py
> +++ b/BaseTools/Tests/TestTools.py
> @@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
> return open(os.path.join(self.testDir, fileName), mode)
>
> def ReadTmpFile(self, fileName):
> - f = open(self.GetTmpFilePath(fileName), 'r')
> + f = codecs.open(self.GetTmpFilePath(fileName), 'r', encoding='utf-8')
> data = f.read()
> f.close()
> return data
>
While this fixes Python3, this also break Python2 :)
======================================================================
ERROR: testRandomDataCycles (TianoCompress.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in
testRandomDataCycles
self.compressionTestCycle(data)
File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in
compressionTestCycle
start = self.ReadTmpFile('input')
File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
data = f.read()
File "/usr/lib/python2.7/codecs.py", line 688, in read
return self.reader.read(size)
File "/usr/lib/python2.7/codecs.py", line 494, in read
newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 0:
invalid start byte
This old thread recommend to use io.open:
https://web.archive.org/web/20180715024113/https://mail.python.org/pipermail/python-list/2015-March/687124.html
And it works in with both 2/3 versions, so I'll respin.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [edk2-devel] [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
2019-12-05 18:36 ` Philippe Mathieu-Daudé
@ 2019-12-05 20:09 ` Laszlo Ersek
0 siblings, 0 replies; 3+ messages in thread
From: Laszlo Ersek @ 2019-12-05 20:09 UTC (permalink / raw)
To: devel, philmd; +Cc: Zhiju Fan, Bob Feng, Liming Gao
On 12/05/19 19:36, Philippe Mathieu-Daudé wrote:
> On 12/4/19 10:38 PM, Philippe Mathieu-Daude wrote:
>> Under Centos 7.7 we get:
>>
>> Build environment:
>> Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
>> [...]
>> ======================================================================
>> ERROR: testRandomDataCycles (TianoCompress.Tests)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in
>> testRandomDataCycles
>> self.compressionTestCycle(data)
>> File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in
>> compressionTestCycle
>> start = self.ReadTmpFile('input')
>> File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
>> data = f.read()
>> File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
>> position 3: ordinal not in range(128)
>>
>> ----------------------------------------------------------------------
>>
>> Fix by specifying the UTF-8 encoding.
>>
>> Cc: Bob Feng <bob.c.feng@intel.com>
>> Cc: Liming Gao <liming.gao@intel.com>
>> Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
>> ---
>> RFC because I'm not sure this is the best way to fix this, but
>> this is similar to commit 31e3eeb5e3d2d.
>> ---
>> BaseTools/Tests/TestTools.py | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
>> index 1099fd4eeaea..41cdb28b0c8c 100644
>> --- a/BaseTools/Tests/TestTools.py
>> +++ b/BaseTools/Tests/TestTools.py
>> @@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
>> return open(os.path.join(self.testDir, fileName), mode)
>> def ReadTmpFile(self, fileName):
>> - f = open(self.GetTmpFilePath(fileName), 'r')
>> + f = codecs.open(self.GetTmpFilePath(fileName), 'r',
>> encoding='utf-8')
>> data = f.read()
>> f.close()
>> return data
>>
>
> While this fixes Python3, this also break Python2 :)
>
> ======================================================================
> ERROR: testRandomDataCycles (TianoCompress.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in
> testRandomDataCycles
> self.compressionTestCycle(data)
> File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in
> compressionTestCycle
> start = self.ReadTmpFile('input')
> File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
> data = f.read()
> File "/usr/lib/python2.7/codecs.py", line 688, in read
> return self.reader.read(size)
> File "/usr/lib/python2.7/codecs.py", line 494, in read
> newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 0:
> invalid start byte
>
> This old thread recommend to use io.open:
> https://web.archive.org/web/20180715024113/https://mail.python.org/pipermail/python-list/2015-March/687124.html
>
>
> And it works in with both 2/3 versions, so I'll respin.
I didn't ask before (because, "commit 31e3eeb5e3d2d must have been
right, right?"), but now I can't resist anymore:
*why* do we have any such character in a *temporary* file's pathname
that is not pure ASCII? It seems wrong.
Thanks
Laszlo
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-12-05 20:09 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-12-04 21:38 [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools Philippe Mathieu-Daudé
2019-12-05 18:36 ` Philippe Mathieu-Daudé
2019-12-05 20:09 ` [edk2-devel] " Laszlo Ersek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox