From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.120]) by mx.groups.io with SMTP id smtpd.web12.1494.1575571006526198302 for ; Thu, 05 Dec 2019 10:36:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M68izbYx; spf=pass (domain: redhat.com, ip: 205.139.110.120, mailfrom: philmd@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1575571005; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IHWzt+5GCqP1wzU25+988ToJ5dJawFFdLjacxM9wJZ0=; b=M68izbYxwaM7y0IKhm8m2C/PSMJaDNyc/q0VaJz33bu3VUzehhdwTAqB9Gai0VLPGWjLX4 Ikb9zUV27BwnQM7IC4xj/f+EqWyoPym2veENBM0m/6o1aBfjSLdcw+MeoatRAGAuZou7BH L1I5h3xe9v6U6hhoPuoB9DwPSTlqUFg= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-182-3ByOYlxfNymB3BeJ2Cm1qA-1; Thu, 05 Dec 2019 13:36:43 -0500 Received: by mail-wr1-f69.google.com with SMTP id l20so1924491wrc.13 for ; Thu, 05 Dec 2019 10:36:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=IHWzt+5GCqP1wzU25+988ToJ5dJawFFdLjacxM9wJZ0=; b=sURSoZS/BPXVhxKZN4NkewJTNIpJVcpnYuMHph02/riLJ7JpHfAa98jq6Ody4W1fzl FqR+AiuAdkhvCHdUFEm1s+MrZkAh/hib6/T5dO6RuLug9kzwIkdpgXn89uIIQQs6MOTd KS9HAX4a+8H5wwkHRzPAcOqn3w8kXcZzc4NFPpRMWN8vW1Kmr38vNP4SWrgh67MUy+xZ kyMWPT0OmdGCyKXqhYM+QxLDid/LqjpvMkMIUi7+h8mF7w6rYny+hfeG+K0IZmp0ef9Q XQrE/vGy6OC8E9M3AXrhx0kEYiN0DSWPIsWV2EcFBjfshkzWCZcuQcDrVyfbrxrcbf6D XIcA== X-Gm-Message-State: APjAAAUONiKovzLIWOghZDi2FTGdDgfDgFSUg2tyISn7fYfyTlSOnDyI NmSRvN0UGe9yOagEF6/77rIbuowJbij0EM2OaUmNNRX0+vGfmamDRNvzx+WkVQz7ZhNY5GaSQIe d+c5clF+ZmjqArw== X-Received: by 2002:a1c:e909:: with SMTP id q9mr6957665wmc.30.1575571002100; Thu, 05 Dec 2019 10:36:42 -0800 (PST) X-Google-Smtp-Source: APXvYqzIftRqjTqQldf4+8Y0CF0r994ACBPz21Yb0BYrmWcSS0P//gNl7mEwZcqi6IPet45fcJNrww== X-Received: by 2002:a1c:e909:: with SMTP id q9mr6957641wmc.30.1575571001864; Thu, 05 Dec 2019 10:36:41 -0800 (PST) Return-Path: Received: from [192.168.1.35] (182.red-88-21-103.staticip.rima-tde.net. [88.21.103.182]) by smtp.gmail.com with ESMTPSA id x10sm13288925wrv.60.2019.12.05.10.36.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 05 Dec 2019 10:36:41 -0800 (PST) Subject: Re: [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools To: devel@edk2.groups.io Cc: Zhiju Fan , Bob Feng , Liming Gao References: <20191204213825.27644-1-philmd@redhat.com> From: =?UTF-8?B?UGhpbGlwcGUgTWF0aGlldS1EYXVkw6k=?= Message-ID: <828306f1-755e-da8a-96f2-af85828e56a4@redhat.com> Date: Thu, 5 Dec 2019 19:36:40 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: <20191204213825.27644-1-philmd@redhat.com> X-MC-Unique: 3ByOYlxfNymB3BeJ2Cm1qA-1 X-Mimecast-Spam-Score: 0 Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 12/4/19 10:38 PM, Philippe Mathieu-Daude wrote: > Under Centos 7.7 we get: > > Build environment: Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core > [...] > ====================================================================== > ERROR: testRandomDataCycles (TianoCompress.Tests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles > self.compressionTestCycle(data) > File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle > start = self.ReadTmpFile('input') > File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile > data = f.read() > File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode > return codecs.ascii_decode(input, self.errors)[0] > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128) > > ---------------------------------------------------------------------- > > Fix by specifying the UTF-8 encoding. > > Cc: Bob Feng > Cc: Liming Gao > Signed-off-by: Philippe Mathieu-Daude > --- > RFC because I'm not sure this is the best way to fix this, but > this is similar to commit 31e3eeb5e3d2d. > --- > BaseTools/Tests/TestTools.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py > index 1099fd4eeaea..41cdb28b0c8c 100644 > --- a/BaseTools/Tests/TestTools.py > +++ b/BaseTools/Tests/TestTools.py > @@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase): > return open(os.path.join(self.testDir, fileName), mode) > > def ReadTmpFile(self, fileName): > - f = open(self.GetTmpFilePath(fileName), 'r') > + f = codecs.open(self.GetTmpFilePath(fileName), 'r', encoding='utf-8') > data = f.read() > f.close() > return data > While this fixes Python3, this also break Python2 :) ====================================================================== ERROR: testRandomDataCycles (TianoCompress.Tests) ---------------------------------------------------------------------- Traceback (most recent call last): File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles self.compressionTestCycle(data) File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle start = self.ReadTmpFile('input') File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile data = f.read() File "/usr/lib/python2.7/codecs.py", line 688, in read return self.reader.read(size) File "/usr/lib/python2.7/codecs.py", line 494, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 0: invalid start byte This old thread recommend to use io.open: https://web.archive.org/web/20180715024113/https://mail.python.org/pipermail/python-list/2015-March/687124.html And it works in with both 2/3 versions, so I'll respin.