public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
From: "Feng, Bob C" <bob.c.feng@intel.com>
To: Laszlo Ersek <lersek@redhat.com>
Cc: "edk2-devel@lists.01.org" <edk2-devel@lists.01.org>,
	"Gao, Liming" <liming.gao@intel.com>
Subject: Re: [Patch 00/33] BaseTools python3 migration patch set
Date: Mon, 28 Jan 2019 10:35:30 +0000	[thread overview]
Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D160076553@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: cab4fed6-4c5d-94a9-b29f-da41ad7f320e@redhat.com

Hi Laszlo,

I sent out 2 patches to fix this issue. And I also pushed the patches to the repo https://github.com/BobCF/edk2.git branch py3basetools

Thanks,
Bob

-----Original Message-----
From: Feng, Bob C 
Sent: Monday, January 28, 2019 10:33 AM
To: 'Laszlo Ersek' <lersek@redhat.com>
Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
Subject: RE: [edk2] [Patch 00/33] BaseTools python3 migration patch set

Hi Laszlo,

Thanks for your testing and detailed testing result. I'll check the issue happened in (a2) and provide a patch for it.

Thanks,
Bob 

-----Original Message-----
From: Laszlo Ersek [mailto:lersek@redhat.com]
Sent: Saturday, January 26, 2019 2:18 AM
To: Feng, Bob C <bob.c.feng@intel.com>
Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
Subject: Re: [edk2] [Patch 00/33] BaseTools python3 migration patch set

On 01/25/19 10:42, Feng, Bob C wrote:
> [...]
>
> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Friday, January 25, 2019 4:57 PM
> To: Feng, Bob C <bob.c.feng@intel.com>
> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
> Subject: Re: [edk2] [Patch 00/33] BaseTools python3 migration patch 
> set
>
> [...]
>
> On 01/25/19 05:55, Feng, Bob C wrote:
>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=55
>>
>> This patch set is to enable python3 on BaseTools. Basetools code will 
>> be compatible with both python3 and python2.
>>
>> We will have two envs PYTHON3_ENABLE and PYTHON_COMMAND. The behavior 
>> can be combined as the below to support this usage.
>> If user wants the specific python interpreter, he only needs to set 
>> PYTHON_COMMAND env.
>> If PYTHON3_ENABLE is set, PYTHON_COMMAND will be set to the found one 
>> by edk2 scripts based on PYTHON3_ENABLE value.
>> If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is set, then 
>> PYTHON_COMMAND will be used to run python script. No version check 
>> here.
>> If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is not set, 
>> PYTHON_COMMAND will be set to the high version python installed in 
>> OS.

I ran the following tests, at commit 41c0616fc081 ("BaseTools: Eot tool
Python3 adaption", 2019-01-25). Each test was performed in a clean tree (after running "git clean -ffdx") and clean environment (I re-sourced "edksetup.sh" for each test in separation). In addition, the base tools were rebuilt (again from a clean tree) for each test, with the following command [1]:

  nice make -C "$EDK_TOOLS_PATH" -j $(getconf _NPROCESSORS_ONLN)

(a) On my RHEL7.5 Workstation laptop, I have both the system-level python packages installed (python-2.7.5-69.el7_5.x86_64), and the extra
python-3.4 stuff from EPEL-7 (python34-3.4.9-1.el7.x86_64).

(a1) Didn't set either PYTHON3_ENABLE or PYTHON_COMMAND. The build utility picked

  PYTHON_COMMAND   = /usr/bin/python3.4

and I successfully built OvmfPkg for IA32, IA32X64, and X64; also ArmVirtQemu for AARCH64. The built firmware images passed a smoke test too.

(a2) I removed all the python34 packages (and the dependent packages) from my laptop. Didn't set either of PYTHON3_ENABLE and PYTHON_COMMAND.
(This is the configuration what a "normal" RHEL7 environment would
provide.)

In this case, rebuilding the base tools [1] failed, in the testing
phase:

> ======================================================================
> FAIL: test_Ecc_CParser4_CLexer (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Ecc/CParser4/CLexer.py, Error is   File ".../BaseTools/Source/Python/Ecc/CParser4/CLexer.py", line 590
>     def __init__(self, input=None, output:TextIO = sys.stdout):
>                                          ^
> SyntaxError: invalid syntax
>
>
> ======================================================================
> FAIL: test_Ecc_CParser4_CListener (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Ecc/CParser4/CListener.py, Error is   File ".../BaseTools/Source/Python/Ecc/CParser4/CListener.py", line 35
>     def enterTranslation_unit(self, ctx:CParser.Translation_unitContext):
>                                        ^
> SyntaxError: invalid syntax
>
>
> ======================================================================
> FAIL: test_Ecc_CParser4_CParser (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Ecc/CParser4/CParser.py, Error is   File ".../BaseTools/Source/Python/Ecc/CParser4/CParser.py", line 744
>     def __init__(self, input:TokenStream, output:TextIO = sys.stdout):
>                             ^
> SyntaxError: invalid syntax
>
>
> ======================================================================
> FAIL: test_Eot_CParser4_CLexer (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Eot/CParser4/CLexer.py, Error is   File ".../BaseTools/Source/Python/Eot/CParser4/CLexer.py", line 591
>     def __init__(self, input=None, output:TextIO = sys.stdout):
>                                          ^
> SyntaxError: invalid syntax
>
>
> ======================================================================
> FAIL: test_Eot_CParser4_CListener (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Eot/CParser4/CListener.py, Error is   File ".../BaseTools/Source/Python/Eot/CParser4/CListener.py", line 35
>     def enterTranslation_unit(self, ctx:CParser.Translation_unitContext):
>                                        ^
> SyntaxError: invalid syntax
>
>
> ======================================================================
> FAIL: test_Eot_CParser4_CParser (CheckPythonSyntax.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 55, in <lambda>
>     newmethod = lambda self: self.SingleFileTest(filename)
>   File ".../BaseTools/Tests/CheckPythonSyntax.py", line 33, in SingleFileTest
>     self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
> AssertionError: syntax error: .../BaseTools/Source/Python/Eot/CParser4/CParser.py, Error is   File ".../BaseTools/Source/Python/Eot/CParser4/CParser.py", line 744
>     def __init__(self, input:TokenStream, output:TextIO = sys.stdout):
>                             ^
> SyntaxError: invalid syntax
>
>
> ----------------------------------------------------------------------
> Ran 270 tests in 0.938s
>
> FAILED (failures=6)

I didn't continue testing on my RHEL7 laptop beyond this point.


(b) RHEL-8 virtual machine, with "/usr/bin/python3.6" from python36-3.6.6-18.el8.x86_64, and "/usr/libexec/platform-python" from platform-python-3.6.8-1.el8.x86_64.

(b1) Didn't set either PYTHON3_ENABLE or PYTHON_COMMAND. The build utility picked

  PYTHON_COMMAND   = /usr/bin/python3.6

and I successfully built OvmfPkg for IA32, IA32X64, and X64. (I don't have a cross-compiler installed in this environment yet, nor a RHEL8
aarch64 KVM guest, so I couldn't test ArmVirtQemu for now).

(b2) I set PYTHON_COMMAND to "/usr/libexec/platform-python". Didn't set PYTHON3_ENABLE. The same builds as in (b1) succeeded.


Thus, what seems to be missing is a little bit of python2 compatibility, in (a2).

Thanks,
Laszlo

  parent reply	other threads:[~2019-01-28 10:35 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-25  4:55 [Patch 00/33] BaseTools python3 migration patch set Feng, Bob C
2019-01-25  4:55 ` [Patch 01/33] BaseTool:Rename xrange() to range() Feng, Bob C
2019-01-25  4:55 ` [Patch 02/33] BaseTools:use iterate list to replace the itertools Feng, Bob C
2019-01-25  4:55 ` [Patch 03/33] BaseTools: Rename iteritems to items Feng, Bob C
2019-01-25  4:55 ` [Patch 04/33] BaseTools: replace get_bytes_le() to bytes_le Feng, Bob C
2019-01-25  4:55 ` [Patch 05/33] BaseTools: use OrderedDict instead of sdict Feng, Bob C
2019-01-25  4:55 ` [Patch 06/33] BaseTools: nametuple not have verbose parameter in python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 07/33] BaseTools: Remove unnecessary super function Feng, Bob C
2019-01-25  4:56 ` [Patch 08/33] BaseTools: replace long by int Feng, Bob C
2019-01-25  4:56 ` [Patch 09/33] BaseTools:Solve the data sorting problem use python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 10/33] BaseTools: Update argparse arguments since it not have version now Feng, Bob C
2019-01-25  4:56 ` [Patch 11/33] BaseTools:Similar to octal data rectification Feng, Bob C
2019-01-25  4:56 ` [Patch 12/33] BaseTools/UPT:merge UPT Tool use Python2 and Python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 13/33] BaseTools: update Test scripts support python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 14/33] BaseTools/Scripts: Porting PackageDocumentTools code to use Python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 15/33] Basetools: It went wrong when use os.linesep Feng, Bob C
2019-01-25  4:56 ` [Patch 16/33] BaseTools:Fv BaseAddress must set If it not set Feng, Bob C
2019-01-25  4:56 ` [Patch 17/33] BaseTools: Make sure AllPcdList valid Feng, Bob C
2019-01-25  4:56 ` [Patch 18/33] BaseTools:TestTools character encoding issue Feng, Bob C
2019-01-25  4:56 ` [Patch 19/33] BaseTools:Double carriage return inserted from Trim.py on Python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 20/33] BaseTools:File open failed for VPD MapFile Feng, Bob C
2019-01-25  4:56 ` [Patch 21/33] BaseTools: change the Division Operator Feng, Bob C
2019-01-25  4:56 ` [Patch 22/33] BaseTools:There is extra blank line in datalog Feng, Bob C
2019-01-25  4:56 ` [Patch 23/33] BaseTools: Similar to octal data rectification Feng, Bob C
2019-01-25  4:56 ` [Patch 24/33] BaseTools: Update windows and linux run scripts file to use Python3 Feng, Bob C
2019-01-25  4:56 ` [Patch 25/33] BaseTools:Update build tool to print python version information Feng, Bob C
2019-01-25  4:56 ` [Patch 26/33] BaseTools:Linux Python highest version check Feng, Bob C
2019-01-25  4:56 ` [Patch 27/33] BaseTools: Update PYTHON env to PYTHON_COMMAND Feng, Bob C
2019-01-25  4:56 ` [Patch 28/33] BaseTools:Fixed Rsa issue and a set define issue Feng, Bob C
2019-01-25  4:56 ` [Patch 29/33] BaseTools:ord() don't match in py2 and py3 Feng, Bob C
2019-01-25  4:56 ` [Patch 30/33] BaseTools: the list and iterator translation Feng, Bob C
2019-01-25  4:56 ` [Patch 31/33] BaseTools: Handle the bytes and str difference Feng, Bob C
2019-01-25  4:56 ` [Patch 32/33] BaseTools: ECC tool Python3 adaption Feng, Bob C
2019-01-25  4:56 ` [Patch 33/33] BaseTools: Eot " Feng, Bob C
2019-01-25  8:56 ` [Patch 00/33] BaseTools python3 migration patch set Laszlo Ersek
2019-01-25  9:42   ` Feng, Bob C
2019-01-25 18:18     ` Laszlo Ersek
2019-01-28  2:33       ` Feng, Bob C
2019-01-28 10:35       ` Feng, Bob C [this message]
2019-01-28 13:48         ` Laszlo Ersek
2019-01-29  2:15           ` Feng, Bob C

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-list from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=08650203BA1BD64D8AD9B6D5D74A85D160076553@SHSMSX101.ccr.corp.intel.com \
    --to=devel@edk2.groups.io \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox