Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

["Gao, Liming" <liming.gao@intel.com>]

https://pythonclock.org/ say Python 2.7 will not be maintained past 2020. One BZ https://bugzilla.tianocore.org/show_bug.cgi?id=55 also requests this migration. So, we migrate BaseTools to support Python3. Because Python27 will not be maintained, we drop Python27 support when we do Python3 migration. In fact, OS supports more than one python tool. In Linux or Mac OS, after you install Python3, you will have "python3" command to trig python3. "python" command to trig python2. In Windows, after you install Python3, there will be py -3 to trig python3, and py -2 to trig python2. You can still use python2 for python2 script. BaseTools will auto find python3 command to call BaseTools script. User only needs to install Python3. It has no impact on other project. 

Why we choose Python3.6 or above? Python3.6 is relative new python version. We choose it to validate BaseTools Python3 functionality. In fact, current BaseTools may work with Python35 or Python34. But, we don't do full validation. So, we don't mention them. We can do basic test for other python3 version. If no big block issue, we can relax this requirement. And, Python37 is new version that does performance optimization. We collect performance data in Windows OS, Python37 improves 20% performance in BaseTools AutoGen phase than Python27. Python36 performance is bad. It decreases 20% performance than Python27. I get the information that early Python3 version performance is not good. So, I suggest to use Python37 for better build performance.

On break issue, I admit this is our mistake. We don't verify Wiki work flow. We need to fix it ASAP. And, I admit Yonghong breaks patch review process. One patch is not reviewed in mail list and pushed. We need to strictly follow the process. Thank you to catch it. 

> -----Original Message-----
> From: afish@apple.com [mailto:afish@apple.com]
> Sent: Saturday, October 13, 2018 11:12 PM
> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Gao, Liming <liming.gao@intel.com>; edk2-devel-01 <edk2-devel@lists.01.org>; Kinney, Michael D <michael.d.kinney@intel.com>;
> Laszlo Ersek <lersek@redhat.com>
> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> 
> 
> 
> > On Oct 13, 2018, at 5:18 AM, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >
> > On 13 October 2018 at 11:23, Gao, Liming <liming.gao@intel.com> wrote:
> >> Ard:
> >>  I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the
> python3 and set PYTHON3 env. This env is used in GNUmakefile.
> >>
> >>  I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle
> it.
> >>
> >
> > Thanks for diagnosing that. As Leif points out, this deviates from the
> > instructions on the wiki. It also deviates from the build script I use
> > in my CI environment, hence the breakage.
> >
> > Unfortunately, our CI environment is based on Debian Stretch, which
> > provides Python 3.5.3 only not Python 3.6 or later. I will work with
> > our infrastructure team to find out if we can support Python 3.6 as
> > well, but it will probably take time.
> >
> > What is the reason we need 3.6 or later?
> >
> 
> All the macOS based builds are broken too. Python 2.7.* comes with macOS. The instructions are broken for macOS as you never had to
> install Python before.
> 
> I'm not able to install an alternate version of Python in our internal CI infrastructure (You build the OS with tools from the OS). This will
> likely force me to have to check in a version of Python 3.6 into the edk2 tree to keep the build working, or fork off an older version of the
> build system that works with Python 2.7.*.
> 
> What problem are we solving by forcing everyone to be on Python 3.6 or later? I understand on Widows you have to install Python, but a
> lot of the Unix installs you can get an older version of Python installed for free.
> 
> Thanks,
> 
> Andrew Fish
> 
> >
> >>> -----Original Message-----
> >>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> >>> Sent: Saturday, October 13, 2018 4:46 PM
> >>> To: Gao, Liming <liming.gao@intel.com>
> >>> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
> >>> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
> >>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>>
> >>> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
> >>>> Ard:
> >>>>  I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found.
> But,
> >>> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37.
> Do
> >>> you install Python36 or Python37 in your machine?
> >>>>
> >>>>  Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
> >>>>
> >>>
> >>> I have python3 on my machine, yes. But that does not mean it is the
> >>> default for everyone.
> >>>
> >>>
> >>>>> -----Original Message-----
> >>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> >>>>> Sent: Saturday, October 13, 2018 4:18 PM
> >>>>> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
> >>> Lindholm
> >>>>> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> >>>>> stephano.cetola@linux.intel.com
> >>>>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>>>>
> >>>>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> >>>>>
> >>>>>  "BaseTools: update Test scripts support python3"
> >>>>>
> >>>>> [which has never been sent out for review to the mailing list, as far
> >>>>> as I can tell] is breaking the BaseTools build
> >>>>>
> >>>>> make -C Tests
> >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> /bin/sh: 1: RunTests.py: not found
> >>>>> GNUmakefile:17: recipe for target 'test' failed
> >>>>> make[1]: *** [test] Error 127
> >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> GNUmakefile:25: recipe for target 'Tests' failed
> >>>>> make: *** [Tests] Error 2
> >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>>>>
> >>>>> I tried reverting that patch alone, but that gives me
> >>>>>
> >>>>> make -C Tests
> >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> Traceback (most recent call last):
> >>>>>  File "RunTests.py", line 36, in <module>
> >>>>>    allTests = GetAllTestsSuite()
> >>>>>  File "RunTests.py", line 33, in GetAllTestsSuite
> >>>>>    return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
> >>>>>  File "RunTests.py", line 30, in GetPythonTestSuite
> >>>>>    return PythonToolsTests.TheTestSuite()
> >>>>>  File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> >>>>> line 27, in TheTestSuite
> >>>>>    import CheckUnicodeSourceFiles
> >>>>>  File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> >>>>> line 25, in <module>
> >>>>>    from Common.Misc import PathClass
> >>>>>  File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> >>>>> line 27, in <module>
> >>>>>    from collections import UserDict as IterableUserDict
> >>>>> ImportError: cannot import name UserDict
> >>>>> GNUmakefile:17: recipe for target 'test' failed
> >>>>> make[1]: *** [test] Error 1
> >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> GNUmakefile:25: recipe for target 'Tests' failed
> >>>>> make: *** [Tests] Error 2
> >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>>>>
> >>>>> Breaking the build for everyone with an unreviewed patch 2 days before
> >>>>> the plugfest is NOT OK.
> >>>>>
> >>>>> Please fix or revert ASAP
> >>>>>
> >>>>> (I suppose that the failing test is a false negative in this case, but
> >>>>> that is not really the point - my CI jobs are broken due to this and
> >>>>> so are many others, I imagine)
> >>>>> _______________________________________________
> >>>>> edk2-devel mailing list
> >>>>> edk2-devel@lists.01.org
> >>>>> https://lists.01.org/mailman/listinfo/edk2-devel
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel