Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Andrew Fish <afish@apple.com>]

> On Oct 13, 2018, at 5:18 AM, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> 
> On 13 October 2018 at 11:23, Gao, Liming <liming.gao@intel.com> wrote:
>> Ard:
>>  I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the python3 and set PYTHON3 env. This env is used in GNUmakefile.
>> 
>>  I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle it.
>> 
> 
> Thanks for diagnosing that. As Leif points out, this deviates from the
> instructions on the wiki. It also deviates from the build script I use
> in my CI environment, hence the breakage.
> 
> Unfortunately, our CI environment is based on Debian Stretch, which
> provides Python 3.5.3 only not Python 3.6 or later. I will work with
> our infrastructure team to find out if we can support Python 3.6 as
> well, but it will probably take time.
> 
> What is the reason we need 3.6 or later?
> 

All the macOS based builds are broken too. Python 2.7.* comes with macOS. The instructions are broken for macOS as you never had to install Python before. 

I'm not able to install an alternate version of Python in our internal CI infrastructure (You build the OS with tools from the OS). This will likely force me to have to check in a version of Python 3.6 into the edk2 tree to keep the build working, or fork off an older version of the build system that works with Python 2.7.*. 

What problem are we solving by forcing everyone to be on Python 3.6 or later? I understand on Widows you have to install Python, but a lot of the Unix installs you can get an older version of Python installed for free. 

Thanks,

Andrew Fish

> 
>>> -----Original Message-----
>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>>> Sent: Saturday, October 13, 2018 4:46 PM
>>> To: Gao, Liming <liming.gao@intel.com>
>>> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
>>> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
>>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>> 
>>> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
>>>> Ard:
>>>>  I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But,
>>> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do
>>> you install Python36 or Python37 in your machine?
>>>> 
>>>>  Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
>>>> 
>>> 
>>> I have python3 on my machine, yes. But that does not mean it is the
>>> default for everyone.
>>> 
>>> 
>>>>> -----Original Message-----
>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
>>>>> Sent: Saturday, October 13, 2018 4:18 PM
>>>>> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
>>> Lindholm
>>>>> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
>>>>> stephano.cetola@linux.intel.com
>>>>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>>>> 
>>>>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
>>>>> 
>>>>>  "BaseTools: update Test scripts support python3"
>>>>> 
>>>>> [which has never been sent out for review to the mailing list, as far
>>>>> as I can tell] is breaking the BaseTools build
>>>>> 
>>>>> make -C Tests
>>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> /bin/sh: 1: RunTests.py: not found
>>>>> GNUmakefile:17: recipe for target 'test' failed
>>>>> make[1]: *** [test] Error 127
>>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> GNUmakefile:25: recipe for target 'Tests' failed
>>>>> make: *** [Tests] Error 2
>>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>>>> 
>>>>> I tried reverting that patch alone, but that gives me
>>>>> 
>>>>> make -C Tests
>>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> Traceback (most recent call last):
>>>>>  File "RunTests.py", line 36, in <module>
>>>>>    allTests = GetAllTestsSuite()
>>>>>  File "RunTests.py", line 33, in GetAllTestsSuite
>>>>>    return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
>>>>>  File "RunTests.py", line 30, in GetPythonTestSuite
>>>>>    return PythonToolsTests.TheTestSuite()
>>>>>  File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
>>>>> line 27, in TheTestSuite
>>>>>    import CheckUnicodeSourceFiles
>>>>>  File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
>>>>> line 25, in <module>
>>>>>    from Common.Misc import PathClass
>>>>>  File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
>>>>> line 27, in <module>
>>>>>    from collections import UserDict as IterableUserDict
>>>>> ImportError: cannot import name UserDict
>>>>> GNUmakefile:17: recipe for target 'test' failed
>>>>> make[1]: *** [test] Error 1
>>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> GNUmakefile:25: recipe for target 'Tests' failed
>>>>> make: *** [Tests] Error 2
>>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>>>> 
>>>>> Breaking the build for everyone with an unreviewed patch 2 days before
>>>>> the plugfest is NOT OK.
>>>>> 
>>>>> Please fix or revert ASAP
>>>>> 
>>>>> (I suppose that the failing test is a false negative in this case, but
>>>>> that is not really the point - my CI jobs are broken due to this and
>>>>> so are many others, I imagine)
>>>>> _______________________________________________
>>>>> edk2-devel mailing list
>>>>> edk2-devel@lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel