BaseTools/ build broken due to unreviewed patch - PLEASE FIX

BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Ard Biesheuvel]

Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f

  "BaseTools: update Test scripts support python3"

[which has never been sent out for review to the mailing list, as far
as I can tell] is breaking the BaseTools build

make -C Tests
make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
/bin/sh: 1: RunTests.py: not found
GNUmakefile:17: recipe for target 'test' failed
make[1]: *** [test] Error 127
make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
GNUmakefile:25: recipe for target 'Tests' failed
make: *** [Tests] Error 2
make: Leaving directory '/home/ard/build/edk2/BaseTools'

I tried reverting that patch alone, but that gives me

make -C Tests
make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
Traceback (most recent call last):
  File "RunTests.py", line 36, in <module>
    allTests = GetAllTestsSuite()
  File "RunTests.py", line 33, in GetAllTestsSuite
    return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
  File "RunTests.py", line 30, in GetPythonTestSuite
    return PythonToolsTests.TheTestSuite()
  File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
line 27, in TheTestSuite
    import CheckUnicodeSourceFiles
  File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
line 25, in <module>
    from Common.Misc import PathClass
  File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
line 27, in <module>
    from collections import UserDict as IterableUserDict
ImportError: cannot import name UserDict
GNUmakefile:17: recipe for target 'test' failed
make[1]: *** [test] Error 1
make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
GNUmakefile:25: recipe for target 'Tests' failed
make: *** [Tests] Error 2
make: Leaving directory '/home/ard/build/edk2/BaseTools'

Breaking the build for everyone with an unreviewed patch 2 days before
the plugfest is NOT OK.

Please fix or revert ASAP

(I suppose that the failing test is a false negative in this case, but
that is not really the point - my CI jobs are broken due to this and
so are many others, I imagine)

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Gao, Liming]

Ard:
  I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But, this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do you install Python36 or Python37 in your machine? 
  
  Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required. 
  

Thanks
Liming
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> Sent: Saturday, October 13, 2018 4:18 PM
> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm
> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> stephano.cetola@linux.intel.com
> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> 
> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> 
>   "BaseTools: update Test scripts support python3"
> 
> [which has never been sent out for review to the mailing list, as far
> as I can tell] is breaking the BaseTools build
> 
> make -C Tests
> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> /bin/sh: 1: RunTests.py: not found
> GNUmakefile:17: recipe for target 'test' failed
> make[1]: *** [test] Error 127
> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> GNUmakefile:25: recipe for target 'Tests' failed
> make: *** [Tests] Error 2
> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> 
> I tried reverting that patch alone, but that gives me
> 
> make -C Tests
> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> Traceback (most recent call last):
>   File "RunTests.py", line 36, in <module>
>     allTests = GetAllTestsSuite()
>   File "RunTests.py", line 33, in GetAllTestsSuite
>     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
>   File "RunTests.py", line 30, in GetPythonTestSuite
>     return PythonToolsTests.TheTestSuite()
>   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> line 27, in TheTestSuite
>     import CheckUnicodeSourceFiles
>   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> line 25, in <module>
>     from Common.Misc import PathClass
>   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> line 27, in <module>
>     from collections import UserDict as IterableUserDict
> ImportError: cannot import name UserDict
> GNUmakefile:17: recipe for target 'test' failed
> make[1]: *** [test] Error 1
> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> GNUmakefile:25: recipe for target 'Tests' failed
> make: *** [Tests] Error 2
> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> 
> Breaking the build for everyone with an unreviewed patch 2 days before
> the plugfest is NOT OK.
> 
> Please fix or revert ASAP
> 
> (I suppose that the failing test is a false negative in this case, but
> that is not really the point - my CI jobs are broken due to this and
> so are many others, I imagine)
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Ard Biesheuvel]

On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
> Ard:
>   I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But, this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do you install Python36 or Python37 in your machine?
>
>   Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
>

I have python3 on my machine, yes. But that does not mean it is the
default for everyone.


>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
>> Sent: Saturday, October 13, 2018 4:18 PM
>> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm
>> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
>> stephano.cetola@linux.intel.com
>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>
>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
>>
>>   "BaseTools: update Test scripts support python3"
>>
>> [which has never been sent out for review to the mailing list, as far
>> as I can tell] is breaking the BaseTools build
>>
>> make -C Tests
>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>> /bin/sh: 1: RunTests.py: not found
>> GNUmakefile:17: recipe for target 'test' failed
>> make[1]: *** [test] Error 127
>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>> GNUmakefile:25: recipe for target 'Tests' failed
>> make: *** [Tests] Error 2
>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>
>> I tried reverting that patch alone, but that gives me
>>
>> make -C Tests
>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>> Traceback (most recent call last):
>>   File "RunTests.py", line 36, in <module>
>>     allTests = GetAllTestsSuite()
>>   File "RunTests.py", line 33, in GetAllTestsSuite
>>     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
>>   File "RunTests.py", line 30, in GetPythonTestSuite
>>     return PythonToolsTests.TheTestSuite()
>>   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
>> line 27, in TheTestSuite
>>     import CheckUnicodeSourceFiles
>>   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
>> line 25, in <module>
>>     from Common.Misc import PathClass
>>   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
>> line 27, in <module>
>>     from collections import UserDict as IterableUserDict
>> ImportError: cannot import name UserDict
>> GNUmakefile:17: recipe for target 'test' failed
>> make[1]: *** [test] Error 1
>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>> GNUmakefile:25: recipe for target 'Tests' failed
>> make: *** [Tests] Error 2
>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>
>> Breaking the build for everyone with an unreviewed patch 2 days before
>> the plugfest is NOT OK.
>>
>> Please fix or revert ASAP
>>
>> (I suppose that the failing test is a false negative in this case, but
>> that is not really the point - my CI jobs are broken due to this and
>> so are many others, I imagine)
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Gao, Liming]

Ard:
  With Python3, do you still meet with the failure?

  On BaseTools Python3 migration, the first patch is sent https://lists.01.org/pipermail/edk2-devel/2018-September/030216.html. And, the check in plan is sent on https://lists.01.org/pipermail/edk2-devel/2018-October/030563.html. There is no reject or concern on this migration. So, we push this change today. And, send the notification https://lists.01.org/pipermail/edk2-devel/2018-October/030722.html, and require edk2 user to install Python3 in the machine. 

Thanks
Liming
> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> Sent: Saturday, October 13, 2018 4:46 PM
> To: Gao, Liming <liming.gao@intel.com>
> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> 
> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
> > Ard:
> >   I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But,
> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do
> you install Python36 or Python37 in your machine?
> >
> >   Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
> >
> 
> I have python3 on my machine, yes. But that does not mean it is the
> default for everyone.
> 
> 
> >> -----Original Message-----
> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> >> Sent: Saturday, October 13, 2018 4:18 PM
> >> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
> Lindholm
> >> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> >> stephano.cetola@linux.intel.com
> >> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>
> >> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> >>
> >>   "BaseTools: update Test scripts support python3"
> >>
> >> [which has never been sent out for review to the mailing list, as far
> >> as I can tell] is breaking the BaseTools build
> >>
> >> make -C Tests
> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >> /bin/sh: 1: RunTests.py: not found
> >> GNUmakefile:17: recipe for target 'test' failed
> >> make[1]: *** [test] Error 127
> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >> GNUmakefile:25: recipe for target 'Tests' failed
> >> make: *** [Tests] Error 2
> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>
> >> I tried reverting that patch alone, but that gives me
> >>
> >> make -C Tests
> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >> Traceback (most recent call last):
> >>   File "RunTests.py", line 36, in <module>
> >>     allTests = GetAllTestsSuite()
> >>   File "RunTests.py", line 33, in GetAllTestsSuite
> >>     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
> >>   File "RunTests.py", line 30, in GetPythonTestSuite
> >>     return PythonToolsTests.TheTestSuite()
> >>   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> >> line 27, in TheTestSuite
> >>     import CheckUnicodeSourceFiles
> >>   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> >> line 25, in <module>
> >>     from Common.Misc import PathClass
> >>   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> >> line 27, in <module>
> >>     from collections import UserDict as IterableUserDict
> >> ImportError: cannot import name UserDict
> >> GNUmakefile:17: recipe for target 'test' failed
> >> make[1]: *** [test] Error 1
> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >> GNUmakefile:25: recipe for target 'Tests' failed
> >> make: *** [Tests] Error 2
> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>
> >> Breaking the build for everyone with an unreviewed patch 2 days before
> >> the plugfest is NOT OK.
> >>
> >> Please fix or revert ASAP
> >>
> >> (I suppose that the failing test is a false negative in this case, but
> >> that is not really the point - my CI jobs are broken due to this and
> >> so are many others, I imagine)
> >> _______________________________________________
> >> edk2-devel mailing list
> >> edk2-devel@lists.01.org
> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Leif Lindholm]

Liming,

The fact that the set includes unreviewed patches[1] should be
sufficient argument for reverting it immediately.

[1] ea787b6b55499e8f931201f8f49daaaeb2e4409f

But beyond that, a set this invasive should only ever be pushed
in the days after a stable tag has been created - not 2/3 through the
period.

This particular one breaks builds using the instructions from
https://github.com/tianocore/tianocore.github.io/wiki/Common-instructions

With the new set, it is mandatory that edksetup.sh is sourced before
building BaseTools. That was not previously the case, meaning existing
CI jobs following the documentation example now fail.

Please revert this set for now. We can revisit and plan a suitable way
forward after that.

Regards,

Leif

On Sat, Oct 13, 2018 at 08:44:39AM +0000, Gao, Liming wrote:
> Ard:
>   I pull the latest edk2 code. I can find RunTests.py in
>   BaseTools/Tests. The below error message shows RunTests.py is not
>   found. But, this file is in BaseTools/Tests. I also run Make
>   command in BaseTools directory. It can work in my Ubuntu
>   machine. I install Python37. Do you install Python36 or Python37
>   in your machine?
>   
>   Yonghong has sent the mail to notify BaseTools Python3 migration
>   is done. Python3.6 or the above is required.
>   
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> > Sent: Saturday, October 13, 2018 4:18 PM
> > To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm
> > <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> > stephano.cetola@linux.intel.com
> > Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> > 
> > Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> > 
> >   "BaseTools: update Test scripts support python3"
> > 
> > [which has never been sent out for review to the mailing list, as far
> > as I can tell] is breaking the BaseTools build
> > 
> > make -C Tests
> > make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> > /bin/sh: 1: RunTests.py: not found
> > GNUmakefile:17: recipe for target 'test' failed
> > make[1]: *** [test] Error 127
> > make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> > GNUmakefile:25: recipe for target 'Tests' failed
> > make: *** [Tests] Error 2
> > make: Leaving directory '/home/ard/build/edk2/BaseTools'
> > 
> > I tried reverting that patch alone, but that gives me
> > 
> > make -C Tests
> > make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> > Traceback (most recent call last):
> >   File "RunTests.py", line 36, in <module>
> >     allTests = GetAllTestsSuite()
> >   File "RunTests.py", line 33, in GetAllTestsSuite
> >     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
> >   File "RunTests.py", line 30, in GetPythonTestSuite
> >     return PythonToolsTests.TheTestSuite()
> >   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> > line 27, in TheTestSuite
> >     import CheckUnicodeSourceFiles
> >   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> > line 25, in <module>
> >     from Common.Misc import PathClass
> >   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> > line 27, in <module>
> >     from collections import UserDict as IterableUserDict
> > ImportError: cannot import name UserDict
> > GNUmakefile:17: recipe for target 'test' failed
> > make[1]: *** [test] Error 1
> > make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> > GNUmakefile:25: recipe for target 'Tests' failed
> > make: *** [Tests] Error 2
> > make: Leaving directory '/home/ard/build/edk2/BaseTools'
> > 
> > Breaking the build for everyone with an unreviewed patch 2 days before
> > the plugfest is NOT OK.
> > 
> > Please fix or revert ASAP
> > 
> > (I suppose that the failing test is a false negative in this case, but
> > that is not really the point - my CI jobs are broken due to this and
> > so are many others, I imagine)
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Gao, Liming]

Ard:
  I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the python3 and set PYTHON3 env. This env is used in GNUmakefile.

  I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle it. 

Thanks
Liming
> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> Sent: Saturday, October 13, 2018 4:46 PM
> To: Gao, Liming <liming.gao@intel.com>
> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> 
> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
> > Ard:
> >   I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But,
> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do
> you install Python36 or Python37 in your machine?
> >
> >   Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
> >
> 
> I have python3 on my machine, yes. But that does not mean it is the
> default for everyone.
> 
> 
> >> -----Original Message-----
> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> >> Sent: Saturday, October 13, 2018 4:18 PM
> >> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
> Lindholm
> >> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> >> stephano.cetola@linux.intel.com
> >> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>
> >> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> >>
> >>   "BaseTools: update Test scripts support python3"
> >>
> >> [which has never been sent out for review to the mailing list, as far
> >> as I can tell] is breaking the BaseTools build
> >>
> >> make -C Tests
> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >> /bin/sh: 1: RunTests.py: not found
> >> GNUmakefile:17: recipe for target 'test' failed
> >> make[1]: *** [test] Error 127
> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >> GNUmakefile:25: recipe for target 'Tests' failed
> >> make: *** [Tests] Error 2
> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>
> >> I tried reverting that patch alone, but that gives me
> >>
> >> make -C Tests
> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >> Traceback (most recent call last):
> >>   File "RunTests.py", line 36, in <module>
> >>     allTests = GetAllTestsSuite()
> >>   File "RunTests.py", line 33, in GetAllTestsSuite
> >>     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
> >>   File "RunTests.py", line 30, in GetPythonTestSuite
> >>     return PythonToolsTests.TheTestSuite()
> >>   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> >> line 27, in TheTestSuite
> >>     import CheckUnicodeSourceFiles
> >>   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> >> line 25, in <module>
> >>     from Common.Misc import PathClass
> >>   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> >> line 27, in <module>
> >>     from collections import UserDict as IterableUserDict
> >> ImportError: cannot import name UserDict
> >> GNUmakefile:17: recipe for target 'test' failed
> >> make[1]: *** [test] Error 1
> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >> GNUmakefile:25: recipe for target 'Tests' failed
> >> make: *** [Tests] Error 2
> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>
> >> Breaking the build for everyone with an unreviewed patch 2 days before
> >> the plugfest is NOT OK.
> >>
> >> Please fix or revert ASAP
> >>
> >> (I suppose that the failing test is a false negative in this case, but
> >> that is not really the point - my CI jobs are broken due to this and
> >> so are many others, I imagine)
> >> _______________________________________________
> >> edk2-devel mailing list
> >> edk2-devel@lists.01.org
> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Ard Biesheuvel]

On 13 October 2018 at 11:23, Gao, Liming <liming.gao@intel.com> wrote:
> Ard:
>   I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the python3 and set PYTHON3 env. This env is used in GNUmakefile.
>
>   I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle it.
>

Thanks for diagnosing that. As Leif points out, this deviates from the
instructions on the wiki. It also deviates from the build script I use
in my CI environment, hence the breakage.

Unfortunately, our CI environment is based on Debian Stretch, which
provides Python 3.5.3 only not Python 3.6 or later. I will work with
our infrastructure team to find out if we can support Python 3.6 as
well, but it will probably take time.

What is the reason we need 3.6 or later?


>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>> Sent: Saturday, October 13, 2018 4:46 PM
>> To: Gao, Liming <liming.gao@intel.com>
>> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
>> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>
>> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
>> > Ard:
>> >   I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But,
>> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do
>> you install Python36 or Python37 in your machine?
>> >
>> >   Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
>> >
>>
>> I have python3 on my machine, yes. But that does not mean it is the
>> default for everyone.
>>
>>
>> >> -----Original Message-----
>> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
>> >> Sent: Saturday, October 13, 2018 4:18 PM
>> >> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
>> Lindholm
>> >> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
>> >> stephano.cetola@linux.intel.com
>> >> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>> >>
>> >> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
>> >>
>> >>   "BaseTools: update Test scripts support python3"
>> >>
>> >> [which has never been sent out for review to the mailing list, as far
>> >> as I can tell] is breaking the BaseTools build
>> >>
>> >> make -C Tests
>> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>> >> /bin/sh: 1: RunTests.py: not found
>> >> GNUmakefile:17: recipe for target 'test' failed
>> >> make[1]: *** [test] Error 127
>> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>> >> GNUmakefile:25: recipe for target 'Tests' failed
>> >> make: *** [Tests] Error 2
>> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>> >>
>> >> I tried reverting that patch alone, but that gives me
>> >>
>> >> make -C Tests
>> >> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>> >> Traceback (most recent call last):
>> >>   File "RunTests.py", line 36, in <module>
>> >>     allTests = GetAllTestsSuite()
>> >>   File "RunTests.py", line 33, in GetAllTestsSuite
>> >>     return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
>> >>   File "RunTests.py", line 30, in GetPythonTestSuite
>> >>     return PythonToolsTests.TheTestSuite()
>> >>   File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
>> >> line 27, in TheTestSuite
>> >>     import CheckUnicodeSourceFiles
>> >>   File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
>> >> line 25, in <module>
>> >>     from Common.Misc import PathClass
>> >>   File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
>> >> line 27, in <module>
>> >>     from collections import UserDict as IterableUserDict
>> >> ImportError: cannot import name UserDict
>> >> GNUmakefile:17: recipe for target 'test' failed
>> >> make[1]: *** [test] Error 1
>> >> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>> >> GNUmakefile:25: recipe for target 'Tests' failed
>> >> make: *** [Tests] Error 2
>> >> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>> >>
>> >> Breaking the build for everyone with an unreviewed patch 2 days before
>> >> the plugfest is NOT OK.
>> >>
>> >> Please fix or revert ASAP
>> >>
>> >> (I suppose that the failing test is a false negative in this case, but
>> >> that is not really the point - my CI jobs are broken due to this and
>> >> so are many others, I imagine)
>> >> _______________________________________________
>> >> edk2-devel mailing list
>> >> edk2-devel@lists.01.org
>> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Andrew Fish]

> On Oct 13, 2018, at 5:18 AM, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> 
> On 13 October 2018 at 11:23, Gao, Liming <liming.gao@intel.com> wrote:
>> Ard:
>>  I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the python3 and set PYTHON3 env. This env is used in GNUmakefile.
>> 
>>  I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle it.
>> 
> 
> Thanks for diagnosing that. As Leif points out, this deviates from the
> instructions on the wiki. It also deviates from the build script I use
> in my CI environment, hence the breakage.
> 
> Unfortunately, our CI environment is based on Debian Stretch, which
> provides Python 3.5.3 only not Python 3.6 or later. I will work with
> our infrastructure team to find out if we can support Python 3.6 as
> well, but it will probably take time.
> 
> What is the reason we need 3.6 or later?
> 

All the macOS based builds are broken too. Python 2.7.* comes with macOS. The instructions are broken for macOS as you never had to install Python before. 

I'm not able to install an alternate version of Python in our internal CI infrastructure (You build the OS with tools from the OS). This will likely force me to have to check in a version of Python 3.6 into the edk2 tree to keep the build working, or fork off an older version of the build system that works with Python 2.7.*. 

What problem are we solving by forcing everyone to be on Python 3.6 or later? I understand on Widows you have to install Python, but a lot of the Unix installs you can get an older version of Python installed for free. 

Thanks,

Andrew Fish

> 
>>> -----Original Message-----
>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>>> Sent: Saturday, October 13, 2018 4:46 PM
>>> To: Gao, Liming <liming.gao@intel.com>
>>> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
>>> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
>>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>> 
>>> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
>>>> Ard:
>>>>  I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found. But,
>>> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37. Do
>>> you install Python36 or Python37 in your machine?
>>>> 
>>>>  Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
>>>> 
>>> 
>>> I have python3 on my machine, yes. But that does not mean it is the
>>> default for everyone.
>>> 
>>> 
>>>>> -----Original Message-----
>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
>>>>> Sent: Saturday, October 13, 2018 4:18 PM
>>>>> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
>>> Lindholm
>>>>> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
>>>>> stephano.cetola@linux.intel.com
>>>>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
>>>>> 
>>>>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
>>>>> 
>>>>>  "BaseTools: update Test scripts support python3"
>>>>> 
>>>>> [which has never been sent out for review to the mailing list, as far
>>>>> as I can tell] is breaking the BaseTools build
>>>>> 
>>>>> make -C Tests
>>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> /bin/sh: 1: RunTests.py: not found
>>>>> GNUmakefile:17: recipe for target 'test' failed
>>>>> make[1]: *** [test] Error 127
>>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> GNUmakefile:25: recipe for target 'Tests' failed
>>>>> make: *** [Tests] Error 2
>>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>>>> 
>>>>> I tried reverting that patch alone, but that gives me
>>>>> 
>>>>> make -C Tests
>>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> Traceback (most recent call last):
>>>>>  File "RunTests.py", line 36, in <module>
>>>>>    allTests = GetAllTestsSuite()
>>>>>  File "RunTests.py", line 33, in GetAllTestsSuite
>>>>>    return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
>>>>>  File "RunTests.py", line 30, in GetPythonTestSuite
>>>>>    return PythonToolsTests.TheTestSuite()
>>>>>  File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
>>>>> line 27, in TheTestSuite
>>>>>    import CheckUnicodeSourceFiles
>>>>>  File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
>>>>> line 25, in <module>
>>>>>    from Common.Misc import PathClass
>>>>>  File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
>>>>> line 27, in <module>
>>>>>    from collections import UserDict as IterableUserDict
>>>>> ImportError: cannot import name UserDict
>>>>> GNUmakefile:17: recipe for target 'test' failed
>>>>> make[1]: *** [test] Error 1
>>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
>>>>> GNUmakefile:25: recipe for target 'Tests' failed
>>>>> make: *** [Tests] Error 2
>>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
>>>>> 
>>>>> Breaking the build for everyone with an unreviewed patch 2 days before
>>>>> the plugfest is NOT OK.
>>>>> 
>>>>> Please fix or revert ASAP
>>>>> 
>>>>> (I suppose that the failing test is a false negative in this case, but
>>>>> that is not really the point - my CI jobs are broken due to this and
>>>>> so are many others, I imagine)
>>>>> _______________________________________________
>>>>> edk2-devel mailing list
>>>>> edk2-devel@lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX

[Gao, Liming]

https://pythonclock.org/ say Python 2.7 will not be maintained past 2020. One BZ https://bugzilla.tianocore.org/show_bug.cgi?id=55 also requests this migration. So, we migrate BaseTools to support Python3. Because Python27 will not be maintained, we drop Python27 support when we do Python3 migration. In fact, OS supports more than one python tool. In Linux or Mac OS, after you install Python3, you will have "python3" command to trig python3. "python" command to trig python2. In Windows, after you install Python3, there will be py -3 to trig python3, and py -2 to trig python2. You can still use python2 for python2 script. BaseTools will auto find python3 command to call BaseTools script. User only needs to install Python3. It has no impact on other project. 

Why we choose Python3.6 or above? Python3.6 is relative new python version. We choose it to validate BaseTools Python3 functionality. In fact, current BaseTools may work with Python35 or Python34. But, we don't do full validation. So, we don't mention them. We can do basic test for other python3 version. If no big block issue, we can relax this requirement. And, Python37 is new version that does performance optimization. We collect performance data in Windows OS, Python37 improves 20% performance in BaseTools AutoGen phase than Python27. Python36 performance is bad. It decreases 20% performance than Python27. I get the information that early Python3 version performance is not good. So, I suggest to use Python37 for better build performance.

On break issue, I admit this is our mistake. We don't verify Wiki work flow. We need to fix it ASAP. And, I admit Yonghong breaks patch review process. One patch is not reviewed in mail list and pushed. We need to strictly follow the process. Thank you to catch it. 

> -----Original Message-----
> From: afish@apple.com [mailto:afish@apple.com]
> Sent: Saturday, October 13, 2018 11:12 PM
> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Gao, Liming <liming.gao@intel.com>; edk2-devel-01 <edk2-devel@lists.01.org>; Kinney, Michael D <michael.d.kinney@intel.com>;
> Laszlo Ersek <lersek@redhat.com>
> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> 
> 
> 
> > On Oct 13, 2018, at 5:18 AM, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >
> > On 13 October 2018 at 11:23, Gao, Liming <liming.gao@intel.com> wrote:
> >> Ard:
> >>  I know the failure reason. . edksetup.sh must be called in edk2 before enter BaseTools with make. edksetup.sh is used to detect the
> python3 and set PYTHON3 env. This env is used in GNUmakefile.
> >>
> >>  I don't aware this change before. Because we usually call edksetup.sh, then make BaseTools. We will try to find the way to handle
> it.
> >>
> >
> > Thanks for diagnosing that. As Leif points out, this deviates from the
> > instructions on the wiki. It also deviates from the build script I use
> > in my CI environment, hence the breakage.
> >
> > Unfortunately, our CI environment is based on Debian Stretch, which
> > provides Python 3.5.3 only not Python 3.6 or later. I will work with
> > our infrastructure team to find out if we can support Python 3.6 as
> > well, but it will probably take time.
> >
> > What is the reason we need 3.6 or later?
> >
> 
> All the macOS based builds are broken too. Python 2.7.* comes with macOS. The instructions are broken for macOS as you never had to
> install Python before.
> 
> I'm not able to install an alternate version of Python in our internal CI infrastructure (You build the OS with tools from the OS). This will
> likely force me to have to check in a version of Python 3.6 into the edk2 tree to keep the build working, or fork off an older version of the
> build system that works with Python 2.7.*.
> 
> What problem are we solving by forcing everyone to be on Python 3.6 or later? I understand on Widows you have to install Python, but a
> lot of the Unix installs you can get an older version of Python installed for free.
> 
> Thanks,
> 
> Andrew Fish
> 
> >
> >>> -----Original Message-----
> >>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> >>> Sent: Saturday, October 13, 2018 4:46 PM
> >>> To: Gao, Liming <liming.gao@intel.com>
> >>> Cc: edk2-devel-01 <edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com>; Leif Lindholm <leif.lindholm@linaro.org>; Kinney,
> >>> Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>; stephano.cetola@linux.intel.com
> >>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>>
> >>> On 13 October 2018 at 10:44, Gao, Liming <liming.gao@intel.com> wrote:
> >>>> Ard:
> >>>>  I pull the latest edk2 code. I can find RunTests.py in BaseTools/Tests. The below error message shows RunTests.py is not found.
> But,
> >>> this file is in BaseTools/Tests. I also run Make command in BaseTools directory. It can work in my Ubuntu machine. I install Python37.
> Do
> >>> you install Python36 or Python37 in your machine?
> >>>>
> >>>>  Yonghong has sent the mail to notify BaseTools Python3 migration is done. Python3.6 or the above is required.
> >>>>
> >>>
> >>> I have python3 on my machine, yes. But that does not mean it is the
> >>> default for everyone.
> >>>
> >>>
> >>>>> -----Original Message-----
> >>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Ard Biesheuvel
> >>>>> Sent: Saturday, October 13, 2018 4:18 PM
> >>>>> To: edk2-devel-01 <edk2-devel@lists.01.org>; Gao, Liming <liming.gao@intel.com>; Laszlo Ersek <lersek@redhat.com>; Leif
> >>> Lindholm
> >>>>> <leif.lindholm@linaro.org>; Kinney, Michael D <michael.d.kinney@intel.com>; Zhu, Yonghong <yonghong.zhu@intel.com>;
> >>>>> stephano.cetola@linux.intel.com
> >>>>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX
> >>>>>
> >>>>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f
> >>>>>
> >>>>>  "BaseTools: update Test scripts support python3"
> >>>>>
> >>>>> [which has never been sent out for review to the mailing list, as far
> >>>>> as I can tell] is breaking the BaseTools build
> >>>>>
> >>>>> make -C Tests
> >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> /bin/sh: 1: RunTests.py: not found
> >>>>> GNUmakefile:17: recipe for target 'test' failed
> >>>>> make[1]: *** [test] Error 127
> >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> GNUmakefile:25: recipe for target 'Tests' failed
> >>>>> make: *** [Tests] Error 2
> >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>>>>
> >>>>> I tried reverting that patch alone, but that gives me
> >>>>>
> >>>>> make -C Tests
> >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> Traceback (most recent call last):
> >>>>>  File "RunTests.py", line 36, in <module>
> >>>>>    allTests = GetAllTestsSuite()
> >>>>>  File "RunTests.py", line 33, in GetAllTestsSuite
> >>>>>    return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()])
> >>>>>  File "RunTests.py", line 30, in GetPythonTestSuite
> >>>>>    return PythonToolsTests.TheTestSuite()
> >>>>>  File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py",
> >>>>> line 27, in TheTestSuite
> >>>>>    import CheckUnicodeSourceFiles
> >>>>>  File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles.py",
> >>>>> line 25, in <module>
> >>>>>    from Common.Misc import PathClass
> >>>>>  File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py",
> >>>>> line 27, in <module>
> >>>>>    from collections import UserDict as IterableUserDict
> >>>>> ImportError: cannot import name UserDict
> >>>>> GNUmakefile:17: recipe for target 'test' failed
> >>>>> make[1]: *** [test] Error 1
> >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests'
> >>>>> GNUmakefile:25: recipe for target 'Tests' failed
> >>>>> make: *** [Tests] Error 2
> >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools'
> >>>>>
> >>>>> Breaking the build for everyone with an unreviewed patch 2 days before
> >>>>> the plugfest is NOT OK.
> >>>>>
> >>>>> Please fix or revert ASAP
> >>>>>
> >>>>> (I suppose that the failing test is a false negative in this case, but
> >>>>> that is not really the point - my CI jobs are broken due to this and
> >>>>> so are many others, I imagine)
> >>>>> _______________________________________________
> >>>>> edk2-devel mailing list
> >>>>> edk2-devel@lists.01.org
> >>>>> https://lists.01.org/mailman/listinfo/edk2-devel
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel