From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.24; helo=mga09.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id EED4521173C8B for ; Sat, 13 Oct 2018 08:48:50 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Oct 2018 08:48:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,377,1534834800"; d="scan'208";a="98931138" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga001.fm.intel.com with ESMTP; 13 Oct 2018 08:48:49 -0700 Received: from fmsmsx112.amr.corp.intel.com (10.18.116.6) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.319.2; Sat, 13 Oct 2018 08:48:49 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by FMSMSX112.amr.corp.intel.com (10.18.116.6) with Microsoft SMTP Server (TLS) id 14.3.319.2; Sat, 13 Oct 2018 08:48:49 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.48]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.245]) with mapi id 14.03.0319.002; Sat, 13 Oct 2018 23:48:47 +0800 From: "Gao, Liming" To: "afish@apple.com" , Ard Biesheuvel CC: edk2-devel-01 , "Kinney, Michael D" , Laszlo Ersek Thread-Topic: [edk2] BaseTools/ build broken due to unreviewed patch - PLEASE FIX Thread-Index: AQHUYs1GqkKzZr7J30KIQycJKcKZ0KUc2kNQ//98doCAAI2hMP//rZkAgAAwlgCAAIcxwA== Date: Sat, 13 Oct 2018 15:48:46 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E3399CE@SHSMSX104.ccr.corp.intel.com> References: <4A89E2EF3DFEDB4C8BFDE51014F606A14E339880@SHSMSX104.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E339900@SHSMSX104.ccr.corp.intel.com> <0A85438F-73CF-4ACC-9710-6157A27B18F5@apple.com> In-Reply-To: <0A85438F-73CF-4ACC-9710-6157A27B18F5@apple.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZmRlMzVhMjktYTUwNS00N2I3LTljMmEtNjFhZmU0MTBmNTljIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiSmVwRDhxSWpyR0dSMFdUSlM0anlvS0swR0o3MlpJXC9TYXN5XC9TQ0NOdjNqaUNcL3VwdjNRNndsbk9STGp4c1hrSCJ9 dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: BaseTools/ build broken due to unreviewed patch - PLEASE FIX X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Oct 2018 15:48:51 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable https://pythonclock.org/ say Python 2.7 will not be maintained past 2020. O= ne BZ https://bugzilla.tianocore.org/show_bug.cgi?id=3D55 also requests thi= s migration. So, we migrate BaseTools to support Python3. Because Python27 = will not be maintained, we drop Python27 support when we do Python3 migrati= on. In fact, OS supports more than one python tool. In Linux or Mac OS, aft= er you install Python3, you will have "python3" command to trig python3. "p= ython" command to trig python2. In Windows, after you install Python3, ther= e will be py -3 to trig python3, and py -2 to trig python2. You can still u= se python2 for python2 script. BaseTools will auto find python3 command to = call BaseTools script. User only needs to install Python3. It has no impact= on other project.=20 Why we choose Python3.6 or above? Python3.6 is relative new python version.= We choose it to validate BaseTools Python3 functionality. In fact, current= BaseTools may work with Python35 or Python34. But, we don't do full valida= tion. So, we don't mention them. We can do basic test for other python3 ver= sion. If no big block issue, we can relax this requirement. And, Python37 i= s new version that does performance optimization. We collect performance da= ta in Windows OS, Python37 improves 20% performance in BaseTools AutoGen ph= ase than Python27. Python36 performance is bad. It decreases 20% performanc= e than Python27. I get the information that early Python3 version performan= ce is not good. So, I suggest to use Python37 for better build performance. On break issue, I admit this is our mistake. We don't verify Wiki work flow= . We need to fix it ASAP. And, I admit Yonghong breaks patch review process= . One patch is not reviewed in mail list and pushed. We need to strictly fo= llow the process. Thank you to catch it.=20 > -----Original Message----- > From: afish@apple.com [mailto:afish@apple.com] > Sent: Saturday, October 13, 2018 11:12 PM > To: Ard Biesheuvel > Cc: Gao, Liming ; edk2-devel-01 ; Kinney, Michael D ; > Laszlo Ersek > Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch - PLE= ASE FIX >=20 >=20 >=20 > > On Oct 13, 2018, at 5:18 AM, Ard Biesheuvel = wrote: > > > > On 13 October 2018 at 11:23, Gao, Liming wrote: > >> Ard: > >> I know the failure reason. . edksetup.sh must be called in edk2 befor= e enter BaseTools with make. edksetup.sh is used to detect the > python3 and set PYTHON3 env. This env is used in GNUmakefile. > >> > >> I don't aware this change before. Because we usually call edksetup.sh= , then make BaseTools. We will try to find the way to handle > it. > >> > > > > Thanks for diagnosing that. As Leif points out, this deviates from the > > instructions on the wiki. It also deviates from the build script I use > > in my CI environment, hence the breakage. > > > > Unfortunately, our CI environment is based on Debian Stretch, which > > provides Python 3.5.3 only not Python 3.6 or later. I will work with > > our infrastructure team to find out if we can support Python 3.6 as > > well, but it will probably take time. > > > > What is the reason we need 3.6 or later? > > >=20 > All the macOS based builds are broken too. Python 2.7.* comes with macOS.= The instructions are broken for macOS as you never had to > install Python before. >=20 > I'm not able to install an alternate version of Python in our internal CI= infrastructure (You build the OS with tools from the OS). This will > likely force me to have to check in a version of Python 3.6 into the edk2= tree to keep the build working, or fork off an older version of the > build system that works with Python 2.7.*. >=20 > What problem are we solving by forcing everyone to be on Python 3.6 or la= ter? I understand on Widows you have to install Python, but a > lot of the Unix installs you can get an older version of Python installed= for free. >=20 > Thanks, >=20 > Andrew Fish >=20 > > > >>> -----Original Message----- > >>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org] > >>> Sent: Saturday, October 13, 2018 4:46 PM > >>> To: Gao, Liming > >>> Cc: edk2-devel-01 ; Laszlo Ersek ; Leif Lindholm ; Kinney, > >>> Michael D ; Zhu, Yonghong ; stephano.cetola@linux.intel.com > >>> Subject: Re: [edk2] BaseTools/ build broken due to unreviewed patch -= PLEASE FIX > >>> > >>> On 13 October 2018 at 10:44, Gao, Liming wrote= : > >>>> Ard: > >>>> I pull the latest edk2 code. I can find RunTests.py in BaseTools/Te= sts. The below error message shows RunTests.py is not found. > But, > >>> this file is in BaseTools/Tests. I also run Make command in BaseTools= directory. It can work in my Ubuntu machine. I install Python37. > Do > >>> you install Python36 or Python37 in your machine? > >>>> > >>>> Yonghong has sent the mail to notify BaseTools Python3 migration is= done. Python3.6 or the above is required. > >>>> > >>> > >>> I have python3 on my machine, yes. But that does not mean it is the > >>> default for everyone. > >>> > >>> > >>>>> -----Original Message----- > >>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf= Of Ard Biesheuvel > >>>>> Sent: Saturday, October 13, 2018 4:18 PM > >>>>> To: edk2-devel-01 ; Gao, Liming ; Laszlo Ersek ; Leif > >>> Lindholm > >>>>> ; Kinney, Michael D ; Zhu, Yonghong ; > >>>>> stephano.cetola@linux.intel.com > >>>>> Subject: [edk2] BaseTools/ build broken due to unreviewed patch - P= LEASE FIX > >>>>> > >>>>> Commit ea787b6b55499e8f931201f8f49daaaeb2e4409f > >>>>> > >>>>> "BaseTools: update Test scripts support python3" > >>>>> > >>>>> [which has never been sent out for review to the mailing list, as f= ar > >>>>> as I can tell] is breaking the BaseTools build > >>>>> > >>>>> make -C Tests > >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests' > >>>>> /bin/sh: 1: RunTests.py: not found > >>>>> GNUmakefile:17: recipe for target 'test' failed > >>>>> make[1]: *** [test] Error 127 > >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests' > >>>>> GNUmakefile:25: recipe for target 'Tests' failed > >>>>> make: *** [Tests] Error 2 > >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools' > >>>>> > >>>>> I tried reverting that patch alone, but that gives me > >>>>> > >>>>> make -C Tests > >>>>> make[1]: Entering directory '/home/ard/build/edk2/BaseTools/Tests' > >>>>> Traceback (most recent call last): > >>>>> File "RunTests.py", line 36, in > >>>>> allTests =3D GetAllTestsSuite() > >>>>> File "RunTests.py", line 33, in GetAllTestsSuite > >>>>> return unittest.TestSuite([GetCTestSuite(), GetPythonTestSuite()= ]) > >>>>> File "RunTests.py", line 30, in GetPythonTestSuite > >>>>> return PythonToolsTests.TheTestSuite() > >>>>> File "/home/ard/build/edk2/BaseTools/Tests/PythonToolsTests.py", > >>>>> line 27, in TheTestSuite > >>>>> import CheckUnicodeSourceFiles > >>>>> File "/home/ard/build/edk2/BaseTools/Tests/CheckUnicodeSourceFiles= .py", > >>>>> line 25, in > >>>>> from Common.Misc import PathClass > >>>>> File "/home/ard/build/edk2/BaseTools/Source/Python/Common/Misc.py"= , > >>>>> line 27, in > >>>>> from collections import UserDict as IterableUserDict > >>>>> ImportError: cannot import name UserDict > >>>>> GNUmakefile:17: recipe for target 'test' failed > >>>>> make[1]: *** [test] Error 1 > >>>>> make[1]: Leaving directory '/home/ard/build/edk2/BaseTools/Tests' > >>>>> GNUmakefile:25: recipe for target 'Tests' failed > >>>>> make: *** [Tests] Error 2 > >>>>> make: Leaving directory '/home/ard/build/edk2/BaseTools' > >>>>> > >>>>> Breaking the build for everyone with an unreviewed patch 2 days bef= ore > >>>>> the plugfest is NOT OK. > >>>>> > >>>>> Please fix or revert ASAP > >>>>> > >>>>> (I suppose that the failing test is a false negative in this case, = but > >>>>> that is not really the point - my CI jobs are broken due to this an= d > >>>>> so are many others, I imagine) > >>>>> _______________________________________________ > >>>>> edk2-devel mailing list > >>>>> edk2-devel@lists.01.org > >>>>> https://lists.01.org/mailman/listinfo/edk2-devel > > _______________________________________________ > > edk2-devel mailing list > > edk2-devel@lists.01.org > > https://lists.01.org/mailman/listinfo/edk2-devel