[RFC] Edk2 BaseTools Python3 Migration Update

[RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Hi, all
  On Python3 migration https://bugzilla.tianocore.org/show_bug.cgi?id=55, we update Edk2 BaseTools python source code with the compatible syntax to support Python2 and Python3 both. Here is code https://github.com/lgao4/edk2/tree/Python3 for dry run. To enable Python3, you just need to set PYTHON3_ENABLE environment as TRUE, then type edksetup.bat/edksetup.sh. Without this setting, BaseTools still run with Python2. So, there is no change for current usage model with Python27.

  But, we have no enough resource to fully verify Python2 and Python3 both. We will focus on Python3 validation. If anyone can help verify Python2, it will be great. And, if you meet with the issue on Python2, please file BZ. We still fix them.

Thanks
Liming

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 12/25/18 08:50, Gao, Liming wrote:
> Hi, all

> On Python3 migration
> https://bugzilla.tianocore.org/show_bug.cgi?id=55, we update Edk2
> BaseTools python source code with the compatible syntax to support
> Python2 and Python3 both. Here is code
> https://github.com/lgao4/edk2/tree/Python3 for dry run. To enable
> Python3, you just need to set PYTHON3_ENABLE environment as TRUE, then
> type edksetup.bat/edksetup.sh.

Hopefully I can test this sometime after I return from my PTO. No
promises as to when, for now.

> Without this setting, BaseTools still run with Python2. So, there is
> no change for current usage model with Python27.
>
> But, we have no enough resource to fully verify Python2 and Python3
> both. We will focus on Python3 validation. If anyone can help verify
> Python2, it will be great. And, if you meet with the issue on Python2,
> please file BZ. We still fix them.

Sounds good to me. I expect that I'll keep building upstream OVMF and
ArmVirtQemu on my RHEL7 laptop for a good while, using Python 2.

Thanks!
Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Lin:
   Thanks for your verification.  This issue has been fixed in the latest version of https://github.com/lgao4/edk2/tree/Python3. Could you try again?

Thanks
Liming
>-----Original Message-----
>From: Gary Lin [mailto:glin@suse.com]
>Sent: Friday, December 28, 2018 6:40 PM
>To: Gao, Liming <liming.gao@intel.com>
>Cc: edk2-devel@lists.01.org; Kinney, Michael D
><michael.d.kinney@intel.com>; Laszlo Ersek (lersek@redhat.com)
><lersek@redhat.com>
>Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>
>On Tue, Dec 25, 2018 at 07:50:43AM +0000, Gao, Liming wrote:
>> Hi, all
>>   On Python3 migration https://bugzilla.tianocore.org/show_bug.cgi?id=55,
>we update Edk2 BaseTools python source code with the compatible syntax to
>support Python2 and Python3 both. Here is code
>https://github.com/lgao4/edk2/tree/Python3 for dry run. To enable Python3,
>you just need to set PYTHON3_ENABLE environment as TRUE, then type
>edksetup.bat/edksetup.sh. Without this setting, BaseTools still run with
>Python2. So, there is no change for current usage model with Python27.
>>
>I've built OVMF with both python2 and python3 based on the branch and
>the VM crashed immediately. The only message in the debug log is:
>
>SecCoreStartupWithStack(0xFFFCC000, 0x820000)
>
>The result of bisect showed the first bad commit is
>
> 500aad1a02c5a8c0f208c9422cce19de7d304faa
> BaseTools: Update windows and linux run scripts file to use Python3
>
>Cheers,
>
>Gary Lin
>
>>   But, we have no enough resource to fully verify Python2 and Python3 both.
>We will focus on Python3 validation. If anyone can help verify Python2, it will
>be great. And, if you meet with the issue on Python2, please file BZ. We still
>fix them.
>>
>> Thanks
>> Liming
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>>

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 12/29/18 07:07, Gao, Liming wrote:
> Lin:
>    Thanks for your verification.  This issue has been fixed in the
>    latest version of https://github.com/lgao4/edk2/tree/Python3. Could
>    you try again?

At commit 4614985ec223 ("BaseTools:The binary data type is incorrect",
2018-12-29):


(1) I tried to build OVMF as follows (using
python-2.7.5-69.el7_5.x86_64):

  build \
    -a IA32 \
    -p OvmfPkg/OvmfPkgIa32.dsc \
    -D SMM_REQUIRE \
    -D SECURE_BOOT_ENABLE \
    -D TLS_ENABLE \
    -D NETWORK_IP6_ENABLE \
    -t GCC48 \
    -n 4 \
    --report-file=/tmp/build.ovmf.32.report \
    --log=/tmp/build.ovmf.32.log \
    -b NOOPT \
    -D HTTP_BOOT_ENABLE \
    --cmd-len=65536 \
    --hash \
    --genfds-multi-thread

This produced messages like:

> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to read report file
> BuildReport.py...
> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to read report file

near the end of the build.


(2) After I removed the "--report-file" switch, the warnings
disappeared.

However, neither of the expected output files

- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd
- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd

existed. To be more precise, none of "OVMF.fd" and "OVMF_CODE.fd"
existed anywhere in the "Build" directory.

Judged from the build log, it seemed that at least some *.efi modules
were compiled and linked, but FVs and FDs were not built. The following
sections of the log were missing:

> Fd File Name:OVMF_VARS (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_VARS.fd)
>
> Generate Region at Offset 0x0
>    Region Size = 0x40000
>    Region Name = DATA
>
> Generate Region at Offset 0x40000
>    Region Size = 0x1000
>    Region Name = None
>
> Generate Region at Offset 0x41000
>    Region Size = 0x1000
>    Region Name = DATA
>
> Generate Region at Offset 0x42000
>    Region Size = 0x42000
>    Region Name = None
>
> Fd File Name:MEMFD (.../Build/OvmfIa32/NOOPT_GCC48/FV/MEMFD.fd)
>
> Generate Region at Offset 0x0
>    Region Size = 0x6000
>    Region Name = None
>
> Generate Region at Offset 0x6000
>    Region Size = 0x1000
>    Region Name = None
>
> Generate Region at Offset 0x7000
>    Region Size = 0x1000
>    Region Name = None
> Padding region starting from offset 0x8000, with size 0x8000
>
> Generate Region at Offset 0x8000
>    Region Size = 0x8000
>    Region Name = None
>
> Generate Region at Offset 0x10000
>    Region Size = 0x10000
>    Region Name = None
>
> Generate Region at Offset 0x20000
>    Region Size = 0xE0000
>    Region Name = FV
>
> Generating PEIFV FV
>
> Generate Region at Offset 0x100000
>    Region Size = 0xB00000
>    Region Name = FV
>
> Generating DXEFV FV
>
> Fd File Name:OVMF (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd)
>
> Generate Region at Offset 0x0
>    Region Size = 0x40000
>    Region Name = DATA
>
> Generate Region at Offset 0x40000
>    Region Size = 0x1000
>    Region Name = None
>
> Generate Region at Offset 0x41000
>    Region Size = 0x1000
>    Region Name = DATA
>
> Generate Region at Offset 0x42000
>    Region Size = 0x42000
>    Region Name = None
>
> Generate Region at Offset 0x84000
>    Region Size = 0x348000
>    Region Name = FV
>
> Generating FVMAIN_COMPACT FV
>
> Generate Region at Offset 0x3CC000
>    Region Size = 0x34000
>    Region Name = FV
>
> Generating SECFV FV
>
> Fd File Name:OVMF_CODE (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd)
>
> Generate Region at Offset 0x0
>    Region Size = 0x348000
>    Region Name = FV
>
> Generate Region at Offset 0x348000
>    Region Size = 0x34000
>    Region Name = FV

and

> FV Space Information
> SECFV [19%Full] 212992 total, 42208 used, 170784 free
> FVMAIN_COMPACT [52%Full] 3440640 total, 1820408 used, 1620232 free
> DXEFV [86%Full] 11534336 total, 9928672 used, 1605664 free
> PEIFV [43%Full] 917504 total, 395944 used, 521560 free
> Build report can be found at .../build.ovmf.32.report

Interestingly, the line

> GUID cross reference file can be found at .../Build/OvmfIa32/NOOPT_GCC48/FV/Guid.xref

which is normally printed between the two listings above, was still
printed, but alone.

Thanks,
Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Laszlo:
  Thanks for your report. I verify the same command in windows OS with VS2015x86. It can pass. But, it fail in Linux OS with GCC5 tool chain.  We will check it. 

Thanks
Liming
>-----Original Message-----
>From: Laszlo Ersek [mailto:lersek@redhat.com]
>Sent: Monday, December 31, 2018 8:16 AM
>To: Gao, Liming <liming.gao@intel.com>; Gary Lin <glin@suse.com>
>Cc: edk2-devel@lists.01.org; Kinney, Michael D <michael.d.kinney@intel.com>
>Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>
>On 12/29/18 07:07, Gao, Liming wrote:
>> Lin:
>>    Thanks for your verification.  This issue has been fixed in the
>>    latest version of https://github.com/lgao4/edk2/tree/Python3. Could
>>    you try again?
>
>At commit 4614985ec223 ("BaseTools:The binary data type is incorrect",
>2018-12-29):
>
>
>(1) I tried to build OVMF as follows (using
>python-2.7.5-69.el7_5.x86_64):
>
>  build \
>    -a IA32 \
>    -p OvmfPkg/OvmfPkgIa32.dsc \
>    -D SMM_REQUIRE \
>    -D SECURE_BOOT_ENABLE \
>    -D TLS_ENABLE \
>    -D NETWORK_IP6_ENABLE \
>    -t GCC48 \
>    -n 4 \
>    --report-file=/tmp/build.ovmf.32.report \
>    --log=/tmp/build.ovmf.32.log \
>    -b NOOPT \
>    -D HTTP_BOOT_ENABLE \
>    --cmd-len=65536 \
>    --hash \
>    --genfds-multi-thread
>
>This produced messages like:
>
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>warning: Fail to read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>warning: Fail to read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>
>near the end of the build.
>
>
>(2) After I removed the "--report-file" switch, the warnings
>disappeared.
>
>However, neither of the expected output files
>
>- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd
>- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd
>
>existed. To be more precise, none of "OVMF.fd" and "OVMF_CODE.fd"
>existed anywhere in the "Build" directory.
>
>Judged from the build log, it seemed that at least some *.efi modules
>were compiled and linked, but FVs and FDs were not built. The following
>sections of the log were missing:
>
>> Fd File Name:OVMF_VARS
>(.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_VARS.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x40000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x40000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x41000
>>    Region Size = 0x1000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x42000
>>    Region Size = 0x42000
>>    Region Name = None
>>
>> Fd File Name:MEMFD (.../Build/OvmfIa32/NOOPT_GCC48/FV/MEMFD.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x6000
>>    Region Name = None
>>
>> Generate Region at Offset 0x6000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x7000
>>    Region Size = 0x1000
>>    Region Name = None
>> Padding region starting from offset 0x8000, with size 0x8000
>>
>> Generate Region at Offset 0x8000
>>    Region Size = 0x8000
>>    Region Name = None
>>
>> Generate Region at Offset 0x10000
>>    Region Size = 0x10000
>>    Region Name = None
>>
>> Generate Region at Offset 0x20000
>>    Region Size = 0xE0000
>>    Region Name = FV
>>
>> Generating PEIFV FV
>>
>> Generate Region at Offset 0x100000
>>    Region Size = 0xB00000
>>    Region Name = FV
>>
>> Generating DXEFV FV
>>
>> Fd File Name:OVMF (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x40000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x40000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x41000
>>    Region Size = 0x1000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x42000
>>    Region Size = 0x42000
>>    Region Name = None
>>
>> Generate Region at Offset 0x84000
>>    Region Size = 0x348000
>>    Region Name = FV
>>
>> Generating FVMAIN_COMPACT FV
>>
>> Generate Region at Offset 0x3CC000
>>    Region Size = 0x34000
>>    Region Name = FV
>>
>> Generating SECFV FV
>>
>> Fd File Name:OVMF_CODE
>(.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x348000
>>    Region Name = FV
>>
>> Generate Region at Offset 0x348000
>>    Region Size = 0x34000
>>    Region Name = FV
>
>and
>
>> FV Space Information
>> SECFV [19%Full] 212992 total, 42208 used, 170784 free
>> FVMAIN_COMPACT [52%Full] 3440640 total, 1820408 used, 1620232 free
>> DXEFV [86%Full] 11534336 total, 9928672 used, 1605664 free
>> PEIFV [43%Full] 917504 total, 395944 used, 521560 free
>> Build report can be found at .../build.ovmf.32.report
>
>Interestingly, the line
>
>> GUID cross reference file can be found
>at .../Build/OvmfIa32/NOOPT_GCC48/FV/Guid.xref
>
>which is normally printed between the two listings above, was still
>printed, but alone.
>
>Thanks,
>Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Laszlo:
  This issue has been fixed in edk2 master. I just cherry pick those fixes from edk2 master to my Python3 branch (https://github.com/lgao4/edk2/tree/Python3). 

Thanks
Liming
>-----Original Message-----
>From: Laszlo Ersek [mailto:lersek@redhat.com]
>Sent: Monday, December 31, 2018 8:16 AM
>To: Gao, Liming <liming.gao@intel.com>; Gary Lin <glin@suse.com>
>Cc: edk2-devel@lists.01.org; Kinney, Michael D <michael.d.kinney@intel.com>
>Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>
>On 12/29/18 07:07, Gao, Liming wrote:
>> Lin:
>>    Thanks for your verification.  This issue has been fixed in the
>>    latest version of https://github.com/lgao4/edk2/tree/Python3. Could
>>    you try again?
>
>At commit 4614985ec223 ("BaseTools:The binary data type is incorrect",
>2018-12-29):
>
>
>(1) I tried to build OVMF as follows (using
>python-2.7.5-69.el7_5.x86_64):
>
>  build \
>    -a IA32 \
>    -p OvmfPkg/OvmfPkgIa32.dsc \
>    -D SMM_REQUIRE \
>    -D SECURE_BOOT_ENABLE \
>    -D TLS_ENABLE \
>    -D NETWORK_IP6_ENABLE \
>    -t GCC48 \
>    -n 4 \
>    --report-file=/tmp/build.ovmf.32.report \
>    --log=/tmp/build.ovmf.32.log \
>    -b NOOPT \
>    -D HTTP_BOOT_ENABLE \
>    --cmd-len=65536 \
>    --hash \
>    --genfds-multi-thread
>
>This produced messages like:
>
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>warning: Fail to read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>warning: Fail to read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>read report file
>> BuildReport.py...
>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>read report file
>
>near the end of the build.
>
>
>(2) After I removed the "--report-file" switch, the warnings
>disappeared.
>
>However, neither of the expected output files
>
>- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd
>- Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd
>
>existed. To be more precise, none of "OVMF.fd" and "OVMF_CODE.fd"
>existed anywhere in the "Build" directory.
>
>Judged from the build log, it seemed that at least some *.efi modules
>were compiled and linked, but FVs and FDs were not built. The following
>sections of the log were missing:
>
>> Fd File Name:OVMF_VARS
>(.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_VARS.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x40000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x40000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x41000
>>    Region Size = 0x1000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x42000
>>    Region Size = 0x42000
>>    Region Name = None
>>
>> Fd File Name:MEMFD (.../Build/OvmfIa32/NOOPT_GCC48/FV/MEMFD.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x6000
>>    Region Name = None
>>
>> Generate Region at Offset 0x6000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x7000
>>    Region Size = 0x1000
>>    Region Name = None
>> Padding region starting from offset 0x8000, with size 0x8000
>>
>> Generate Region at Offset 0x8000
>>    Region Size = 0x8000
>>    Region Name = None
>>
>> Generate Region at Offset 0x10000
>>    Region Size = 0x10000
>>    Region Name = None
>>
>> Generate Region at Offset 0x20000
>>    Region Size = 0xE0000
>>    Region Name = FV
>>
>> Generating PEIFV FV
>>
>> Generate Region at Offset 0x100000
>>    Region Size = 0xB00000
>>    Region Name = FV
>>
>> Generating DXEFV FV
>>
>> Fd File Name:OVMF (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x40000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x40000
>>    Region Size = 0x1000
>>    Region Name = None
>>
>> Generate Region at Offset 0x41000
>>    Region Size = 0x1000
>>    Region Name = DATA
>>
>> Generate Region at Offset 0x42000
>>    Region Size = 0x42000
>>    Region Name = None
>>
>> Generate Region at Offset 0x84000
>>    Region Size = 0x348000
>>    Region Name = FV
>>
>> Generating FVMAIN_COMPACT FV
>>
>> Generate Region at Offset 0x3CC000
>>    Region Size = 0x34000
>>    Region Name = FV
>>
>> Generating SECFV FV
>>
>> Fd File Name:OVMF_CODE
>(.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd)
>>
>> Generate Region at Offset 0x0
>>    Region Size = 0x348000
>>    Region Name = FV
>>
>> Generate Region at Offset 0x348000
>>    Region Size = 0x34000
>>    Region Name = FV
>
>and
>
>> FV Space Information
>> SECFV [19%Full] 212992 total, 42208 used, 170784 free
>> FVMAIN_COMPACT [52%Full] 3440640 total, 1820408 used, 1620232 free
>> DXEFV [86%Full] 11534336 total, 9928672 used, 1605664 free
>> PEIFV [43%Full] 917504 total, 395944 used, 521560 free
>> Build report can be found at .../build.ovmf.32.report
>
>Interestingly, the line
>
>> GUID cross reference file can be found
>at .../Build/OvmfIa32/NOOPT_GCC48/FV/Guid.xref
>
>which is normally printed between the two listings above, was still
>printed, but alone.
>
>Thanks,
>Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 01/04/19 04:29, Gao, Liming wrote:
> Laszlo:
>   This issue has been fixed in edk2 master. I just cherry pick those fixes from edk2 master to my Python3 branch (https://github.com/lgao4/edk2/tree/Python3). 

Thank you, I'll try to return to this topic soon, and retest the branch.

Cheers
Laszlo

> 
> Thanks
> Liming
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Monday, December 31, 2018 8:16 AM
>> To: Gao, Liming <liming.gao@intel.com>; Gary Lin <glin@suse.com>
>> Cc: edk2-devel@lists.01.org; Kinney, Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>>
>> On 12/29/18 07:07, Gao, Liming wrote:
>>> Lin:
>>>    Thanks for your verification.  This issue has been fixed in the
>>>    latest version of https://github.com/lgao4/edk2/tree/Python3. Could
>>>    you try again?
>>
>> At commit 4614985ec223 ("BaseTools:The binary data type is incorrect",
>> 2018-12-29):
>>
>>
>> (1) I tried to build OVMF as follows (using
>> python-2.7.5-69.el7_5.x86_64):
>>
>>  build \
>>    -a IA32 \
>>    -p OvmfPkg/OvmfPkgIa32.dsc \
>>    -D SMM_REQUIRE \
>>    -D SECURE_BOOT_ENABLE \
>>    -D TLS_ENABLE \
>>    -D NETWORK_IP6_ENABLE \
>>    -t GCC48 \
>>    -n 4 \
>>    --report-file=/tmp/build.ovmf.32.report \
>>    --log=/tmp/build.ovmf.32.log \
>>    -b NOOPT \
>>    -D HTTP_BOOT_ENABLE \
>>    --cmd-len=65536 \
>>    --hash \
>>    --genfds-multi-thread
>>
>> This produced messages like:
>>
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>> warning: Fail to read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/FVMAIN_COMPACT.Fv.txt(...):
>> warning: Fail to read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/SECFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/PEIFV.Fv.txt(...): warning: Fail to
>> read report file
>>> BuildReport.py...
>>> .../Build/OvmfIa32/NOOPT_GCC48/FV/DXEFV.Fv.txt(...): warning: Fail to
>> read report file
>>
>> near the end of the build.
>>
>>
>> (2) After I removed the "--report-file" switch, the warnings
>> disappeared.
>>
>> However, neither of the expected output files
>>
>> - Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd
>> - Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd
>>
>> existed. To be more precise, none of "OVMF.fd" and "OVMF_CODE.fd"
>> existed anywhere in the "Build" directory.
>>
>> Judged from the build log, it seemed that at least some *.efi modules
>> were compiled and linked, but FVs and FDs were not built. The following
>> sections of the log were missing:
>>
>>> Fd File Name:OVMF_VARS
>> (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_VARS.fd)
>>>
>>> Generate Region at Offset 0x0
>>>    Region Size = 0x40000
>>>    Region Name = DATA
>>>
>>> Generate Region at Offset 0x40000
>>>    Region Size = 0x1000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x41000
>>>    Region Size = 0x1000
>>>    Region Name = DATA
>>>
>>> Generate Region at Offset 0x42000
>>>    Region Size = 0x42000
>>>    Region Name = None
>>>
>>> Fd File Name:MEMFD (.../Build/OvmfIa32/NOOPT_GCC48/FV/MEMFD.fd)
>>>
>>> Generate Region at Offset 0x0
>>>    Region Size = 0x6000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x6000
>>>    Region Size = 0x1000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x7000
>>>    Region Size = 0x1000
>>>    Region Name = None
>>> Padding region starting from offset 0x8000, with size 0x8000
>>>
>>> Generate Region at Offset 0x8000
>>>    Region Size = 0x8000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x10000
>>>    Region Size = 0x10000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x20000
>>>    Region Size = 0xE0000
>>>    Region Name = FV
>>>
>>> Generating PEIFV FV
>>>
>>> Generate Region at Offset 0x100000
>>>    Region Size = 0xB00000
>>>    Region Name = FV
>>>
>>> Generating DXEFV FV
>>>
>>> Fd File Name:OVMF (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF.fd)
>>>
>>> Generate Region at Offset 0x0
>>>    Region Size = 0x40000
>>>    Region Name = DATA
>>>
>>> Generate Region at Offset 0x40000
>>>    Region Size = 0x1000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x41000
>>>    Region Size = 0x1000
>>>    Region Name = DATA
>>>
>>> Generate Region at Offset 0x42000
>>>    Region Size = 0x42000
>>>    Region Name = None
>>>
>>> Generate Region at Offset 0x84000
>>>    Region Size = 0x348000
>>>    Region Name = FV
>>>
>>> Generating FVMAIN_COMPACT FV
>>>
>>> Generate Region at Offset 0x3CC000
>>>    Region Size = 0x34000
>>>    Region Name = FV
>>>
>>> Generating SECFV FV
>>>
>>> Fd File Name:OVMF_CODE
>> (.../Build/OvmfIa32/NOOPT_GCC48/FV/OVMF_CODE.fd)
>>>
>>> Generate Region at Offset 0x0
>>>    Region Size = 0x348000
>>>    Region Name = FV
>>>
>>> Generate Region at Offset 0x348000
>>>    Region Size = 0x34000
>>>    Region Name = FV
>>
>> and
>>
>>> FV Space Information
>>> SECFV [19%Full] 212992 total, 42208 used, 170784 free
>>> FVMAIN_COMPACT [52%Full] 3440640 total, 1820408 used, 1620232 free
>>> DXEFV [86%Full] 11534336 total, 9928672 used, 1605664 free
>>> PEIFV [43%Full] 917504 total, 395944 used, 521560 free
>>> Build report can be found at .../build.ovmf.32.report
>>
>> Interestingly, the line
>>
>>> GUID cross reference file can be found
>> at .../Build/OvmfIa32/NOOPT_GCC48/FV/Guid.xref
>>
>> which is normally printed between the two listings above, was still
>> printed, but alone.
>>
>> Thanks,
>> Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

Hi Liming,

On 01/04/19 04:29, Gao, Liming wrote:
> Laszlo:
>   This issue has been fixed in edk2 master. I just cherry pick those
>   fixes from edk2 master to my Python3 branch
>   (https://github.com/lgao4/edk2/tree/Python3).

At commit 90d8b4834fd1 ("BaseTools: Reset FdsGlobalVariable",
2019-01-04):

(a) My regression tests using python-2.7.5-69.el7_5.x86_64 were all
successful (build and boot, plus S3 wherever applicable).

They covered IA32, IA32X64, and X64 OVMF, with/without SMM, and with
Fedora and some Windows guests. They also covered ArmVirtQemu on aarch64
KVM.


(b) For testing the Python3 enablement, I used a "RHEL8 Beta" virtual
machine. (The Python situation in RHEL8 is documented in the following
blog post:
<https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/>.)

My testing with Python3 failed. I did the following, in a clean checkout
of your repo/branch, and a clean shell environment:

# export PYTHON3_ENABLE=TRUE
# source edksetup.sh
# nice make -C "$EDK_TOOLS_PATH" -j3

The final output was:

> ======================================================================
> ERROR: testRandomDataCycles (TianoCompress.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/root/liming/BaseTools/Tests/TianoCompress.py", line 66, in testRandomDataCycles
>     self.compressionTestCycle(data)
>   File "/root/liming/BaseTools/Tests/TianoCompress.py", line 39, in compressionTestCycle
>     self.WriteTmpFile('input', data)
>   File "/root/liming/BaseTools/Tests/TestTools.py", line 154, in WriteTmpFile
>     f.write(data)
>   File "/usr/lib64/python3.6/encodings/iso8859_2.py", line 19, in encode
>     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
> UnicodeEncodeError: 'charmap' codec can't encode character '\xc0' in position 27: character maps to <undefined>
>
> ----------------------------------------------------------------------

(The backtrace mentions "iso8859_2.py" because I happen to use

  LC_CTYPE=hu_HU.ISO8859-2

in my locale settings.)

Either way, I think the issue is that an object ("data") is being
considered a string, rather than a byte array.


(c) To re-iterate my earlier request, would it be possible to set
PYTHON_COMMAND externally (in addition to PYTHON3_ENABLE=TRUE)? Because,
once we package a new version of edk2 for RHEL8, we'd like to set
PYTHON_COMMAND to "/usr/libexec/platform-python", and not to the
auto-detected "/usr/bin/python3'. (In fact, "/usr/bin/python3" could be
missing from the restricted package build environment.)

See the blog post I referenced above -- one of its takeaways is, "use
platform-python if you are writing system/admin code for RHEL 8". In
other words, when users build upstream edk2 on a RHEL8 machine, it's
fine if edk2 detects and uses /usr/bin/python3. However, when the edk2
package of the distro itself is built, it should be possible to direct
edk2 towards "/usr/libexec/platform-python".

Thank you,
Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gary Lin]

On Sat, Dec 29, 2018 at 06:07:10AM +0000, Gao, Liming wrote:
> Lin:
>    Thanks for your verification.  This issue has been fixed in the latest version of https://github.com/lgao4/edk2/tree/Python3. Could you try again?
> 
Hi Liming,

I can confirm that the crash I had was fixed in the latest git branch.

Thanks!

Gary Lin

> Thanks
> Liming
> >-----Original Message-----
> >From: Gary Lin [mailto:glin@suse.com]
> >Sent: Friday, December 28, 2018 6:40 PM
> >To: Gao, Liming <liming.gao@intel.com>
> >Cc: edk2-devel@lists.01.org; Kinney, Michael D
> ><michael.d.kinney@intel.com>; Laszlo Ersek (lersek@redhat.com)
> ><lersek@redhat.com>
> >Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >
> >On Tue, Dec 25, 2018 at 07:50:43AM +0000, Gao, Liming wrote:
> >> Hi, all
> >>   On Python3 migration https://bugzilla.tianocore.org/show_bug.cgi?id=55,
> >we update Edk2 BaseTools python source code with the compatible syntax to
> >support Python2 and Python3 both. Here is code
> >https://github.com/lgao4/edk2/tree/Python3 for dry run. To enable Python3,
> >you just need to set PYTHON3_ENABLE environment as TRUE, then type
> >edksetup.bat/edksetup.sh. Without this setting, BaseTools still run with
> >Python2. So, there is no change for current usage model with Python27.
> >>
> >I've built OVMF with both python2 and python3 based on the branch and
> >the VM crashed immediately. The only message in the debug log is:
> >
> >SecCoreStartupWithStack(0xFFFCC000, 0x820000)
> >
> >The result of bisect showed the first bad commit is
> >
> > 500aad1a02c5a8c0f208c9422cce19de7d304faa
> > BaseTools: Update windows and linux run scripts file to use Python3
> >
> >Cheers,
> >
> >Gary Lin
> >
> >>   But, we have no enough resource to fully verify Python2 and Python3 both.
> >We will focus on Python3 validation. If anyone can help verify Python2, it will
> >be great. And, if you meet with the issue on Python2, please file BZ. We still
> >fix them.
> >>
> >> Thanks
> >> Liming
> >>
> >> _______________________________________________
> >> edk2-devel mailing list
> >> edk2-devel@lists.01.org
> >> https://lists.01.org/mailman/listinfo/edk2-devel
> >>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
> 

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Ni, Ruiyu]

On 12/25/2018 3:50 PM, Gao, Liming wrote:
> Hi, all
>    On Python3 migration https://bugzilla.tianocore.org/show_bug.cgi?id=55, we update Edk2 BaseTools python source code with the compatible syntax to support Python2 and Python3 both. Here is code https://github.com/lgao4/edk2/tree/Python3 for dry run. To enable Python3, you just need to set PYTHON3_ENABLE environment as TRUE, then type edksetup.bat/edksetup.sh. Without this setting, BaseTools still run with Python2. So, there is no change for current usage model with Python27.

Liming,
I like Python3. But I don't like the idea of enabling Python3 depending 
on PYTHON3_ENABLE environment variable.
I prefer BaseTools to use Python3 by default when PYTHON3_ENABLE is not set.
When PYTHON3_ENABLE is set, BaseTools can use the desired python version 
following the environment variable.

Do you agree? Or any objection?


> 
>    But, we have no enough resource to fully verify Python2 and Python3 both. We will focus on Python3 validation. If anyone can help verify Python2, it will be great. And, if you meet with the issue on Python2, please file BZ. We still fix them.
> 
> Thanks
> Liming
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
> 


-- 
Thanks,
Ray

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Ray:
 I think this proposal is good to recommend Python3 as the default interpreter. I summary the updated proposal. 

1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find higher version python installed in OS. If Python3 is found, Python3 will be used. Then, if python2 is found, and python2 is used. If not found, report error and stop build. This will change the default python interpreter from Python2 to Python3 when they both are installed. 
2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will find Python3. If Python3 is found, Python3 will be used. If not found, report error and stop build.
3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh will find Python2. If Python2 is found, Python2 will be used. If not found, report error and stop build.
Once Python is found, edksetup.bat/edksetup.sh and build tool will both print message to let user aware which version python tool is used in this build. 

Thanks
Liming
> -----Original Message-----
> From: Ni, Ray
> Sent: Monday, January 7, 2019 4:40 PM
> To: Gao, Liming <liming.gao@intel.com>; edk2-devel@lists.01.org
> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Laszlo Ersek (lersek@redhat.com) <lersek@redhat.com>
> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> 
> On 12/25/2018 3:50 PM, Gao, Liming wrote:
> > Hi, all
> >    On Python3 migration https://bugzilla.tianocore.org/show_bug.cgi?id=55, we update Edk2 BaseTools python source code with the
> compatible syntax to support Python2 and Python3 both. Here is code https://github.com/lgao4/edk2/tree/Python3 for dry run. To
> enable Python3, you just need to set PYTHON3_ENABLE environment as TRUE, then type edksetup.bat/edksetup.sh. Without this setting,
> BaseTools still run with Python2. So, there is no change for current usage model with Python27.
> 
> Liming,
> I like Python3. But I don't like the idea of enabling Python3 depending
> on PYTHON3_ENABLE environment variable.
> I prefer BaseTools to use Python3 by default when PYTHON3_ENABLE is not set.
> When PYTHON3_ENABLE is set, BaseTools can use the desired python version
> following the environment variable.
> 
> Do you agree? Or any objection?
> 
> 
> >
> >    But, we have no enough resource to fully verify Python2 and Python3 both. We will focus on Python3 validation. If anyone can help
> verify Python2, it will be great. And, if you meet with the issue on Python2, please file BZ. We still fix them.
> >
> > Thanks
> > Liming
> >
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> >
> 
> 
> --
> Thanks,
> Ray

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 01/07/19 14:41, Gao, Liming wrote:
> Ray:
>  I think this proposal is good to recommend Python3 as the default interpreter. I summary the updated proposal. 
> 
> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find higher version python installed in OS. If Python3 is found, Python3 will be used. Then, if python2 is found, and python2 is used. If not found, report error and stop build. This will change the default python interpreter from Python2 to Python3 when they both are installed. 
> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will find Python3. If Python3 is found, Python3 will be used. If not found, report error and stop build.
> 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh will find Python2. If Python2 is found, Python2 will be used. If not found, report error and stop build.
> Once Python is found, edksetup.bat/edksetup.sh and build tool will both print message to let user aware which version python tool is used in this build. 

If we're going for this level of flexibility, I'd like to suggest /
request another improvement. Some Linux distros intend to accommodate
multiple Python3 versions at the same time (this is not a typo; I don't
mean Python2+Python3, but multiple Python3 versions). So basically I'd
suggest that we offer a method for specifying a python version
(2/3/auto-detect), plus, in case a specific major version is specified,
that we allow the user to specify the precise interpreter pathname too.

Thanks,
Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Laszlo:
  Yes. This can be supported. But, I don't know what purpose to specify python minor version of Python3. Current implementation in Python3 branch always tries to find the high version installed in OS. For example, Python3.4, Python3.7 are both installed, Python3.7 will be chosen. Does this policy meet with your usage?

Thanks
Liming
> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Tuesday, January 8, 2019 3:04 AM
> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org; leif.lindholm@linaro.org;
> afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> 
> On 01/07/19 14:41, Gao, Liming wrote:
> > Ray:
> >  I think this proposal is good to recommend Python3 as the default interpreter. I summary the updated proposal.
> >
> > 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find higher version python installed in OS. If Python3 is found,
> Python3 will be used. Then, if python2 is found, and python2 is used. If not found, report error and stop build. This will change the
> default python interpreter from Python2 to Python3 when they both are installed.
> > 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will find Python3. If Python3 is found, Python3 will be used. If not
> found, report error and stop build.
> > 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh will find Python2. If Python2 is found, Python2 will be used. If
> not found, report error and stop build.
> > Once Python is found, edksetup.bat/edksetup.sh and build tool will both print message to let user aware which version python tool is
> used in this build.
> 
> If we're going for this level of flexibility, I'd like to suggest /
> request another improvement. Some Linux distros intend to accommodate
> multiple Python3 versions at the same time (this is not a typo; I don't
> mean Python2+Python3, but multiple Python3 versions). So basically I'd
> suggest that we offer a method for specifying a python version
> (2/3/auto-detect), plus, in case a specific major version is specified,
> that we allow the user to specify the precise interpreter pathname too.
> 
> Thanks,
> Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Carsey, Jaben]

Liming and Laszlo,
	What if we add a 4th option to the environment variable - the path to a specific python interpreter for use.

-Jaben


> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Gao, Liming
> Sent: Tuesday, January 08, 2019 6:23 AM
> To: Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>; edk2-
> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> 
> Laszlo:
>   Yes. This can be supported. But, I don't know what purpose to specify
> python minor version of Python3. Current implementation in Python3 branch
> always tries to find the high version installed in OS. For example, Python3.4,
> Python3.7 are both installed, Python3.7 will be chosen. Does this policy meet
> with your usage?
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: Laszlo Ersek [mailto:lersek@redhat.com]
> > Sent: Tuesday, January 8, 2019 3:04 AM
> > To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
> devel@lists.01.org; leif.lindholm@linaro.org;
> > afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
> > Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >
> > On 01/07/19 14:41, Gao, Liming wrote:
> > > Ray:
> > >  I think this proposal is good to recommend Python3 as the default
> interpreter. I summary the updated proposal.
> > >
> > > 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find
> higher version python installed in OS. If Python3 is found,
> > Python3 will be used. Then, if python2 is found, and python2 is used. If not
> found, report error and stop build. This will change the
> > default python interpreter from Python2 to Python3 when they both are
> installed.
> > > 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will
> find Python3. If Python3 is found, Python3 will be used. If not
> > found, report error and stop build.
> > > 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh
> will find Python2. If Python2 is found, Python2 will be used. If
> > not found, report error and stop build.
> > > Once Python is found, edksetup.bat/edksetup.sh and build tool will both
> print message to let user aware which version python tool is
> > used in this build.
> >
> > If we're going for this level of flexibility, I'd like to suggest /
> > request another improvement. Some Linux distros intend to accommodate
> > multiple Python3 versions at the same time (this is not a typo; I don't
> > mean Python2+Python3, but multiple Python3 versions). So basically I'd
> > suggest that we offer a method for specifying a python version
> > (2/3/auto-detect), plus, in case a specific major version is specified,
> > that we allow the user to specify the precise interpreter pathname too.
> >
> > Thanks,
> > Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 01/08/19 17:22, Carsey, Jaben wrote:
> Liming and Laszlo,
> 	What if we add a 4th option to the environment variable - the path to a specific python interpreter for use.

I thought of that, but how do the build tools derive the python version
just from the pathname of the interpreter?

Will they run "$INTERPRETER --version" and parse the output?

I think that could be brittle; distributions sometimes customize the
version strings of their executables. The "--version" output is usually
human-readable, not machine-readable (per intent).

Thanks,
Laszlo

> 
> -Jaben
> 
> 
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>> Gao, Liming
>> Sent: Tuesday, January 08, 2019 6:23 AM
>> To: Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>; edk2-
>> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
>> Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>>
>> Laszlo:
>>   Yes. This can be supported. But, I don't know what purpose to specify
>> python minor version of Python3. Current implementation in Python3 branch
>> always tries to find the high version installed in OS. For example, Python3.4,
>> Python3.7 are both installed, Python3.7 will be chosen. Does this policy meet
>> with your usage?
>>
>> Thanks
>> Liming
>>> -----Original Message-----
>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>> Sent: Tuesday, January 8, 2019 3:04 AM
>>> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
>> devel@lists.01.org; leif.lindholm@linaro.org;
>>> afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
>>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>>>
>>> On 01/07/19 14:41, Gao, Liming wrote:
>>>> Ray:
>>>>  I think this proposal is good to recommend Python3 as the default
>> interpreter. I summary the updated proposal.
>>>>
>>>> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find
>> higher version python installed in OS. If Python3 is found,
>>> Python3 will be used. Then, if python2 is found, and python2 is used. If not
>> found, report error and stop build. This will change the
>>> default python interpreter from Python2 to Python3 when they both are
>> installed.
>>>> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will
>> find Python3. If Python3 is found, Python3 will be used. If not
>>> found, report error and stop build.
>>>> 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh
>> will find Python2. If Python2 is found, Python2 will be used. If
>>> not found, report error and stop build.
>>>> Once Python is found, edksetup.bat/edksetup.sh and build tool will both
>> print message to let user aware which version python tool is
>>> used in this build.
>>>
>>> If we're going for this level of flexibility, I'd like to suggest /
>>> request another improvement. Some Linux distros intend to accommodate
>>> multiple Python3 versions at the same time (this is not a typo; I don't
>>> mean Python2+Python3, but multiple Python3 versions). So basically I'd
>>> suggest that we offer a method for specifying a python version
>>> (2/3/auto-detect), plus, in case a specific major version is specified,
>>> that we allow the user to specify the precise interpreter pathname too.
>>>
>>> Thanks,
>>> Laszlo
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Carsey, Jaben]

> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Tuesday, January 08, 2019 9:26 AM
> To: Carsey, Jaben <jaben.carsey@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> Importance: High
> 
> On 01/08/19 17:22, Carsey, Jaben wrote:
> > Liming and Laszlo,
> > 	What if we add a 4th option to the environment variable - the path to
> a specific python interpreter for use.
> 
> I thought of that, but how do the build tools derive the python version
> just from the pathname of the interpreter?
> 
> Will they run "$INTERPRETER --version" and parse the output?
> 
> I think that could be brittle; distributions sometimes customize the
> version strings of their executables. The "--version" output is usually
> human-readable, not machine-readable (per intent).

Laszlo, you lost me. How is that related to an exact path?   If the user specifies the path, then always use that specific interpreter.

> 
> Thanks,
> Laszlo
> 
> >
> > -Jaben
> >
> >
> >> -----Original Message-----
> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> >> Gao, Liming
> >> Sent: Tuesday, January 08, 2019 6:23 AM
> >> To: Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>; edk2-
> >> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
> >> Michael D <michael.d.kinney@intel.com>
> >> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >>
> >> Laszlo:
> >>   Yes. This can be supported. But, I don't know what purpose to specify
> >> python minor version of Python3. Current implementation in Python3
> branch
> >> always tries to find the high version installed in OS. For example,
> Python3.4,
> >> Python3.7 are both installed, Python3.7 will be chosen. Does this policy
> meet
> >> with your usage?
> >>
> >> Thanks
> >> Liming
> >>> -----Original Message-----
> >>> From: Laszlo Ersek [mailto:lersek@redhat.com]
> >>> Sent: Tuesday, January 8, 2019 3:04 AM
> >>> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>;
> edk2-
> >> devel@lists.01.org; leif.lindholm@linaro.org;
> >>> afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
> >>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >>>
> >>> On 01/07/19 14:41, Gao, Liming wrote:
> >>>> Ray:
> >>>>  I think this proposal is good to recommend Python3 as the default
> >> interpreter. I summary the updated proposal.
> >>>>
> >>>> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find
> >> higher version python installed in OS. If Python3 is found,
> >>> Python3 will be used. Then, if python2 is found, and python2 is used. If
> not
> >> found, report error and stop build. This will change the
> >>> default python interpreter from Python2 to Python3 when they both are
> >> installed.
> >>>> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will
> >> find Python3. If Python3 is found, Python3 will be used. If not
> >>> found, report error and stop build.
> >>>> 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh
> >> will find Python2. If Python2 is found, Python2 will be used. If
> >>> not found, report error and stop build.
> >>>> Once Python is found, edksetup.bat/edksetup.sh and build tool will
> both
> >> print message to let user aware which version python tool is
> >>> used in this build.
> >>>
> >>> If we're going for this level of flexibility, I'd like to suggest /
> >>> request another improvement. Some Linux distros intend to
> accommodate
> >>> multiple Python3 versions at the same time (this is not a typo; I don't
> >>> mean Python2+Python3, but multiple Python3 versions). So basically I'd
> >>> suggest that we offer a method for specifying a python version
> >>> (2/3/auto-detect), plus, in case a specific major version is specified,
> >>> that we allow the user to specify the precise interpreter pathname too.
> >>>
> >>> Thanks,
> >>> Laszlo
> >> _______________________________________________
> >> edk2-devel mailing list
> >> edk2-devel@lists.01.org
> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Gao, Liming]

Jaben:
  I also think this way. Now, we have two envs PYTHON3_ENABLE and PYTHON_COMMAND. The behavior can be combined as the below to support this usage. If user wants the specific python interpreter, he only needs to set PYTHON_COMMAND env.

If PYTHON3_ENABLE is set, PYTHON_COMMAND will be set to the found one by edk2 scripts based on PYTHON3_ENABLE value. 
If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is set, then PYTHON_COMMAND will be used to run python script. No version check here. 
If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is not set, PYTHON_COMMAND will be set to the high version python installed in OS. 

Thanks
Liming
>-----Original Message-----
>From: Carsey, Jaben
>Sent: Wednesday, January 09, 2019 2:06 AM
>To: Laszlo Ersek <lersek@redhat.com>; Gao, Liming <liming.gao@intel.com>;
>Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org; leif.lindholm@linaro.org;
>afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
>Subject: RE: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>
>
>
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Tuesday, January 08, 2019 9:26 AM
>> To: Carsey, Jaben <jaben.carsey@intel.com>; Gao, Liming
>> <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
>> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
>> Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>> Importance: High
>>
>> On 01/08/19 17:22, Carsey, Jaben wrote:
>> > Liming and Laszlo,
>> > 	What if we add a 4th option to the environment variable - the path to
>> a specific python interpreter for use.
>>
>> I thought of that, but how do the build tools derive the python version
>> just from the pathname of the interpreter?
>>
>> Will they run "$INTERPRETER --version" and parse the output?
>>
>> I think that could be brittle; distributions sometimes customize the
>> version strings of their executables. The "--version" output is usually
>> human-readable, not machine-readable (per intent).
>
>Laszlo, you lost me. How is that related to an exact path?   If the user specifies
>the path, then always use that specific interpreter.
>
>>
>> Thanks,
>> Laszlo
>>
>> >
>> > -Jaben
>> >
>> >
>> >> -----Original Message-----
>> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
>Of
>> >> Gao, Liming
>> >> Sent: Tuesday, January 08, 2019 6:23 AM
>> >> To: Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>;
>edk2-
>> >> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
>> >> Michael D <michael.d.kinney@intel.com>
>> >> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>> >>
>> >> Laszlo:
>> >>   Yes. This can be supported. But, I don't know what purpose to specify
>> >> python minor version of Python3. Current implementation in Python3
>> branch
>> >> always tries to find the high version installed in OS. For example,
>> Python3.4,
>> >> Python3.7 are both installed, Python3.7 will be chosen. Does this policy
>> meet
>> >> with your usage?
>> >>
>> >> Thanks
>> >> Liming
>> >>> -----Original Message-----
>> >>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> >>> Sent: Tuesday, January 8, 2019 3:04 AM
>> >>> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>;
>> edk2-
>> >> devel@lists.01.org; leif.lindholm@linaro.org;
>> >>> afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
>> >>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>> >>>
>> >>> On 01/07/19 14:41, Gao, Liming wrote:
>> >>>> Ray:
>> >>>>  I think this proposal is good to recommend Python3 as the default
>> >> interpreter. I summary the updated proposal.
>> >>>>
>> >>>> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will
>find
>> >> higher version python installed in OS. If Python3 is found,
>> >>> Python3 will be used. Then, if python2 is found, and python2 is used. If
>> not
>> >> found, report error and stop build. This will change the
>> >>> default python interpreter from Python2 to Python3 when they both
>are
>> >> installed.
>> >>>> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh
>will
>> >> find Python3. If Python3 is found, Python3 will be used. If not
>> >>> found, report error and stop build.
>> >>>> 3. PYTHON3_ENABLE env is set to not TRUE.
>edksetup.bat/edksetup.sh
>> >> will find Python2. If Python2 is found, Python2 will be used. If
>> >>> not found, report error and stop build.
>> >>>> Once Python is found, edksetup.bat/edksetup.sh and build tool will
>> both
>> >> print message to let user aware which version python tool is
>> >>> used in this build.
>> >>>
>> >>> If we're going for this level of flexibility, I'd like to suggest /
>> >>> request another improvement. Some Linux distros intend to
>> accommodate
>> >>> multiple Python3 versions at the same time (this is not a typo; I don't
>> >>> mean Python2+Python3, but multiple Python3 versions). So basically I'd
>> >>> suggest that we offer a method for specifying a python version
>> >>> (2/3/auto-detect), plus, in case a specific major version is specified,
>> >>> that we allow the user to specify the precise interpreter pathname too.
>> >>>
>> >>> Thanks,
>> >>> Laszlo
>> >> _______________________________________________
>> >> edk2-devel mailing list
>> >> edk2-devel@lists.01.org
>> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Carsey, Jaben]

Liming,

I think that the user should not need to set PYTHON3_ENABLE at all if they manually set PYTHON_COMMAND.

-Jaben


> -----Original Message-----
> From: Gao, Liming
> Sent: Tuesday, January 08, 2019 4:43 PM
> To: Carsey, Jaben <jaben.carsey@intel.com>; Laszlo Ersek
> <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org;
> leif.lindholm@linaro.org; afish@apple.com; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Subject: RE: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> Importance: High
> 
> Jaben:
>   I also think this way. Now, we have two envs PYTHON3_ENABLE and
> PYTHON_COMMAND. The behavior can be combined as the below to
> support this usage. If user wants the specific python interpreter, he only
> needs to set PYTHON_COMMAND env.
> 
> If PYTHON3_ENABLE is set, PYTHON_COMMAND will be set to the found one
> by edk2 scripts based on PYTHON3_ENABLE value.
> If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is set, then
> PYTHON_COMMAND will be used to run python script. No version check
> here.
> If PYTHON3_ENABLE is not set, but PYTHON_COMMAND is not set,
> PYTHON_COMMAND will be set to the high version python installed in OS.
> 
> Thanks
> Liming
> >-----Original Message-----
> >From: Carsey, Jaben
> >Sent: Wednesday, January 09, 2019 2:06 AM
> >To: Laszlo Ersek <lersek@redhat.com>; Gao, Liming
> <liming.gao@intel.com>;
> >Ni, Ray <ray.ni@intel.com>; edk2-devel@lists.01.org;
> leif.lindholm@linaro.org;
> >afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
> >Subject: RE: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >
> >
> >
> >> -----Original Message-----
> >> From: Laszlo Ersek [mailto:lersek@redhat.com]
> >> Sent: Tuesday, January 08, 2019 9:26 AM
> >> To: Carsey, Jaben <jaben.carsey@intel.com>; Gao, Liming
> >> <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
> >> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
> >> Michael D <michael.d.kinney@intel.com>
> >> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >> Importance: High
> >>
> >> On 01/08/19 17:22, Carsey, Jaben wrote:
> >> > Liming and Laszlo,
> >> > 	What if we add a 4th option to the environment variable - the path to
> >> a specific python interpreter for use.
> >>
> >> I thought of that, but how do the build tools derive the python version
> >> just from the pathname of the interpreter?
> >>
> >> Will they run "$INTERPRETER --version" and parse the output?
> >>
> >> I think that could be brittle; distributions sometimes customize the
> >> version strings of their executables. The "--version" output is usually
> >> human-readable, not machine-readable (per intent).
> >
> >Laszlo, you lost me. How is that related to an exact path?   If the user
> specifies
> >the path, then always use that specific interpreter.
> >
> >>
> >> Thanks,
> >> Laszlo
> >>
> >> >
> >> > -Jaben
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
> >Of
> >> >> Gao, Liming
> >> >> Sent: Tuesday, January 08, 2019 6:23 AM
> >> >> To: Laszlo Ersek <lersek@redhat.com>; Ni, Ray <ray.ni@intel.com>;
> >edk2-
> >> >> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
> >> >> Michael D <michael.d.kinney@intel.com>
> >> >> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >> >>
> >> >> Laszlo:
> >> >>   Yes. This can be supported. But, I don't know what purpose to specify
> >> >> python minor version of Python3. Current implementation in Python3
> >> branch
> >> >> always tries to find the high version installed in OS. For example,
> >> Python3.4,
> >> >> Python3.7 are both installed, Python3.7 will be chosen. Does this policy
> >> meet
> >> >> with your usage?
> >> >>
> >> >> Thanks
> >> >> Liming
> >> >>> -----Original Message-----
> >> >>> From: Laszlo Ersek [mailto:lersek@redhat.com]
> >> >>> Sent: Tuesday, January 8, 2019 3:04 AM
> >> >>> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>;
> >> edk2-
> >> >> devel@lists.01.org; leif.lindholm@linaro.org;
> >> >>> afish@apple.com; Kinney, Michael D <michael.d.kinney@intel.com>
> >> >>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
> >> >>>
> >> >>> On 01/07/19 14:41, Gao, Liming wrote:
> >> >>>> Ray:
> >> >>>>  I think this proposal is good to recommend Python3 as the default
> >> >> interpreter. I summary the updated proposal.
> >> >>>>
> >> >>>> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will
> >find
> >> >> higher version python installed in OS. If Python3 is found,
> >> >>> Python3 will be used. Then, if python2 is found, and python2 is used.
> If
> >> not
> >> >> found, report error and stop build. This will change the
> >> >>> default python interpreter from Python2 to Python3 when they both
> >are
> >> >> installed.
> >> >>>> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh
> >will
> >> >> find Python3. If Python3 is found, Python3 will be used. If not
> >> >>> found, report error and stop build.
> >> >>>> 3. PYTHON3_ENABLE env is set to not TRUE.
> >edksetup.bat/edksetup.sh
> >> >> will find Python2. If Python2 is found, Python2 will be used. If
> >> >>> not found, report error and stop build.
> >> >>>> Once Python is found, edksetup.bat/edksetup.sh and build tool will
> >> both
> >> >> print message to let user aware which version python tool is
> >> >>> used in this build.
> >> >>>
> >> >>> If we're going for this level of flexibility, I'd like to suggest /
> >> >>> request another improvement. Some Linux distros intend to
> >> accommodate
> >> >>> multiple Python3 versions at the same time (this is not a typo; I don't
> >> >>> mean Python2+Python3, but multiple Python3 versions). So basically
> I'd
> >> >>> suggest that we offer a method for specifying a python version
> >> >>> (2/3/auto-detect), plus, in case a specific major version is specified,
> >> >>> that we allow the user to specify the precise interpreter pathname
> too.
> >> >>>
> >> >>> Thanks,
> >> >>> Laszlo
> >> >> _______________________________________________
> >> >> edk2-devel mailing list
> >> >> edk2-devel@lists.01.org
> >> >> https://lists.01.org/mailman/listinfo/edk2-devel

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 01/08/19 19:05, Carsey, Jaben wrote:
> 
> 
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Tuesday, January 08, 2019 9:26 AM
>> To: Carsey, Jaben <jaben.carsey@intel.com>; Gao, Liming
>> <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>; edk2-
>> devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com; Kinney,
>> Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>> Importance: High
>>
>> On 01/08/19 17:22, Carsey, Jaben wrote:
>>> Liming and Laszlo,
>>> 	What if we add a 4th option to the environment variable - the path to
>> a specific python interpreter for use.
>>
>> I thought of that, but how do the build tools derive the python version
>> just from the pathname of the interpreter?
>>
>> Will they run "$INTERPRETER --version" and parse the output?
>>
>> I think that could be brittle; distributions sometimes customize the
>> version strings of their executables. The "--version" output is usually
>> human-readable, not machine-readable (per intent).
> 
> Laszlo, you lost me. How is that related to an exact path?   If the user specifies the path, then always use that specific interpreter.

Sorry, my fault. I've been confused by an *earlier* approach that was
described here:

  https://bugzilla.tianocore.org/show_bug.cgi?id=55#c10
  https://bugzilla.tianocore.org/show_bug.cgi?id=55#c12

In that case, if the user only specified the interpreter path, it would
be clear what *interpreter* to use, however it would not be not clear
what *sub-codebase* from BaseTools to run on that interpreter: the
Python2 one, or the Python3 one.

However, now I remember, from

  https://bugzilla.tianocore.org/show_bug.cgi?id=55#c14
  https://bugzilla.tianocore.org/show_bug.cgi?id=55#c15

that after all, the goal is a single common set of BaseTools source code
that runs on Python2 and Python3 alike. In that regard, I agree that the
interpreter pathname is the *only* input we should take & accept from
the user, and the particular python *language version* need neither be
provided by the user, nor be detected by BaseTools themselves.

Sorry about the confusion!

Thanks,
Laszlo

Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek]

On 01/08/19 15:22, Gao, Liming wrote:
> Laszlo:
>   Yes. This can be supported. But, I don't know what purpose to
>   specify python minor version of Python3. Current implementation in
>   Python3 branch always tries to find the high version installed in
>   OS. For example, Python3.4, Python3.7 are both installed, Python3.7
>   will be chosen. Does this policy meet with your usage?

Not really; please see paragraph (c) in my other reply at
<https://lists.01.org/pipermail/edk2-devel/2019-January/034762.html>.

The rationale is given in the blog post at
<https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/>, near
the "platform-python" mentions. Basically the OS should be able to offer
different Python3 installations (at the same time) to end-users and
internally to OS-level packages.

Thanks!
Laszlo


>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Tuesday, January 8, 2019 3:04 AM
>> To: Gao, Liming <liming.gao@intel.com>; Ni, Ray <ray.ni@intel.com>;
>> edk2-devel@lists.01.org; leif.lindholm@linaro.org; afish@apple.com;
>> Kinney, Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2] [RFC] Edk2 BaseTools Python3 Migration Update
>>
>> On 01/07/19 14:41, Gao, Liming wrote:
>>> Ray:
>>>  I think this proposal is good to recommend Python3 as the default
>>>  interpreter. I summary the updated proposal.
>>>
>>> 1. PYTHON3_ENABLE env is not set. edksetup.bat/edksetup.sh will find
>>> higher version python installed in OS. If Python3 is found, Python3
>>> will be used. Then, if python2 is found, and python2 is used. If not
>>> found, report error and stop build. This will change the default
>>> python interpreter from Python2 to Python3 when they both are
>>> installed.
>>> 2. PYTHON3_EANBLE env is set to TRUE. edksetup.bat/edksetup.sh will
>>> find Python3. If Python3 is found, Python3 will be used. If not
>>> found, report error and stop build.
>>> 3. PYTHON3_ENABLE env is set to not TRUE. edksetup.bat/edksetup.sh
>>> will find Python2. If Python2 is found, Python2 will be used. If not
>>> found, report error and stop build. Once Python is found,
>>> edksetup.bat/edksetup.sh and build tool will both print message to
>>> let user aware which version python tool is used in this build.
>>
>> If we're going for this level of flexibility, I'd like to suggest /
>> request another improvement. Some Linux distros intend to accommodate
>> multiple Python3 versions at the same time (this is not a typo; I
>> don't mean Python2+Python3, but multiple Python3 versions). So
>> basically I'd suggest that we offer a method for specifying a python
>> version (2/3/auto-detect), plus, in case a specific major version is
>> specified, that we allow the user to specify the precise interpreter
>> pathname too.
>>
>> Thanks,
>> Laszlo