Re: [RFC] Edk2 BaseTools Python3 Migration Update

[Laszlo Ersek <lersek@redhat.com>]

Hi Liming,

On 01/04/19 04:29, Gao, Liming wrote:
> Laszlo:
>   This issue has been fixed in edk2 master. I just cherry pick those
>   fixes from edk2 master to my Python3 branch
>   (https://github.com/lgao4/edk2/tree/Python3).

At commit 90d8b4834fd1 ("BaseTools: Reset FdsGlobalVariable",
2019-01-04):

(a) My regression tests using python-2.7.5-69.el7_5.x86_64 were all
successful (build and boot, plus S3 wherever applicable).

They covered IA32, IA32X64, and X64 OVMF, with/without SMM, and with
Fedora and some Windows guests. They also covered ArmVirtQemu on aarch64
KVM.


(b) For testing the Python3 enablement, I used a "RHEL8 Beta" virtual
machine. (The Python situation in RHEL8 is documented in the following
blog post:
<https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/>.)

My testing with Python3 failed. I did the following, in a clean checkout
of your repo/branch, and a clean shell environment:

# export PYTHON3_ENABLE=TRUE
# source edksetup.sh
# nice make -C "$EDK_TOOLS_PATH" -j3

The final output was:

> ======================================================================
> ERROR: testRandomDataCycles (TianoCompress.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/root/liming/BaseTools/Tests/TianoCompress.py", line 66, in testRandomDataCycles
>     self.compressionTestCycle(data)
>   File "/root/liming/BaseTools/Tests/TianoCompress.py", line 39, in compressionTestCycle
>     self.WriteTmpFile('input', data)
>   File "/root/liming/BaseTools/Tests/TestTools.py", line 154, in WriteTmpFile
>     f.write(data)
>   File "/usr/lib64/python3.6/encodings/iso8859_2.py", line 19, in encode
>     return codecs.charmap_encode(input,self.errors,encoding_table)[0]
> UnicodeEncodeError: 'charmap' codec can't encode character '\xc0' in position 27: character maps to <undefined>
>
> ----------------------------------------------------------------------

(The backtrace mentions "iso8859_2.py" because I happen to use

  LC_CTYPE=hu_HU.ISO8859-2

in my locale settings.)

Either way, I think the issue is that an object ("data") is being
considered a string, rather than a byte array.


(c) To re-iterate my earlier request, would it be possible to set
PYTHON_COMMAND externally (in addition to PYTHON3_ENABLE=TRUE)? Because,
once we package a new version of edk2 for RHEL8, we'd like to set
PYTHON_COMMAND to "/usr/libexec/platform-python", and not to the
auto-detected "/usr/bin/python3'. (In fact, "/usr/bin/python3" could be
missing from the restricted package build environment.)

See the blog post I referenced above -- one of its takeaways is, "use
platform-python if you are writing system/admin code for RHEL 8". In
other words, when users build upstream edk2 on a RHEL8 machine, it's
fine if edk2 detects and uses /usr/bin/python3. However, when the edk2
package of the distro itself is built, it should be possible to direct
edk2 towards "/usr/libexec/platform-python".

Thank you,
Laszlo