Official port of Python on EDK2

Official port of Python on EDK2

[Thiebaud Weksteen]

Hi,

UEFI has become the standard for firmware (BIOS) interface. Intel has
provided an open source implementation under the name EDK2 (part of
the TianoCore initiative) [1] for some time. This implementation has
evolved significantly and now provides the functionalities of a small
OS with a standard library similar to POSIX.

In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
This port then evolved to 2.7.2 which is still defined as the
reference port [3]. In 2015, another port was added of Python 2.7.10
in parallel of 2.7.2 [4]. Since then, both implementations have
diverged from upstream and know vulnerabilities have not been fixed.

I would like to bring support for edk2 in the official Python
repository to remediate this situation, that is officially support
edk2 as a platform. Technically, there would be three main aspects for
the on-boarding work:

1) Fix headers and source to resolve definition conflicts, similarly
to ABS definition in [5];
2) Add the edk2module.c [6] to handle platform-specific
functionalities, similarly to the posixmodule.c;
3) Add the build configuration file [7] and necessary modifications
within Python to handle the edk2 toolchain;

This work would target the master branch (that is Python 3). I would
be interested in hearing your thoughts on this idea.

Thanks,
Thiebaud

[1] https://github.com/tianocore/edk2
[2] https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b36fab6690bf2b2fa11829
[3] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonReadMe.txt
[4] https://github.com/tianocore/edk2/commit/c8042e10763bca064df257547d04ae3dfcdfaf91
[5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
[6] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/Efi/edk2module.c
[7] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonCore.inf

Re: Official port of Python on EDK2

[Antoine Pitrou]

On Wed, 1 Nov 2017 13:46:27 +0100
Christian Heimes <christian@python.org> wrote:
> 
> > This work would target the master branch (that is Python 3). I would
> > be interested in hearing your thoughts on this idea.  
> 
> In general your proposal sounds like a good idea. A new platform may
> require a PEP, though.

It would also require a maintainer and a maintenance promise for
several years (5 or 10? I don't know).  I doubt any other core
developers are interested in/equipped for dealing with EDK2 issues,
regressions and subtleties.

> You can start now by submitting pull requests for the header fixes. Even
> in the case we decide not to support EDK2, we make your life easier by
> reducing the amount of extra patches.

Agreed.

Regards

Antoine.

Re: Official port of Python on EDK2

[Richardson, Brian]

Thiebaud:

Thank you. I have started discussions within Intel for updating the UEFI CPython implementation to Python 3.x. The TianoCore community would appreciate contributions by people with Python experience to bring this code up to current standards.

Please review the contribution guidelines for TianoCore and let me know if you have any questions.
http://www.tianocore.org/contrib/ 

Thanks ... br
---
Brian Richardson, Senior Technical Marketing Engineer, Intel Software
brian.richardson@intel.com -- @intel_brian (Twitter & WeChat)
https://software.intel.com/en-us/meet-the-developers/evangelists/team/brian-richardson 

-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Thiebaud Weksteen
Sent: Wednesday, November 1, 2017 5:07 AM
To: python-dev@python.org
Cc: edk2-devel@lists.01.org
Subject: [edk2] Official port of Python on EDK2

Hi,

UEFI has become the standard for firmware (BIOS) interface. Intel has provided an open source implementation under the name EDK2 (part of the TianoCore initiative) [1] for some time. This implementation has evolved significantly and now provides the functionalities of a small OS with a standard library similar to POSIX.

In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
This port then evolved to 2.7.2 which is still defined as the reference port [3]. In 2015, another port was added of Python 2.7.10 in parallel of 2.7.2 [4]. Since then, both implementations have diverged from upstream and know vulnerabilities have not been fixed.

I would like to bring support for edk2 in the official Python repository to remediate this situation, that is officially support
edk2 as a platform. Technically, there would be three main aspects for the on-boarding work:

1) Fix headers and source to resolve definition conflicts, similarly to ABS definition in [5];
2) Add the edk2module.c [6] to handle platform-specific functionalities, similarly to the posixmodule.c;
3) Add the build configuration file [7] and necessary modifications within Python to handle the edk2 toolchain;

This work would target the master branch (that is Python 3). I would be interested in hearing your thoughts on this idea.

Thanks,
Thiebaud

[1] https://github.com/tianocore/edk2
[2] https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b36fab6690bf2b2fa11829
[3] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonReadMe.txt
[4] https://github.com/tianocore/edk2/commit/c8042e10763bca064df257547d04ae3dfcdfaf91
[5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
[6] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/Efi/edk2module.c
[7] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonCore.inf
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: Official port of Python on EDK2

[Thiebaud Weksteen]

Christian, Antoine, Brett: Thanks for the clarification on what an official
support would require. As Christian mentioned, sending simple headers
patches is an obvious starting point, no matter if the support becomes
official or not.

Brian: Thanks for your email. As I suggested, by having the support
directly within the Python community, you would avoid having to maintain a
separate port. I don't think that having a new Python3 port as part of EDK2
is a good idea. What I am suggesting is that Intel should contribute
directly to the Python repository by sending your modifications upstream
and not expect someone to re-import Python into EDK2. That is, bringing
your UEFI experience to Python and not the opposite. This would be a much
better use of anyone's time.

Thanks,
Thiebaud

On Thu, Nov 2, 2017 at 2:36 AM, Richardson, Brian <
brian.richardson@intel.com> wrote:

> Thiebaud:
>
> Thank you. I have started discussions within Intel for updating the UEFI
> CPython implementation to Python 3.x. The TianoCore community would
> appreciate contributions by people with Python experience to bring this
> code up to current standards.
>
> Please review the contribution guidelines for TianoCore and let me know if
> you have any questions.
> http://www.tianocore.org/contrib/
>
> Thanks ... br
> ---
> Brian Richardson, Senior Technical Marketing Engineer, Intel Software
> brian.richardson@intel.com -- @intel_brian (Twitter & WeChat)
> https://software.intel.com/en-us/meet-the-developers/evangel
> ists/team/brian-richardson
>
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Thiebaud Weksteen
> Sent: Wednesday, November 1, 2017 5:07 AM
> To: python-dev@python.org
> Cc: edk2-devel@lists.01.org
> Subject: [edk2] Official port of Python on EDK2
>
> Hi,
>
> UEFI has become the standard for firmware (BIOS) interface. Intel has
> provided an open source implementation under the name EDK2 (part of the
> TianoCore initiative) [1] for some time. This implementation has evolved
> significantly and now provides the functionalities of a small OS with a
> standard library similar to POSIX.
>
> In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
> This port then evolved to 2.7.2 which is still defined as the reference
> port [3]. In 2015, another port was added of Python 2.7.10 in parallel of
> 2.7.2 [4]. Since then, both implementations have diverged from upstream and
> know vulnerabilities have not been fixed.
>
> I would like to bring support for edk2 in the official Python repository
> to remediate this situation, that is officially support
> edk2 as a platform. Technically, there would be three main aspects for the
> on-boarding work:
>
> 1) Fix headers and source to resolve definition conflicts, similarly to
> ABS definition in [5];
> 2) Add the edk2module.c [6] to handle platform-specific functionalities,
> similarly to the posixmodule.c;
> 3) Add the build configuration file [7] and necessary modifications within
> Python to handle the edk2 toolchain;
>
> This work would target the master branch (that is Python 3). I would be
> interested in hearing your thoughts on this idea.
>
> Thanks,
> Thiebaud
>
> [1] https://github.com/tianocore/edk2
> [2] https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b
> 36fab6690bf2b2fa11829
> [3] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/PythonReadMe.txt
> [4] https://github.com/tianocore/edk2/commit/c8042e10763bca064df
> 257547d04ae3dfcdfaf91
> [5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
> [6] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/Efi/edk2module.c
> [7] https://github.com/tianocore/edk2/blob/master/AppPkg/Applica
> tions/Python/PythonCore.inf
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: Official port of Python on EDK2

[Jayaprakash, N]

Would you consider adding thread support in this port of Python for EDK2 shell?

Regards,
JP

-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Richardson, Brian
Sent: Thursday, November 2, 2017 7:07 AM
To: Thiebaud Weksteen <tweek@google.com>; python-dev@python.org
Cc: edk2-devel@lists.01.org
Subject: Re: [edk2] Official port of Python on EDK2

Thiebaud:

Thank you. I have started discussions within Intel for updating the UEFI CPython implementation to Python 3.x. The TianoCore community would appreciate contributions by people with Python experience to bring this code up to current standards.

Please review the contribution guidelines for TianoCore and let me know if you have any questions.
http://www.tianocore.org/contrib/ 

Thanks ... br
---
Brian Richardson, Senior Technical Marketing Engineer, Intel Software brian.richardson@intel.com -- @intel_brian (Twitter & WeChat) https://software.intel.com/en-us/meet-the-developers/evangelists/team/brian-richardson 

-----Original Message-----
From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Thiebaud Weksteen
Sent: Wednesday, November 1, 2017 5:07 AM
To: python-dev@python.org
Cc: edk2-devel@lists.01.org
Subject: [edk2] Official port of Python on EDK2

Hi,

UEFI has become the standard for firmware (BIOS) interface. Intel has provided an open source implementation under the name EDK2 (part of the TianoCore initiative) [1] for some time. This implementation has evolved significantly and now provides the functionalities of a small OS with a standard library similar to POSIX.

In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2].
This port then evolved to 2.7.2 which is still defined as the reference port [3]. In 2015, another port was added of Python 2.7.10 in parallel of 2.7.2 [4]. Since then, both implementations have diverged from upstream and know vulnerabilities have not been fixed.

I would like to bring support for edk2 in the official Python repository to remediate this situation, that is officially support
edk2 as a platform. Technically, there would be three main aspects for the on-boarding work:

1) Fix headers and source to resolve definition conflicts, similarly to ABS definition in [5];
2) Add the edk2module.c [6] to handle platform-specific functionalities, similarly to the posixmodule.c;
3) Add the build configuration file [7] and necessary modifications within Python to handle the edk2 toolchain;

This work would target the master branch (that is Python 3). I would be interested in hearing your thoughts on this idea.

Thanks,
Thiebaud

[1] https://github.com/tianocore/edk2
[2] https://github.com/tianocore/edk2/commit/006fecd5a177b4b7b6b36fab6690bf2b2fa11829
[3] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonReadMe.txt
[4] https://github.com/tianocore/edk2/commit/c8042e10763bca064df257547d04ae3dfcdfaf91
[5] https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e
[6] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/Efi/edk2module.c
[7] https://github.com/tianocore/edk2/blob/master/AppPkg/Applications/Python/PythonCore.inf
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: Official port of Python on EDK2

[Blibbet]

On 11/02/2017 09:41 AM, Jayaprakash, N wrote:
> Would you consider adding thread support in this port of Python for
EDK2 shell?

FYI, this library adds thread support to UEFI:

https://github.com/Openwide-Ingenierie/GreenThreads-UEFI

Note that the library is GPLv2, ...but the author (a 1-person project)
could be asked to relicense to BSD to fit into Tianocore.

Note that library is currently Intel x64-centric, and contains a bit of
assembly. Will need some ARM/RISC-V/x86 contributions.

HTH,
Lee Fisher

Re: Official port of Python on EDK2

[Michael Zimmermann]

> FYI, this library adds thread support to UEFI:
>
> https://github.com/Openwide-Ingenierie/GreenThreads-UEFI

IMO this library has some crucial problems like changing the TPL
during context switching.
For my project "EFIDroid" I've invested many months analyzing, testing
and implementing my own threading implementation based on
LK(LittleKernel, a MIT licensed project) threads and get/set -context.

The result is a pretty stable implementation which can even be used in
UEFI drivers:
https://github.com/efidroid/uefi_edk2packages_EFIDroidLKLPkg/tree/master/UEFIThreads
I'm currently using this lib for my LKL(LinuxKernelLibrary) port to be
able to use linux touchscreen drivers in UEFI - so you could say it
has been well tested.

The only "problem" is that it only supports ARM right now and that the
get/set context implementation was copied (and simplified) from glibc
which means that this part is GPL code.

Thanks
Michael Zimmermann

On Thu, Nov 2, 2017 at 8:37 PM, Blibbet <blibbet@gmail.com> wrote:
> On 11/02/2017 09:41 AM, Jayaprakash, N wrote:
>> Would you consider adding thread support in this port of Python for
> EDK2 shell?
>
> FYI, this library adds thread support to UEFI:
>
> https://github.com/Openwide-Ingenierie/GreenThreads-UEFI
>
> Note that the library is GPLv2, ...but the author (a 1-person project)
> could be asked to relicense to BSD to fit into Tianocore.
>
> Note that library is currently Intel x64-centric, and contains a bit of
> assembly. Will need some ARM/RISC-V/x86 contributions.
>
> HTH,
> Lee Fisher
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel