Re: [edk2-devel] [edk2-libc Patch 1/1] AppPkg\Applications\Python\Python-3.6.8: GCC tool chain support

["Michael D Kinney" <michael.d.kinney@intel.com>]

pushed as 71dab4b0eaa4214586b9e03c1c42c7562095353c

Mike

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Tuesday, May 31, 2022 4:45 PM
> To: devel@edk2.groups.io; Jayaprakash, N <n.jayaprakash@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: RE: [edk2-devel] [edk2-libc Patch 1/1] AppPkg\Applications\Python\Python-3.6.8: GCC tool chain support
> 
> Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>
> 
> I do see that there are 3 documentation files in Python-3.6.8.
> 
> One is from the Python project.  The other 2 look like they both
> contain EDK II build instructions.  Perhaps these should be combined into
> a single file for EDK II builds for all supported tool chains and archs.
> 
> Mike
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Jayaprakash, N
> > Sent: Wednesday, May 18, 2022 11:42 AM
> > To: devel@edk2.groups.io
> > Subject: [edk2-devel] [edk2-libc Patch 1/1] AppPkg\Applications\Python\Python-3.6.8: GCC tool chain support
> >
> >    REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3779
> >
> >    This commit contains the changes made to Python interpreter
> >    3.6.8 to cross compile using the GCC tool chain on Ubuntu 20.04 LTS
> >    system for UEFI. The ctypes module has been excluded from GCC compilation as it requires
> >    additional porting work to use GCC tool chain support for UEFI environment.
> >    Added GCC Compilation steps as BKM document to facilitate the environment setup and
> >    compilation on Linux using GCC tool chain. Besides this provided a shell script to
> >    generate the usable Python UEFI package with all the dependencies taken from the
> >    build environment.
> >
> >    Cc: Rebecca Cran <rebecca@nuviainc.com>
> >    Cc: Michael D Kinney <michael.d.kinney@intel.com>
> >    Signed-off-by: Jayaprakash N <n.jayaprakash@intel.com>
> > ---
> >  .../Python-3.6.8/GCCCompilationBKMs.rst       | 193 ++++++++++++++++++
> >  .../PyMod-3.6.8/Include/pyconfig.h            |   2 +-
> >  .../Python-3.6.8/PyMod-3.6.8/Modules/config.c |   7 +-
> >  .../PyMod-3.6.8/Modules/getpath.c             |   2 +-
> >  .../Python/Python-3.6.8/Python368.inf         |  21 +-
> >  .../Python-3.6.8/create_python368_pkg.sh      | 100 +++++++++
> >  6 files changed, 312 insertions(+), 13 deletions(-)
> >  create mode 100644 AppPkg/Applications/Python/Python-3.6.8/GCCCompilationBKMs.rst
> >  create mode 100644 AppPkg/Applications/Python/Python-3.6.8/create_python368_pkg.sh
> >
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/GCCCompilationBKMs.rst b/AppPkg/Applications/Python/Python-
> > 3.6.8/GCCCompilationBKMs.rst
> > new file mode 100644
> > index 0000000..ed5fcbe
> > --- /dev/null
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/GCCCompilationBKMs.rst
> > @@ -0,0 +1,193 @@
> > +1. Introduction
> > +---------------
> > +This is a Best-Known Methods document capturing the details of build environment setup for compiling the Python UEFI
> > +interpreter version 3.6.8 using GCC tool chain on Linux based operating systems. These BKMs have been tested with WSL2
> > +(Windows Subsystem for Linux 2) based Ubuntu 20.04 LTS environment on Windows 10. Only 64-bit builds are possible due
> > +to a limitation in the LibC support for GCC compiler tool chain on UEFI. If you are working on a Ubuntu 20.04 LTS system
> > +you may skip the sections 1.1 and 1.2
> > +
> > +
> > +1.1.  WSL2 Installation on Windows 10 OS
> > +----------------------------------------
> > +
> > +The WSL2 environment can be setup by following the instructions given in the below webpage
> > +https://pureinfotech.com/install-windows-subsystem-linux-2-windows-10/
> > +
> > +1.2.  Installing Ubuntu 20.04 LTS on WSL2 on Windows 10 OS
> > +----------------------------------------------------------
> > +
> > +The instructions provided in the below webpage will help in installing the Ubuntu 20.04 LTS OS environment on WSL2 on Windows
> 10
> > OS.
> > +https://www.altisconsulting.com/au/insights/installing-ubuntu-bash-for-windows-10-wsl2-setup/
> > +
> > +2. EDK2 build environment setup
> > +-------------------------------
> > +
> > +You may follow the instructions provided in the below webpage to setup the edk2 build environment.
> > +
> > +https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC
> > +
> > +For the purposes of describing the BKMs, we will be using the following paths.
> > +Edk2 source tree:	$HOME/src/edk2
> > +Native GCC version >=4.x compiler installation:	/usr/bin/gcc
> > +Intel ASL Compiler installation:	/usr/bin/iasl
> > +
> > +
> > +2.1.  Installing essential packages for GCC compiler
> > +----------------------------------------------------
> > +
> > +Install the below required essential packages to facilitate compilation of the edk2 repo using GCC compiler tool chain.
> > +Several Ubuntu packages will be needed to set up the build environment for EDK II.
> > +
> > +The following command will install all required packages::
> > +
> > +        bash$ sudo apt-get update
> > +        bash$ sudo apt install build-essential
> > +        bash$ sudo apt install uuid-dev iasl git  nasm  python-is-python3
> > +
> > +Package - Description
> > +    * build-essential : Informational list of build-essential packages
> > +    * uuid-dev : Universally Unique ID library (headers and static libraries)
> > +    * iasl : Intel ASL compiler/decompiler (also provided by acpica-tools)
> > +    * git : Support for git revision control system
> > +    * nasm : General-purpose x86 assembler
> > +    * python-is-python3	: Ubuntu 20.04 python command is 'python3' but edk2 tools use 'python'
> > +
> > +Once the above packages are installed, then follow the instructions in the webpage to setup the rest of the build
> > +environment
> > +https://github.com/tianocore/tianocore.github.io/wiki/Common-instructions
> > +
> > +
> > +2.2.  Cloning edk2 git repo
> > +---------------------------
> > +
> > +Get the edk2 source tree using Git commands.
> > +Execute the below commands to clone the edk2 git repo::
> > +        bash$ mkdir ~/src
> > +        bash$ cd ~/src
> > +        bash$ git clone https://github.com/tianocore/edk2
> > +
> > +This will create a folder named edk2 as a clone of the edk2 git repo.
> > +Execute the below commands to checkout / initialize all the git submodules::
> > +        bash$ cd ~/src/edk2
> > +        bash$ git submodule update --init
> > +
> > +The above command initializes all the submodules in the git repo
> > +The below commands help to setup the environment variables required for building the edk2 repo::
> > +
> > +        bash$ make -C BaseTools
> > +        bash$ . edksetup.sh
> > +
> > +Run the below make command from ~/src folder i.e change working directory to ~/src and run the make command as given below ::
> > +
> > +        bash$ cd ~/src
> > +        bash$ make -C edk2/BaseTools
> > +
> > +Setup build shell environment ::
> > +
> > +        bash$ cd ~/src/edk2
> > +        bash$ export EDK_TOOLS_PATH=$HOME/src/edk2/BaseTools
> > +        bash$ . edksetup.sh BaseTools
> > +
> > +
> > +Set Build Target Information
> > +For the Conf/target.txt file, find the following lines:
> > +ACTIVE_PLATFORM       = Nt32Pkg/Nt32Pkg.dsc
> > +
> > +TOOL_CHAIN_TAG        = MYTOOLS
> > +
> > +And change the corresponding lines to match these:
> > +
> > +ACTIVE_PLATFORM       = MdeModulePkg/MdeModulePkg.dsc
> > +
> > +TOOL_CHAIN_TAG        = GCC5
> > +
> > +Install GCC5 on Ubuntu 20.04 LTS by downloading it from
> > +https://askubuntu.com/questions/1235819/ubuntu-20-04-gcc-version-lower-than-gcc-7
> > +
> > +This requires latest version of the nasm tool to be installed.
> > +The nasm latest version >= 2.15.05
> > +
> > +Debian package for nasm 2.15.05 is available at
> > +
> > +https://ubuntu.pkgs.org/21.10/ubuntu-universe-amd64/nasm_2.15.05-1_amd64.deb.html
> > +
> > +Download it using the wget command as given below::
> > +
> > +        bash$ sudo wget http://archive.ubuntu.com/ubuntu/pool/universe/n/nasm/nasm_2.15.05-1_amd64.deb
> > +
> > +
> > +Install the the above downloaded nasm debian package using the following command::
> > +
> > +        bash$ sudo dpkg -i  nasm_2.15.05-1_amd64.deb
> > +
> > +2.3.  Install additional packages required for building edk2 with GCC
> > +---------------------------------------------------------------------
> > +
> > +Install additional package required for building edk2 repo::
> > +
> > +        bash$ sudo apt install libx11-dev
> > +        bash$ sudo apt install libxext-dev
> > +
> > +
> > +2.4  Clone edk2-libc and copy the contents to edk2
> > +--------------------------------------------------
> > +
> > +Now clone the edk2-libc git repo by following / executing the below commands::
> > +
> > +        bash$ cd ~/src
> > +        bash$ git clone https://github.com/tianocore/edk2-libc.git
> > +        bash$ export PACKAGES_PATH=$HOME/src/edk2:$HOME/src/edk2-libc
> > +
> > +
> > +Verify that you can build the hello world application from AppPkg under edk2-libc by running the following commands
> > +Build hello.inf from edk2-libc applications::
> > +
> > +        $bash build -p AppPkg/AppPkg.dsc -m AppPkg/Applications/Hello/Hello.inf
> > +
> > +Copy the contents of edk2-libc to edk2 folder::
> > +
> > +        bash$ cp -rf ~/src/edk2-libc/* ~/src/edk2/
> > +
> > +2.5.  Build Python Interpreter using GCC
> > +----------------------------------------
> > +Uncomment the line containing following inf file name in AppPkg/AppPkg.dsc file
> > +AppPkg/Applications/Python/Python-3.6.8/Python368.inf
> > +
> > +Execute the below command to build the X64 version of the Python 3.6.8 interpreter using GCC tool chain::
> > +
> > +        bash$ cd AppPkg/Applications/Python/Python-3.6.8/
> > +        bash$ python srcprep.py
> > +        bash$ cd ~/src/edk2
> > +        bash$ build -a X64 -b RELEASE -p AppPkg/AppPkg.dsc -m AppPkg/Applications/Python/Python-3.6.8/Python368.inf
> > +             or
> > +        bash$ build -a X64 -b RELEASE -p AppPkg/AppPkg.dsc
> > +
> > +
> > +2.5  Create Python UEFI package with all dependencies
> > +-----------------------------------------------------
> > +
> > +To create an usable Python UEFI package with all the dependencies from the build environment,
> > +you can use the bash shell script create_python368_pkg.sh available under /AppPkg/Applications/Python/Python-3.6.8
> > +folder.
> > +
> > +Use the following commands to create Python UEFI package::
> > +
> > +        bash$ cd ~/src/edk2/AppPkg/Applications/Python/Python-3.6.8/
> > +        bash$ chmod 777 create_python368_pkg.sh
> > +        ./create_python368_pkg.sh GCC5 RELEASE X64 myPyUEFI
> > +
> > +where GCC5 refers to the tool chain, RELEASE refers to the build type, X64 refers to the architecture
> > +myPyUEFI refers to the name of the folder where the Python UEFI package needs to be created.
> > +
> > +When you run the create_python368_pkg.sh without any parameters it lists the help
> > +information as shown below:
> > +
> > +Shell Script to create Python EFI Package.
> > +
> > +Usage: ./create_python368_pkg.sh <ToolChain> <Target> <Architecture> <OutFolder>
> > +
> > +Where
> > +    * ToolChain     :  name of the tool chain such as GCC5
> > +    * Target        :  build type such as RELEASE, DEBUG
> > +    * Architecture  :  Architecture such as X64
> > +    * OutFolder     :  Output directory for creating the package
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Include/pyconfig.h b/AppPkg/Applications/Python/Python-
> > 3.6.8/PyMod-3.6.8/Include/pyconfig.h
> > index d4685da..9d08982 100644
> > --- a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Include/pyconfig.h
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Include/pyconfig.h
> > @@ -870,7 +870,7 @@
> >  #undef HAVE_THREAD_H
> >
> >  /* Define to 1 if you have the 'timegm' function. */
> > -#undef HAVE_TIMEGM
> > +#define HAVE_TIMEGM 1
> >
> >  /* Define to 1 if you have the 'times' function. */
> >  #undef HAVE_TIMES
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/config.c b/AppPkg/Applications/Python/Python-
> > 3.6.8/PyMod-3.6.8/Modules/config.c
> > index 0e7f65e..7217a80 100644
> > --- a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/config.c
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/config.c
> > @@ -83,7 +83,10 @@ extern PyObject* PyInit__stat(void);
> >  extern PyObject* PyInit__opcode(void);
> >  extern PyObject* PyInit_faulthandler(void);
> >  // _ctypes
> > +#if defined(UEFI_MSVC_32) || defined(UEFI_MSVC_64)
> >  extern PyObject* PyInit__ctypes(void);
> > +#endif
> > +
> >  extern PyObject* init_sqlite3(void);
> >
> >  // EfiPy
> > @@ -155,7 +158,9 @@ struct _inittab _PyImport_Inittab[] = {
> >      {"_string", PyInit__string},
> >      {"_stat", PyInit__stat},
> >      {"_opcode", PyInit__opcode},
> > -	{ "_ctypes", PyInit__ctypes },
> > +#if defined(UEFI_MSVC_32) || defined(UEFI_MSVC_64)
> > +    { "_ctypes", PyInit__ctypes },
> > +#endif
> >      /* Sentinel */
> >      {0, 0}
> >  };
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/getpath.c b/AppPkg/Applications/Python/Python-
> > 3.6.8/PyMod-3.6.8/Modules/getpath.c
> > index ad10784..7ee4d5e 100644
> > --- a/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/getpath.c
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/PyMod-3.6.8/Modules/getpath.c
> > @@ -112,7 +112,7 @@
> >  #endif
> >
> >
> > -#define SIFY_I( x ) L#x
> > +#define SIFY_I( x ) L"" #x
> >  #define SIFY( y )   SIFY_I( y )
> >
> >  #ifndef PREFIX
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/Python368.inf b/AppPkg/Applications/Python/Python-3.6.8/Python368.inf
> > index 4ab0cd3..207b907 100644
> > --- a/AppPkg/Applications/Python/Python-3.6.8/Python368.inf
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/Python368.inf
> > @@ -251,23 +251,24 @@ Modules/zlib
> >    Modules/zlib/zutil.c                 #
> >
> >  #Modules/ctypes
> > -  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/_ctypes.c              #
> > -  Modules/_ctypes/stgdict.c              #
> > -  Modules/_ctypes/libffi_msvc/prep_cif.c #
> > -  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/malloc_closure.c       #
> > -  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/libffi_msvc/ffi.c      #
> > -  Modules/_ctypes/cfield.c               #
> > -  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/callproc.c             #
> > -  Modules/_ctypes/callbacks.c            #
> > +  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/_ctypes.c    | MSFT          #
> > +  Modules/_ctypes/stgdict.c           | MSFT   #
> > +  Modules/_ctypes/libffi_msvc/prep_cif.c  | MSFT#
> > +  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/malloc_closure.c   | MSFT    #
> > +  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/libffi_msvc/ffi.c  | MSFT     #
> > +  Modules/_ctypes/cfield.c    | MSFT           #
> > +  PyMod-$(PYTHON_VERSION)/Modules/_ctypes/callproc.c  | MSFT           #
> > +  Modules/_ctypes/callbacks.c        | MSFT    #
> >
> >  [Sources.IA32]
> > -  Modules/_ctypes/libffi_msvc/win32.c   #
> > +  Modules/_ctypes/libffi_msvc/win32.c  | MSFT #
> >
> >  [Sources.X64]
> > -  Modules/_ctypes/libffi_msvc/win64.asm   #
> > +  Modules/_ctypes/libffi_msvc/win64.asm  | MSFT #
> >
> >  [BuildOptions]
> >    MSFT:*_*_*_CC_FLAGS = /GL- /Oi- /wd4018 /wd4054 /wd4055 /wd4101 /wd4131 /wd4152 /wd4204 /wd4210 /wd4244 /wd4267 /wd4305
> > /wd4310 /wd4389 /wd4701 /wd4702 /wd4706 /wd4456 /wd4312 /wd4457 /wd4459 /wd4474 /wd4476
> > /I$(WORKSPACE)\AppPkg\Applications\Python\Python-3.6.8\Include /DHAVE_MEMMOVE /DUSE_PYEXPAT_CAPI /DXML_STATIC -D UEFI /WX-
> > /DXML_POOR_ENTROPY /DUEFI_C_SOURCE
> > +  GCC:*_*_*_CC_FLAGS = -Wno-unused-function -Wno-format -Wno-error -fno-strict-aliasing -
> > I$(WORKSPACE)/AppPkg/Applications/Python/Python-3.6.8/Include -DHAVE_MEMMOVE -DUSE_PYEXPAT_CAPI -DXML_STATIC -DXML_POOR_ENTROPY
> >
> >  [BuildOptions.IA32]
> >    MSFT:*_*_*_CC_FLAGS = /DUEFI_MSVC_32
> > diff --git a/AppPkg/Applications/Python/Python-3.6.8/create_python368_pkg.sh b/AppPkg/Applications/Python/Python-
> > 3.6.8/create_python368_pkg.sh
> > new file mode 100644
> > index 0000000..f529fa4
> > --- /dev/null
> > +++ b/AppPkg/Applications/Python/Python-3.6.8/create_python368_pkg.sh
> > @@ -0,0 +1,100 @@
> > +#!/bin/bash
> > +
> > +TOOL_CHAIN_TAG=$1
> > +TARGET=$2
> > +ARCH=$3
> > +OUT_FOLDER=$4
> > +
> > +function usage {
> > +    echo
> > +    echo Shell Script to create Python EFI Package.
> > +    echo
> > +    echo
> > +    echo "Usage: $0 <ToolChain> <Target> <Architecture> <OutFolder>"
> > +    echo "Where "
> > +    echo "    ToolChain     :  name of the tool chain such as GCC5"
> > +    echo "    Target        :  build type such as RELEASE, DEBUG"
> > +    echo "    Architecture  :  Architecture such as X64"
> > +    echo "    OutFolder     :  Output directory for creating the package"
> > +    echo
> > +}
> > +
> > +function error {
> > +    echo Failed to Create Python EFI Package
> > +    echo Python368.efi application is not available at "Build/AppPkg/$TARGET"_"$TOOL_CHAIN_TAG/$ARCH/"
> > +    echo Follow the instructions in Py368ReadMe.txt to build Python interpreter
> > +    echo Then use this script to create a Python EFI package
> > +}
> > +
> > +
> > +# Validate the ToolChain input
> > +if [ -z $TOOL_CHAIN_TAG ]
> > +then
> > +    echo Invalid command line arguments passed, please see the below usage instructions
> > +    usage
> > +    exit
> > +fi
> > +
> > +# Validate the Target input
> > +if [ -z $TARGET ]
> > +then
> > +    echo Invalid command line arguments passed, please see the below usage instructions
> > +   usage
> > +   exit
> > +fi
> > +
> > +# Validate the Architecture input
> > +if [ -z $ARCH ]
> > +then
> > +    echo Invalid command line arguments passed, please see the below usage instructions
> > +    usage
> > +    exit
> > +fi
> > +
> > +# Validate the OutFolder input
> > +if [ -z $OUT_FOLDER ]
> > +then
> > +    echo Invalid command line arguments passed, please see the below usage instructions
> > +   usage
> > +   exit
> > +fi
> > +
> > +cd ../../../../
> > +PYTHON_BIN="Build/AppPkg/$TARGET"_"$TOOL_CHAIN_TAG/$ARCH/Python368.efi"
> > +if [ ! -f $PYTHON_BIN ]
> > +then
> > +    error
> > +    exit
> > +fi
> > +
> > +
> > +# create /EFI/Tools folder in the Output folder if it doesn't exists
> > +if [ ! -d $OUT_FOLDER/EFI/Tools ]
> > +then
> > +    mkdir -p $OUT_FOLDER/EFI/Tools
> > +fi
> > +
> > +cp -f $PYTHON_BIN $OUT_FOLDER/EFI/Tools/
> > +
> > +if [ ! -d $OUT_FOLDER/EFI/StdLib/lib/python36.8 ]
> > +then
> > +    mkdir -p $OUT_FOLDER/EFI/StdLib/lib/python36.8
> > +fi
> > +
> > +if [ ! -d $OUT_FOLDER/EFI/StdLib/etc ]
> > +then
> > +    mkdir -p $OUT_FOLDER/EFI/StdLib/etc
> > +fi
> > +
> > +cp -rf AppPkg/Applications/Python/Python-3.6.8/Lib/*  $OUT_FOLDER/EFI/StdLib/lib/python36.8/
> > +cp -rf StdLib/Efi/StdLib/etc/*  $OUT_FOLDER/EFI/StdLib/etc/
> > +
> > +if [ ${OUT_FOLDER:0:1} == "/" ]
> > +then
> > +    echo Python EFI package available at $OUT_FOLDER
> > +else
> > +    echo Python EFI package available at `pwd`/$OUT_FOLDER
> > +fi
> > +
> > +
> > +
> > --
> > 2.33.0.windows.1
> >
> >
> >
> > 
> >