From mboxrd@z Thu Jan 1 00:00:00 1970 Subject: RFC for Edk2-ToolEnv To: devel@edk2.groups.io From: "Sean" X-Originating-IP: 131.107.174.83 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Mon, 13 May 2019 19:55:06 -0700 Message-ID: <5965.1557802506227860431@groups.io> Content-Type: multipart/alternative; boundary="4jyowAVbcqdAKr7eRo8t" --4jyowAVbcqdAKr7eRo8t Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable RFC=C2=A0 Edk2-ToolEnv creation Create a new tianocore owned repository to host python code to support an = extensible, pluggable, rich environment.=C2=A0 This environment has command= line interfaces to support building a product, building CI, running tests,= and downloading dependencies. This environment also provides the building = blocks for developers to write their own tools to launch in the environment= and leverage the capabilities provided by the environment. The unique capa= bilities provided help support building products with multiple repositories= and having each repository contribute/plugin to the build process in a sca= lable way. The environment will scan the files in the code tree (multiple r= epos) and discover plugins, dependencies, path adjustments, environment var= iable settings, etc. This provides easy methods for common repos to share b= uild tools/steps. Inclusion of this package and dependency management should be managed usin= g Pip/Pypi.=C2=A0 =C2=A0To start this is a supplemental package and is not = required to be used for edk2 builds. More details below but much of this content is coming from combining two e= xisting repos * https://github.com/microsoft/mu_pip_environment ( https://github.com/mic= rosoft/mu_pip_environment ) * https://github.com/microsoft/mu_pip_build ( https://github.com/microsoft= /mu_pip_build ) This RFC is part 2 of content from design review: https://edk2.groups.io/g= /devel/files/Designs/2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%2= 0-%20Design%20Review%20.pdf ( https://edk2.groups.io/g/devel/files/Designs/= 2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%= 20.pdf ) Examples of content here * CI build support - https://github.com/microsoft/mu_pip_build/blob/master= /MuBuild/MuBuild.py ( https://github.com/microsoft/mu_pip_build/blob/master= /MuBuild/MuBuild.py ) * Binary Dependency resolution * Nuget - https://github.com/microsoft/mu_pip_environment/blob/master/MuEn= vironment/ExternalDependencies.py ( https://github.com/microsoft/mu_pip_env= ironment/blob/master/MuEnvironment/ExternalDependencies.py ) * GitHub Releases - WIP * URL dependency - WIP * Logging * File logger * Markdown logger * In Memory loggers - For tool parsing * Console loggers with colors * PlugIns * Support pre/post build steps * Support supplying functions * UefiBuild - A wrapper around Edk2 build process that a Platform would su= bclass. https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvir= onment/UefiBuild.py ( https://github.com/microsoft/mu_pip_environment/blob/= master/MuEnvironment/UefiBuild.py ) * VarDict and ShellEnvironment - Manage the shell environment and all the = name/value pairs used in the build process (including pre/post). * Omnicache - Support a super cache of git repos to speed up creating and = updating multiple work spaces=C2=A0and minimizing filesystem impact Maintainers * Sean Brogan * Bret Barkelew * Placeholder for existing maintainer from the basetools License * BSD + Patent (edk2 aligned) Contribution process and issue tracking * Follow Github PR process for contributions and issue tracking * Contributor forks repo in github * Contributor creates branch for work * Contributor updates release notes to indicate change (if necessary) * Contributor submits PR to master branch of tianocore/Edk2-ToolEnv repo * Review feedback is given in PR * Python Tests are run on the repo (new contributions need unit tests) * Python Style (flake8) must pass * All review feedback must be completed, maintainers approved, and tests r= un successfully before PR is *squash merged* into master Documentation * Use Github IO documentation/wiki hosting * Example content i. https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/ ( htt= ps://microsoft.github.io/mu/dyn/mu_pip_environment/developing/ ) ii. https://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/ ( ht= tps://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/ ) * Readme at root of repo * Example: https://github.com/Microsoft/mu_pip_environment ( https://githu= b.com/Microsoft/mu_pip_environment ) CI Builds * CI build process using dev ops * Validation is done thru build process * Release publication done thru manual CI Build * Examples from Mu-Environment * Windows CI - https://dev.azure.com/projectmu/mu%20pip/_build?definitionI= d=3D10 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=3D10 = ) * Linux CI - https://dev.azure.com/projectmu/mu%20pip/_build?definitionId= =3D11 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=3D11 = ) * Publishing - https://dev.azure.com/projectmu/mu%20pip/_build?definitionI= d=3D17 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=3D17 = ) Release * Release to Pypi as Edk2-ToolEnv for easy usage in product environment * Versioned follows: Aa.bb.cc.dd * AA =3D=3D Major version.=C2=A0 Changes don=E2=80=99t need to be backward= compatible * BB =3D=3D Minor version.=C2=A0 Significant new features.=C2=A0 Backward = compatibility maintained * CC =3D=3D Bug fix/patch/small optional feature * DD =3D=3D build/Release version. * Package on Pypi will be owned by Tianocore group * Example for mu-environment: https://pypi.org/project/mu-environment/ ( h= ttps://pypi.org/project/mu-environment/ ) Other Notes * Only support Python 3 (prefer 3.7+) * This was discussed on the edk2 design meetings (4/18) https://edk2.group= s.io/g/devel/files/Designs/2019/0418 ( https://edk2.groups.io/g/devel/files= /Designs/2019/0418 ) * There is RFC #2. The 1 st RFC is for Edk2-Library which this RFC depends= on. * Simple example of usage on an open platform: https://github.com/ms-iot/M= U_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py = ( https://github.com/ms-iot/MU_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EV= K_2GB/PlatformBuildWorker.py ) * Example of CI of something like Tianocore: * Windows - Mu_Basecore: https://dev.azure.com/projectmu/mu/_build?definit= ionId=3D4&view=3DbuildsHistory ( https://dev.azure.com/projectmu/mu/_build?= definitionId=3D4&view=3DbuildsHistory ) * Linux - Mu_Basecore: https://dev.azure.com/projectmu/mu/_build?definitio= nId=3D19&view=3DbuildsHistory ( https://dev.azure.com/projectmu/mu/_build?d= efinitionId=3D19&view=3DbuildsHistory ) * Windows - Project Mu Plus code: https://dev.azure.com/projectmu/mu/_buil= d?definitionId=3D6&view=3DbuildsHistory ( https://dev.azure.com/projectmu/m= u/_build?definitionId=3D6&view=3DbuildsHistory ) Timeline * RFC open for comment thru 5/21/2019 since the same as RFC Edk2-Library --4jyowAVbcqdAKr7eRo8t Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

RFC=C2=A0 Edk2-ToolEnv creation

=C2=A0

Create a new tianocore owned repository to host python code to support an = extensible, pluggable, rich environment.=C2=A0 This environment has command= line interfaces to support building a product, building CI, running tests,= and downloading dependencies.=C2=A0 This environment also provides the building blocks for developers to wr= ite their own tools to launch in the environment and leverage the capabilit= ies provided by the environment. =C2=A0<= /span>The unique capabilities provided help support building products with = multiple repositories and having each repository contribute/plugin to the b= uild process in a scalable way. =C2=A0The environment will scan the files in the code tree (multiple repos) = and discover plugins, dependencies, path adjustments, environment variable = settings, etc.=C2=A0=C2=A0 This p= rovides easy methods for common repos to share build tools/steps.=C2=A0

=C2=A0

Inclusion of this package and dependency management should be managed usin= g Pip/Pypi.=C2=A0 =C2=A0To start this is a supplemental package and is not = required to be used for edk2 builds.

=C2=A0

<= span style=3D"mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-l= atin; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; ms= o-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-b= idi;">More details below but much of this content is coming from combining = two existing repos

  1. https://github.com/microsoft/mu_pip_environment
  2. https://github.com/microsoft/mu_pip_build

= = =C2=A0

This RFC is part 2 of content from design review: https://edk2.groups.io/g/devel/files/Designs/2019/0418/201= 9-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%20.pdf=

=C2=A0

=C2=A0

E= xamples of content here

  1. CI build support - https://github.co= m/microsoft/mu_pip_build/blob/master/MuBuild/MuBuild.py
  2. Binary Dependency resolution<= /span>
    1. Nuget - h= ttps://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/Ex= ternalDependencies.py
    2. GitHub Releases - WIP<= /span>
    3. URL dependency - WIP
  3. Logging
    1. File logger
    2. Markdown logger=
    3. In Memory loggers - Fo= r tool parsing
    4. Console loggers with c= olors
  4. PlugIns
    1. Support pre/post build= steps
    2. Support supplying func= tions
  5. UefiBuild - A wrapper = around Edk2 build process that a Platform would subclass.=C2=A0 https://github.com/microsoft/mu_pip_envi= ronment/blob/master/MuEnvironment/UefiBuild.py
  6. VarDict and ShellEnviron= ment - Manage the shell environment and all the name/value pairs used in th= e build process (including pre/post).=C2= = =A0
  7. Omnicache - Support a super cache of git repos to speed up creat= ing and updating multiple work spaces=C2=A0and minimizing filesystem impact=

Maintainers

  1. Sean Brogan
  2. Bret Barkelew
  3. Placeholder for existing mainta= iner from the basetools

License

  1. BSD + Patent (edk2 aligned)

Contribution process and issue tracking

  1. Follow Github PR process for c= ontributions and issue tracking
    1. Contributor forks repo in git= hub
    2. Contributor creates branch fo= r work
    3. Contributor updates release n= otes to indicate change (if necessary)
    4. Contributor submits PR to mas= ter branch of tianocore/Edk2-ToolEnv repo
    5. Review feedback is given in P= R
    6. Python Tests are run on the r= epo (new contributions need unit tests)
    7. Python Style (flake8) must pa= ss
    8. All review feedback must be com= pleted, maintainers approved, and tests run successfully before PR is squash merged into master

Documentation

  1. Use Github IO documentation/wi= ki hosting
    1. Example content=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 i.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ii.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <= !--[endif]-->https:/= /microsoft.github.io/mu/dyn/mu_pip_environment/publishing/

  1. Readme at root of repo=
    1. Example: https://github.com/Microsoft/mu_pip_environment

CI Builds

  1. CI build process using dev ops=
  2. Validation is done thru build= process
  3. Release publication done thru= manual CI Build
  4. Examples from Mu-Environment<= /span>
    1. Windows CI - https://dev.azure.com/project= mu/mu%20pip/_build?definitionId=3D10
    2. Linux CI - <= span style=3D"mso-bookmark: _MailOriginal;">https://dev.azure.com/projectmu= /mu%20pip/_build?definitionId=3D11<= /li>
    3. Publishing - <= span style=3D"mso-bookmark: _MailOriginal;">https://dev.azure.com/projectmu= /mu%20pip/_build?definitionId=3D17<= /li>

Release

  1. Release to Pypi as Edk2-ToolEn= v for easy usage in product environment
    1. Versioned follows: Aa.bb.cc.d= d
    2. AA =3D=3D Major version.=C2= =A0 Changes don=E2=80=99t need to be backward compatible
    3. BB =3D=3D Minor version.=C2= =A0 Significant new features.=C2=A0 Backward compatibility maintained
    4. CC =3D=3D Bug fix/patch/small= optional feature
    5. DD =3D=3D build/Release versi= on.=C2=A0
    6. Package on Pypi will be owned= by Tianocore group
  2. Example for mu-environment: https://pypi.org/project/mu-environment/=

Other Notes

  1. Only support Python 3 (prefer = 3.7+)
  2. This was discus= sed on the edk2 design meetings (4/18) https://edk2.groups.io/g/devel/files/Designs/2019/0418
  3. There is RFC #2= .=C2=A0 The 1st RFC is= for Edk2-Library which this RFC depends on.=C2=A0
  4. Simple example = of usage on an open platform:=C2=A0 https://github.com/ms-iot/MU_PLATFORM_NXP/blob/ma= ster/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py
  5. Example of CI of somet= hing like Tianocore:
    1. Windows - Mu_Basecore:=
    2. Linux - Mu_Basecore: <= /span>ht= tps://dev.azure.com/projectmu/mu/_build?definitionId=3D19&view=3Dbuilds= History
    3. Windows - Project Mu Plu= s code: https://dev.azure.com/projectmu/mu/_build?definitionId=3D6&view= =3DbuildsHistory

=C2=A0

T= imeline

  1. RFC open for comment thru 5/21/2019 since the same as RFC Edk2-Library
--4jyowAVbcqdAKr7eRo8t--