RFC  Edk2-ToolEnv creation

Create a new tianocore owned repository to host python code to support an extensible, pluggable, rich environment.  This environment has command line interfaces to support building a product, building CI, running tests, and downloading dependencies. This environment also provides the building blocks for developers to write their own tools to launch in the environment and leverage the capabilities provided by the environment. The unique capabilities provided help support building products with multiple repositories and having each repository contribute/plugin to the build process in a scalable way. The environment will scan the files in the code tree (multiple repos) and discover plugins, dependencies, path adjustments, environment variable settings, etc. This provides easy methods for common repos to share build tools/steps.

Inclusion of this package and dependency management should be managed using Pip/Pypi.   To start this is a supplemental package and is not required to be used for edk2 builds.

More details below but much of this content is coming from combining two existing repos

* https://github.com/microsoft/mu_pip_environment ( https://github.com/microsoft/mu_pip_environment )
* https://github.com/microsoft/mu_pip_build ( https://github.com/microsoft/mu_pip_build )

This RFC is part 2 of content from design review: https://edk2.groups.io/g/devel/files/Designs/2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%20.pdf ( https://edk2.groups.io/g/devel/files/Designs/2019/0418/2019-04-18%20Microsoft%20-%20Build%20Tools%20-%20Design%20Review%20.pdf )

Examples of content here

* CI build support - https://github.com/microsoft/mu_pip_build/blob/master/MuBuild/MuBuild.py ( https://github.com/microsoft/mu_pip_build/blob/master/MuBuild/MuBuild.py )
* Binary Dependency resolution
* Nuget - https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/ExternalDependencies.py ( https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/ExternalDependencies.py )
* GitHub Releases - WIP
* URL dependency - WIP
* Logging
* File logger
* Markdown logger
* In Memory loggers - For tool parsing
* Console loggers with colors
* PlugIns
* Support pre/post build steps
* Support supplying functions
* UefiBuild - A wrapper around Edk2 build process that a Platform would subclass. https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/UefiBuild.py ( https://github.com/microsoft/mu_pip_environment/blob/master/MuEnvironment/UefiBuild.py )
* VarDict and ShellEnvironment - Manage the shell environment and all the name/value pairs used in the build process (including pre/post).
* Omnicache - Support a super cache of git repos to speed up creating and updating multiple work spaces and minimizing filesystem impact

Maintainers

* Sean Brogan
* Bret Barkelew
* Placeholder for existing maintainer from the basetools

License

* BSD + Patent (edk2 aligned)

Contribution process and issue tracking

* Follow Github PR process for contributions and issue tracking
* Contributor forks repo in github
* Contributor creates branch for work
* Contributor updates release notes to indicate change (if necessary)
* Contributor submits PR to master branch of tianocore/Edk2-ToolEnv repo
* Review feedback is given in PR
* Python Tests are run on the repo (new contributions need unit tests)
* Python Style (flake8) must pass
* All review feedback must be completed, maintainers approved, and tests run successfully before PR is *squash merged* into master

Documentation

* Use Github IO documentation/wiki hosting
* Example content

i. https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/ ( https://microsoft.github.io/mu/dyn/mu_pip_environment/developing/ )

ii. https://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/ ( https://microsoft.github.io/mu/dyn/mu_pip_environment/publishing/ )

* Readme at root of repo
* Example: https://github.com/Microsoft/mu_pip_environment ( https://github.com/Microsoft/mu_pip_environment )

CI Builds

* CI build process using dev ops
* Validation is done thru build process
* Release publication done thru manual CI Build
* Examples from Mu-Environment
* Windows CI - https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=10 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=10 )
* Linux CI - https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=11 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=11 )
* Publishing - https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=17 ( https://dev.azure.com/projectmu/mu%20pip/_build?definitionId=17 )

Release

* Release to Pypi as Edk2-ToolEnv for easy usage in product environment
* Versioned follows: Aa.bb.cc.dd
* AA == Major version.  Changes don’t need to be backward compatible
* BB == Minor version.  Significant new features.  Backward compatibility maintained
* CC == Bug fix/patch/small optional feature
* DD == build/Release version.
* Package on Pypi will be owned by Tianocore group
* Example for mu-environment: https://pypi.org/project/mu-environment/ ( https://pypi.org/project/mu-environment/ )

Other Notes

* Only support Python 3 (prefer 3.7+)
* This was discussed on the edk2 design meetings (4/18) https://edk2.groups.io/g/devel/files/Designs/2019/0418 ( https://edk2.groups.io/g/devel/files/Designs/2019/0418 )
* There is RFC #2. The 1 st RFC is for Edk2-Library which this RFC depends on.
* Simple example of usage on an open platform: https://github.com/ms-iot/MU_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py ( https://github.com/ms-iot/MU_PLATFORM_NXP/blob/master/NXP/MCIMX8M_MINI_EVK_2GB/PlatformBuildWorker.py )
* Example of CI of something like Tianocore:
* Windows - Mu_Basecore: https://dev.azure.com/projectmu/mu/_build?definitionId=4&view=buildsHistory ( https://dev.azure.com/projectmu/mu/_build?definitionId=4&view=buildsHistory )
* Linux - Mu_Basecore: https://dev.azure.com/projectmu/mu/_build?definitionId=19&view=buildsHistory ( https://dev.azure.com/projectmu/mu/_build?definitionId=19&view=buildsHistory )
* Windows - Project Mu Plus code: https://dev.azure.com/projectmu/mu/_build?definitionId=6&view=buildsHistory ( https://dev.azure.com/projectmu/mu/_build?definitionId=6&view=buildsHistory )

Timeline

* RFC open for comment thru 5/21/2019 since the same as RFC Edk2-Library