[RFC] EDK II Continuous Integration Phase 1

[RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Hello,

This is a proposal for a first step towards continuous
integration for all TianoCore repositories to help 
improve to quality of commits and automate testing and
release processes for all EDK II packages and platforms.

This is based on work from a number of EDK II community
members that have provide valuable input and evaluations.

* Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
* Laszlo Ersek <lersek@redhat.com> GitLab evaluation
* Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
* Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
* Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
* Jiewen Yao <jiewen.yao@intel.com> HBFA

The following link is a link to an EDK II WIKI page that
contains a summary of the work to date.  Please provide
feedback in the EDK II mailing lists.  The WIKI pages will
be updated with input from the entire EDK II community.

    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration

Proposal
========
Phase 1 of adding continuous integration is limited to the
edk2 repository.  Additional repositories will be added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2 repository.
  Only EDK II Administrators will continue to have write 
  access, and that should only be used to handle extraordinary
  events.
* EDK II Maintainers use a GitHub Pull Request instead of push
  to commit a patch series to the edk2 repository.  There are
  no other changes to the development and review process.  The
  patch series is prepared in an EDK II maintainer branch with
  all commit message requirements met on each patch in the series.
* The edk2 repository only accepts Pull Requests from members
  of the EDK II Maintainers team.  Pull Requests from anyone else
  are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series is auto
  committed.  The result of this commit must match the contents
  and commit history that would have occurred using the previous
  push operation.
* If any pre-commit checks fail, then notify the submitter.
  A typical reason for a failure would be a merge conflict with
  another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that to allow
  EDK II Maintainers to submit a branch through for the same
  set of pre-commit checks without submitting a pull request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check ideas
that could be quickly added once the initial version using
PatchCheck.py is fully functional.  Please provide feedback
on the ones you like and additional ones you think may 
improve the quality of the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
  correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against modified
  modules/libs

Best regards,

Mike

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael Zimmermann]

Hi Michael,

would it make sense to run SCT (using UnixHost and/or qemu) to verify
the high level logic or do you think that would be too much to do for
each PR?

Also, do we want to run all these checks for each commit in the PR to
verify that they pass at each point in the git history?

Thanks
Michael

On Thu, Aug 29, 2019 at 10:22 PM Michael D Kinney
<michael.d.kinney@intel.com> wrote:
>
> Hello,
>
> This is a proposal for a first step towards continuous
> integration for all TianoCore repositories to help
> improve to quality of commits and automate testing and
> release processes for all EDK II packages and platforms.
>
> This is based on work from a number of EDK II community
> members that have provide valuable input and evaluations.
>
> * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
> * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
> * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
> * Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
> * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
> * Jiewen Yao <jiewen.yao@intel.com> HBFA
>
> The following link is a link to an EDK II WIKI page that
> contains a summary of the work to date.  Please provide
> feedback in the EDK II mailing lists.  The WIKI pages will
> be updated with input from the entire EDK II community.
>
>     https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration
>
> Proposal
> ========
> Phase 1 of adding continuous integration is limited to the
> edk2 repository.  Additional repositories will be added later.
>
> The following changes are proposed:
> * Remove EDK II Maintainers write access to edk2 repository.
>   Only EDK II Administrators will continue to have write
>   access, and that should only be used to handle extraordinary
>   events.
> * EDK II Maintainers use a GitHub Pull Request instead of push
>   to commit a patch series to the edk2 repository.  There are
>   no other changes to the development and review process.  The
>   patch series is prepared in an EDK II maintainer branch with
>   all commit message requirements met on each patch in the series.
> * The edk2 repository only accepts Pull Requests from members
>   of the EDK II Maintainers team.  Pull Requests from anyone else
>   are rejected.
> * Run pre-commit checks using Azure Pipelines
> * If all pre-commit checks pass, then the patch series is auto
>   committed.  The result of this commit must match the contents
>   and commit history that would have occurred using the previous
>   push operation.
> * If any pre-commit checks fail, then notify the submitter.
>   A typical reason for a failure would be a merge conflict with
>   another pull request that was just processed.
> * Limit pre-commit checks execution time to 10 minutes.
> * Provide on-demand builds to EDK II Maintainers that to allow
>   EDK II Maintainers to submit a branch through for the same
>   set of pre-commit checks without submitting a pull request.
>
> ## Pre-Commit Checks in Phase 1
> * Run and pass PatchCheck.py with no errors
>
> =====================================================
>
> The following are some additional pre-commit check ideas
> that could be quickly added once the initial version using
> PatchCheck.py is fully functional.  Please provide feedback
> on the ones you like and additional ones you think may
> improve the quality of the commits to the edk2 repository.
>
> ## Proposed Pre-Commit Checks in Phase 2
> * Verify Reviewed-by and Acked-by tags are present with
>   correct maintainer email addresses
> * Verify no non-ASCII characters in modified files
> * Verify no binary files in set of modified files
> * Verify package dependency rules in modified files
>
> ## Proposed Pre-Commit Checks in Phase 3
> * Run ECC on modified files
> * Verify modified modules/libs build
> * Run available host based tests (HBFA) against modified
>   modules/libs
>
> Best regards,
>
> Mike
>
>
>
>
>
>
> 
>

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Hi Michael,

SCTs are in scope.  It is only deciding when they get run
and how much pre-commit execution time developers are
willing to wait for a pass/fail result.

The on-demand testing feature is one place we can add extra
testing to help improve the quality of commits.  The
Daily and Weekly scoped tests are another option.

Mike

> -----Original Message-----
> From: Michael Zimmermann <sigmaepsilon92@gmail.com>
> Sent: Thursday, August 29, 2019 1:40 PM
> To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Cc: rfc@edk2.groups.io
> Subject: Re: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> 
> Hi Michael,
> 
> would it make sense to run SCT (using UnixHost and/or
> qemu) to verify the high level logic or do you think
> that would be too much to do for each PR?
> 
> Also, do we want to run all these checks for each
> commit in the PR to verify that they pass at each point
> in the git history?
> 
> Thanks
> Michael
> 
> On Thu, Aug 29, 2019 at 10:22 PM Michael D Kinney
> <michael.d.kinney@intel.com> wrote:
> >
> > Hello,
> >
> > This is a proposal for a first step towards
> continuous integration for
> > all TianoCore repositories to help improve to quality
> of commits and
> > automate testing and release processes for all EDK II
> packages and
> > platforms.
> >
> > This is based on work from a number of EDK II
> community members that
> > have provide valuable input and evaluations.
> >
> > * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
> > * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
> > * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab
> evaluation
> > * Sean Brogan <sean.brogan@microsoft.com> Azure
> Pipelines and HBFA
> > * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure
> Pipelines and HBFA
> > * Jiewen Yao <jiewen.yao@intel.com> HBFA
> >
> > The following link is a link to an EDK II WIKI page
> that contains a
> > summary of the work to date.  Please provide feedback
> in the EDK II
> > mailing lists.  The WIKI pages will be updated with
> input from the
> > entire EDK II community.
> >
> >
> >
> https://github.com/tianocore/tianocore.github.io/wiki/E
> DK-II-Continuou
> > s-Integration
> >
> > Proposal
> > ========
> > Phase 1 of adding continuous integration is limited
> to the
> > edk2 repository.  Additional repositories will be
> added later.
> >
> > The following changes are proposed:
> > * Remove EDK II Maintainers write access to edk2
> repository.
> >   Only EDK II Administrators will continue to have
> write
> >   access, and that should only be used to handle
> extraordinary
> >   events.
> > * EDK II Maintainers use a GitHub Pull Request
> instead of push
> >   to commit a patch series to the edk2 repository.
> There are
> >   no other changes to the development and review
> process.  The
> >   patch series is prepared in an EDK II maintainer
> branch with
> >   all commit message requirements met on each patch
> in the series.
> > * The edk2 repository only accepts Pull Requests from
> members
> >   of the EDK II Maintainers team.  Pull Requests from
> anyone else
> >   are rejected.
> > * Run pre-commit checks using Azure Pipelines
> > * If all pre-commit checks pass, then the patch
> series is auto
> >   committed.  The result of this commit must match
> the contents
> >   and commit history that would have occurred using
> the previous
> >   push operation.
> > * If any pre-commit checks fail, then notify the
> submitter.
> >   A typical reason for a failure would be a merge
> conflict with
> >   another pull request that was just processed.
> > * Limit pre-commit checks execution time to 10
> minutes.
> > * Provide on-demand builds to EDK II Maintainers that
> to allow
> >   EDK II Maintainers to submit a branch through for
> the same
> >   set of pre-commit checks without submitting a pull
> request.
> >
> > ## Pre-Commit Checks in Phase 1
> > * Run and pass PatchCheck.py with no errors
> >
> > =====================================================
> >
> > The following are some additional pre-commit check
> ideas that could be
> > quickly added once the initial version using
> PatchCheck.py is fully
> > functional.  Please provide feedback on the ones you
> like and
> > additional ones you think may improve the quality of
> the commits to
> > the edk2 repository.
> >
> > ## Proposed Pre-Commit Checks in Phase 2
> > * Verify Reviewed-by and Acked-by tags are present
> with
> >   correct maintainer email addresses
> > * Verify no non-ASCII characters in modified files
> > * Verify no binary files in set of modified files
> > * Verify package dependency rules in modified files
> >
> > ## Proposed Pre-Commit Checks in Phase 3
> > * Run ECC on modified files
> > * Verify modified modules/libs build
> > * Run available host based tests (HBFA) against
> modified
> >   modules/libs
> >
> > Best regards,
> >
> > Mike
> >
> >
> >
> >
> >
> >
> > 
> >

Re: [RFC] EDK II Continuous Integration Phase 1

[Sean]

Mike, as you mentioned we have been working towards enabling a practical and extensible CI for Edk2 using Azure dev ops and the recently added edk2-pytool infrastructure.  We have been using similar CI for Project Mu for the last few years.

Our approach is a little different in that we focus on validating the whole code base rather than just the incoming patch.  We do this because we have found unexpected consequences of patches and overall we want all code to be compliant not just new additions.  We have found the time to test the whole tree is not much longer than only the parts impacted by a code change (except maybe when running the entire compile test on every package).  This obviously comes with an initial tax of needing to get the codebase into compliant form.  Anyway we have prepared an RFC in addition to yours and would like to see these two efforts merged together.

We are still working on making a few optimizations.  Currently if the full set of tests are run we take about 20 minutes.  This is because compiling MdeModulePkg for debug, release, and host based tests take a while.  Most other packages are in the 10 minute range.  We do have easy ways to disable or limit certain tests as well as expand the matrix to leverage more cloud resources (more parallel builds).   


Content is best viewed online with links to helpful content but is also attached below:     
https://github.com/spbrogan/edk2-staging/blob/edk2-stuart-ci-latest/Readme-CI-RFC.md 

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and stewards have done a fantastic job of keeping contribution policies consistent and contributions clean and well-documented, there have been few processes that ran to verify the sanity, cleanliness, and efficacy of the codebase, and even fewer that publicly published their results for the community at large. This has caused inconsistancies and issues within the codebase from time to time.

Adding continuous integration (and potentially PR gates) to the checkin process ensures that simple errors like these are caught and can be fixed on a regular basis.

## Strategy

While a number of CI solutions exist, this proposal will focus on the usage of Azure Dev Ops and Build Pipelines. For demonstration, a sample [TianoCore repo](https://github.com/spbrogan/edk2-staging.git) (branch edk2-stuart-ci-latest) and [Dev Ops Pipeline](https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore python tools PIP modules: [library](https://pypi.org/project/edk2-pytool-library/) and [extensions](https://pypi.org/project/edk2-pytool-extensions/) (with repos located [here](https://github.com/tianocore/edk2-pytool-library) and [here](https://github.com/tianocore/edk2-pytool-extensions)).

The primary execution flows can be found in the `azure-pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-linux.yml` files. These YAML files are consumed by the Azure Dev Ops Build Pipeline and dictate what server resources should be used, how they should be configured, and what processes should be run on them. An overview of this schema can be found [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema).

Inspection of these files reveals the EDKII Tools commands that make up the primary processes for the CI build: 'stuart_setup', 'stuart_update', and 'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and are configured as described below. More documentation on the stuart tools can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/using.md) and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_invocables.md).

## Configuration

Configuration of the CI process consists of (in order of precedence):
* command-line arguments passed in via the Pipeline YAML
* a per-package configuration file (e.g. `<package-name>.mu.yaml`) that is detected by the CI system in EDKII Tools.
* a global configuration Python module (e.g. `CISetting.py`) passed in via the command-line

The global configuration file is described in [this readme](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_settings_manager.md) from the EDKII Tools documentation. This configuration is written as a Python module so that decisions can be made dynamically based on command line parameters and codebase state.

The per-package configuration file can override most settings in the global configuration file, but is not dynamic. This file can be used to skip or customize tests that may be incompatible with a specific package. By default, the global configuration will try to run all tests on all packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins. Documentation on the plugin system can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_plugin_manager.md) and [here](https://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/feature_plugin_manager.md). Upon invocation, each plugin will be passed the path to the current package under test and a dictionary containing its targeted configuration, as assembled from the command line, per-package configuration, and global configuration.

Note: CI plugins are considered unique from build plugins and helper plugins, even though some CI plugins may execute steps of a build.

In the example, these plugins live alongside the code under test (in the `BaseTools` directory), but may be moved to the 'edk2-test' repo if that location makes more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files) and compares them to the package-level DSC file for the package each module is contained within. The test considers it an error if any module does not appear in the `Components` section of at least one package-level DSC (indicating that it would not be built if the package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all modules would be built if all package-level DSCs were built, the Code Compilation Test simply runs through and builds every package-level DSC on every toolchain and for every architecture that is supported. Any module that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin and HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much more detail on this, but the basic idea is that host-based unit tests can be compiled against individual modules and libraries and run on the build agent (in this case, the Dev Ops build server). The successful and failing test case results are collected and included in the final build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages rather than an individual package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and ensures that they are unique for the codebase. This prevents, for example, accidental duplication of GUIDs when using an existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in INFs files for a given package against a list of "allowed dependencies" in plugin configuration for that package. Any module that depends on a disallowed package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a package's `Include/Library` directory and ensures that all files have a matching LibraryClass declaration in the DEC file for the package. Any missing declarations will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure that there are no invalid Unicode characters that may cause build errors in some character sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing RFCs from other TianoCore contributors.
* Optimize testing beharior.
  * Only run a subset of tests on PRs or individual commits.
  * Run full testing either once per day or once every several commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
  * Much of this documentation effort is pending community feedback on which parts are needed and what phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Michael D Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1

Hello,

This is a proposal for a first step towards continuous integration for all TianoCore repositories to help improve to quality of commits and automate testing and release processes for all EDK II packages and platforms.

This is based on work from a number of EDK II community members that have provide valuable input and evaluations.

* Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins evaluation
* Laszlo Ersek <mailto:lersek@redhat.com> GitLab evaluation
* Philippe Mathieu-Daudé <mailto:philmd@redhat.com> GitLab evaluation
* Sean Brogan <mailto:sean.brogan@microsoft.com> Azure Pipelines and HBFA
* Bret Barkelew <mailto:Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA

The following link is a link to an EDK II WIKI page that contains a summary of the work to date.  Please provide feedback in the EDK II mailing lists.  The WIKI pages will be updated with input from the entire EDK II community.

    https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Continuous-Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sdata=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to the
edk2 repository.  Additional repositories will be added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2 repository.
  Only EDK II Administrators will continue to have write
  access, and that should only be used to handle extraordinary
  events.
* EDK II Maintainers use a GitHub Pull Request instead of push
  to commit a patch series to the edk2 repository.  There are
  no other changes to the development and review process.  The
  patch series is prepared in an EDK II maintainer branch with
  all commit message requirements met on each patch in the series.
* The edk2 repository only accepts Pull Requests from members
  of the EDK II Maintainers team.  Pull Requests from anyone else
  are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series is auto
  committed.  The result of this commit must match the contents
  and commit history that would have occurred using the previous
  push operation.
* If any pre-commit checks fail, then notify the submitter.
  A typical reason for a failure would be a merge conflict with
  another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that to allow
  EDK II Maintainers to submit a branch through for the same
  set of pre-commit checks without submitting a pull request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check ideas that could be quickly added once the initial version using PatchCheck.py is fully functional.  Please provide feedback on the ones you like and additional ones you think may improve the quality of the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
  correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against modified
  modules/libs

Best regards,

Mike

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

Hi Sean,

On 08/30/19 04:21, Sean via Groups.Io wrote:
> would like to see these two efforts merged together.

> Currently if the full set of tests are run we take about 20 minutes.
> This is because compiling MdeModulePkg for debug, release, and host
> based tests take a while.  Most other packages are in the 10 minute
> range.  We do have easy ways to disable or limit certain tests as well
> as expand the matrix to leverage more cloud resources (more parallel
> builds).

> ### Module Inclusion Test - DscCompleteCheck
> ### Code Compilation Test - CompilerPlugin
> ### Host-Based UnitTests - HostUnitTestCompilerPlugin and
>     HostUnitTestDscCompleteCheck
> ### GUID Uniqueness Test - GuidCheck
> ### Cross-Package Dependency Test - DependencyCheck
> ### Library Declaration Test - LibraryClassCheck
> ### Invalid Character Test - CharEncodingCheck

These tests sound awesome!

> ## Next Steps
>
> * Receive community feedback on RFC.
> * Determine where this phase makes sense given existing RFCs from
>   other TianoCore contributors.
> * Optimize testing beharior.
>   * Only run a subset of tests on PRs or individual commits.
>   * Run full testing either once per day or once every several
      commits.

I'd like to keep the per-PR tests down to 10 minutes.

On the other hand, it would be great if all of these tests could be
performed daily or weekly.

Are these tests easy to integrate with the infrastructure described by
Mike?

Thanks,
Laszlo

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Sean]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

These tests require using the "edk2-pytool" stuff but are easy to integrate with the github PR or CI flow.  Example of it running the code compliance tests is here: https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=13&_a=summary A test run takes about 3 minutes.

The past week or so we also have been working on a more complete test run which adds to the above test suite three more tests.  Compile each package for Debug, Release, and Host Based Unit tests.  Obviously this adds more time and takes more resources but at the moment we have it down to 13 minutes. You can see it here: https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=12&_a=summary

To enable this framework you need the following.

1. pip install a couple of edk2-pytool packages (see requirements.txt file)
2. Add the pytool based plugins to the edk2 code tree  See here: https://github.com/spbrogan/edk2-staging/tree/edk2-stuart-ci-latest/BaseTools/Plugin/Ci
3. Add a CiSettings.py file that satisfies the setup, update, and ci_build  See here: https://github.com/spbrogan/edk2-staging/blob/edk2-stuart-ci-latest/CISettings.py
4. Add a *.ci.yaml file to each package to configure test settings, ignore files, and other configuration for testing the package.  Example here: https://github.com/spbrogan/edk2-staging/blob/edk2-stuart-ci-latest/MdeModulePkg/MdeModulePkg.ci.yaml
5.  For Azure pipeline support (cloud based CI) you will need to add Azure pipeline.yaml flles.

This can all be seen in the branch here. https://github.com/spbrogan/edk2-staging/tree/edk2-stuart-ci-latest
Be aware that in that same branch we are enabling "Host based unit tests" which also requires more code changes so if diffing with clean edk2 you can ignore those (although that is all part of another RFC for unit test so any feedback on that would be great too).

Feedback would be appreciated.  I am hoping this RFC can move forward in the next few weeks.

Thanks
Sean

[-- Attachment #2: Type: text/html, Size: 2801 bytes --]

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

Hi Sean,

On 09/13/19 23:00, sean.brogan via [] wrote:
> These tests require using the "edk2-pytool" stuff but are easy to integrate with the github PR or CI flow.  Example of it running the code compliance tests is here: https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=13&_a=summary A test run takes about 3 minutes.
> 
> The past week or so we also have been working on a more complete test run which adds to the above test suite three more tests.  Compile each package for Debug, Release, and Host Based Unit tests.  Obviously this adds more time and takes more resources but at the moment we have it down to 13 minutes. You can see it here: https://dev.azure.com/tianocore/edk2-ci-play/_build?definitionId=12&_a=summary
> 
> To enable this framework you need the following.
> 
> 1. pip install a couple of edk2-pytool packages (see requirements.txt file)
> 2. Add the pytool based plugins to the edk2 code tree  See here: https://github.com/spbrogan/edk2-staging/tree/edk2-stuart-ci-latest/BaseTools/Plugin/Ci
> 3. Add a CiSettings.py file that satisfies the setup, update, and ci_build  See here: https://github.com/spbrogan/edk2-staging/blob/edk2-stuart-ci-latest/CISettings.py
> 4. Add a *.ci.yaml file to each package to configure test settings, ignore files, and other configuration for testing the package.  Example here: https://github.com/spbrogan/edk2-staging/blob/edk2-stuart-ci-latest/MdeModulePkg/MdeModulePkg.ci.yaml
> 5.  For Azure pipeline support (cloud based CI) you will need to add Azure pipeline.yaml flles.
> 
> This can all be seen in the branch here. https://github.com/spbrogan/edk2-staging/tree/edk2-stuart-ci-latest
> Be aware that in that same branch we are enabling "Host based unit tests" which also requires more code changes so if diffing with clean edk2 you can ignore those (although that is all part of another RFC for unit test so any feedback on that would be great too).
> 
> Feedback would be appreciated.  I am hoping this RFC can move forward in the next few weeks.

I'm really sorry, I don't have any cycles left for this topic at the
moment. I hope Mike and others can proceed with you.

(I prefer to share whenever I'm out of time for a topic; I find that a
lot better than unjustifiedly blocking others from doing their work.)

Thanks & sorry again
Laszlo

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Hi Sean,

This looks really good and I agree we can combine the 
RFCs and enable more pre-commits tests.

Additional responses below.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Sean via Groups.Io
> Sent: Thursday, August 29, 2019 7:22 PM
> To: rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> 
> Mike, as you mentioned we have been working towards
> enabling a practical and extensible CI for Edk2 using
> Azure dev ops and the recently added edk2-pytool
> infrastructure.  We have been using similar CI for
> Project Mu for the last few years.
> 
> Our approach is a little different in that we focus on
> validating the whole code base rather than just the
> incoming patch.  We do this because we have found
> unexpected consequences of patches and overall we want
> all code to be compliant not just new additions.  We
> have found the time to test the whole tree is not much
> longer than only the parts impacted by a code change
> (except maybe when running the entire compile test on
> every package).  This obviously comes with an initial
> tax of needing to get the codebase into compliant form.
> Anyway we have prepared an RFC in addition to yours and
> would like to see these two efforts merged together.

I am in favor of complete code base testing as long as the
time to complete the tests is reasonable.  We can do both
patch level testing to make sure the patches are well
formatted.  Focused testing on the components modified by
the patches and some amount of complete testing to find any
unexpected side effects.

> 
> We are still working on making a few optimizations.
> Currently if the full set of tests are run we take
> about 20 minutes.  This is because compiling
> MdeModulePkg for debug, release, and host based tests
> take a while.  Most other packages are in the 10 minute
> range.  We do have easy ways to disable or limit
> certain tests as well as expand the matrix to leverage
> more cloud resources (more parallel builds).
> 
> 
> Content is best viewed online with links to helpful
> content but is also attached below:
> https://github.com/spbrogan/edk2-staging/blob/edk2-
> stuart-ci-latest/Readme-CI-RFC.md
> 
> # CI and PR Gates
> 
> ## Background
> 
> Historically, while the TianoCore maintainers and
> stewards have done a fantastic job of keeping
> contribution policies consistent and contributions
> clean and well-documented, there have been few
> processes that ran to verify the sanity, cleanliness,
> and efficacy of the codebase, and even fewer that
> publicly published their results for the community at
> large. This has caused inconsistancies and issues
> within the codebase from time to time.
> 
> Adding continuous integration (and potentially PR
> gates) to the checkin process ensures that simple
> errors like these are caught and can be fixed on a
> regular basis.
> 
> ## Strategy
> 
> While a number of CI solutions exist, this proposal
> will focus on the usage of Azure Dev Ops and Build
> Pipelines. For demonstration, a sample [TianoCore
> repo](https://github.com/spbrogan/edk2-staging.git)
> (branch edk2-stuart-ci-latest) and [Dev Ops
> Pipeline](https://dev.azure.com/tianocore/edk2-ci-
> play/_build?definitionId=12) have been set up.
> 
> Furthermore, this proposal will leverage the TianoCore
> python tools PIP modules:
> [library](https://pypi.org/project/edk2-pytool-
> library/) and
> [extensions](https://pypi.org/project/edk2-pytool-
> extensions/) (with repos located
> [here](https://github.com/tianocore/edk2-pytool-
> library) and [here](https://github.com/tianocore/edk2-
> pytool-extensions)).
> 
> The primary execution flows can be found in the `azure-
> pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
> linux.yml` files. These YAML files are consumed by the
> Azure Dev Ops Build Pipeline and dictate what server
> resources should be used, how they should be
> configured, and what processes should be run on them.
> An overview of this schema can be found
> [here](https://docs.microsoft.com/en-
> us/azure/devops/pipelines/yaml-schema?view=azure-
> devops&tabs=schema).
> 
> Inspection of these files reveals the EDKII Tools
> commands that make up the primary processes for the CI
> build: 'stuart_setup', 'stuart_update', and
> 'stuart_ci_build'. These commands come from the EDKII
> Tools PIP modules and are configured as described
> below. More documentation on the stuart tools can be
> found [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/using.md) and
> [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/features/feature_invocables
> .md).
> 
> ## Configuration
> 
> Configuration of the CI process consists of (in order
> of precedence):
> * command-line arguments passed in via the Pipeline
> YAML
> * a per-package configuration file (e.g. `<package-
> name>.mu.yaml`) that is detected by the CI system in
> EDKII Tools.
> * a global configuration Python module (e.g.
> `CISetting.py`) passed in via the command-line
> 
> The global configuration file is described in [this
> readme](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/usability/using_settings_ma
> nager.md) from the EDKII Tools documentation. This
> configuration is written as a Python module so that
> decisions can be made dynamically based on command line
> parameters and codebase state.
> 
> The per-package configuration file can override most
> settings in the global configuration file, but is not
> dynamic. This file can be used to skip or customize
> tests that may be incompatible with a specific package.
> By default, the global configuration will try to run
> all tests on all packages.
> 
> ## CI Test Types
> 
> All CI tests are instances of EDKII Tools plugins.
> Documentation on the plugin system can be found
> [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/usability/using_plugin_mana
> ger.md) and [here](https://github.com/tianocore/edk2-
> pytool-
> extensions/blob/master/docs/features/feature_plugin_man
> ager.md). Upon invocation, each plugin will be passed
> the path to the current package under test and a
> dictionary containing its targeted configuration, as
> assembled from the command line, per-package
> configuration, and global configuration.
> 
> Note: CI plugins are considered unique from build
> plugins and helper plugins, even though some CI plugins
> may execute steps of a build.
> 
> In the example, these plugins live alongside the code
> under test (in the `BaseTools` directory), but may be
> moved to the 'edk2-test' repo if that location makes
> more sense for the community.
> 

Which of the tests below are passing for all edk2 packages?
Leif has been working on fixing some of the non-ASCII files
and line ending issues.  I have been working on some of the
UNI file format issues.  If there is a small set of tasks that
would allow more of these tests to be enabled quickly, then
let's get those entered into TianoCore Bugzilla.

> ### Module Inclusion Test - DscCompleteCheck
> 
> This test scans all available modules (via INF files)
> and compares them to the package-level DSC file for the
> package each module is contained within. The test
> considers it an error if any module does not appear in
> the `Components` section of at least one package-level
> DSC (indicating that it would not be built if the
> package were built).
> 
> ### Code Compilation Test - CompilerPlugin
> 
> Once the Module Inclusion Test has verified that all
> modules would be built if all package-level DSCs were
> built, the Code Compilation Test simply runs through
> and builds every package-level DSC on every toolchain
> and for every architecture that is supported. Any
> module that fails to build is considered an error.
> 
> ### Host-Based UnitTests - HostUnitTestCompilerPlugin
> and HostUnitTestDscCompleteCheck
> 
> The [Testing RFC doc](Readme-Testing-RFC.md) has much
> more detail on this, but the basic idea is that host-
> based unit tests can be compiled against individual
> modules and libraries and run on the build agent (in
> this case, the Dev Ops build server). The successful
> and failing test case results are collected and
> included in the final build report.
> 
> ### GUID Uniqueness Test - GuidCheck
> 
> This test works on the collection of all packages
> rather than an individual package. It looks at all
> FILE_GUIDs and GUIDs declared in DEC files and ensures
> that they are unique for the codebase. This prevents,
> for example, accidental duplication of GUIDs when using
> an existing INF as a template for a new module.
> 
> ### Cross-Package Dependency Test - DependencyCheck
> 
> This test compares the list of all packages used in
> INFs files for a given package against a list of
> "allowed dependencies" in plugin configuration for that
> package. Any module that depends on a disallowed
> package will cause a test failure.
> 
> ### Library Declaration Test - LibraryClassCheck
> 
> This test looks at all library header files found in a
> package's `Include/Library` directory and ensures that
> all files have a matching LibraryClass declaration in
> the DEC file for the package. Any missing declarations
> will cause a failure.
> 
> ### Invalid Character Test - CharEncodingCheck
> 
> This test scans all files in a package to make sure
> that there are no invalid Unicode characters that may
> cause build errors in some character
> sets/localizations.
> 
> ## Next Steps
> 
> * Receive community feedback on RFC.
> * Determine where this phase makes sense given existing
> RFCs from other TianoCore contributors.
> * Optimize testing beharior.
>   * Only run a subset of tests on PRs or individual
> commits.
>   * Run full testing either once per day or once every
> several commits.
> * Add more tests/capabilities.
> * Continue to improve results formatting.
> * Continue to improve CI documentation.
>   * Much of this documentation effort is pending
> community feedback on which parts are needed and what
> phases are priorities.
> 
> Thanks
> 
> 
> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
> Of Michael D Kinney via Groups.Io
> Sent: Thursday, August 29, 2019 1:23 PM
> To: devel@edk2.groups.io; rfc@edk2.groups.io
> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
> Phase 1
> 
> Hello,
> 
> This is a proposal for a first step towards continuous
> integration for all TianoCore repositories to help
> improve to quality of commits and automate testing and
> release processes for all EDK II packages and
> platforms.
> 
> This is based on work from a number of EDK II community
> members that have provide valuable input and
> evaluations.
> 
> * Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
> evaluation
> * Laszlo Ersek <mailto:lersek@redhat.com> GitLab
> evaluation
> * Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
> GitLab evaluation
> * Sean Brogan <mailto:sean.brogan@microsoft.com> Azure
> Pipelines and HBFA
> * Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
> Azure Pipelines and HBFA
> * Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA
> 
> The following link is a link to an EDK II WIKI page
> that contains a summary of the work to date.  Please
> provide feedback in the EDK II mailing lists.  The WIKI
> pages will be updated with input from the entire EDK II
> community.
> 
> 
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-Continuous-
> Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
> com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
> ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
> ;reserved=0
> 
> Proposal
> ========
> Phase 1 of adding continuous integration is limited to
> the
> edk2 repository.  Additional repositories will be added
> later.
> 
> The following changes are proposed:
> * Remove EDK II Maintainers write access to edk2
> repository.
>   Only EDK II Administrators will continue to have
> write
>   access, and that should only be used to handle
> extraordinary
>   events.
> * EDK II Maintainers use a GitHub Pull Request instead
> of push
>   to commit a patch series to the edk2 repository.
> There are
>   no other changes to the development and review
> process.  The
>   patch series is prepared in an EDK II maintainer
> branch with
>   all commit message requirements met on each patch in
> the series.
> * The edk2 repository only accepts Pull Requests from
> members
>   of the EDK II Maintainers team.  Pull Requests from
> anyone else
>   are rejected.
> * Run pre-commit checks using Azure Pipelines
> * If all pre-commit checks pass, then the patch series
> is auto
>   committed.  The result of this commit must match the
> contents
>   and commit history that would have occurred using the
> previous
>   push operation.
> * If any pre-commit checks fail, then notify the
> submitter.
>   A typical reason for a failure would be a merge
> conflict with
>   another pull request that was just processed.
> * Limit pre-commit checks execution time to 10 minutes.
> * Provide on-demand builds to EDK II Maintainers that
> to allow
>   EDK II Maintainers to submit a branch through for the
> same
>   set of pre-commit checks without submitting a pull
> request.
> 
> ## Pre-Commit Checks in Phase 1
> * Run and pass PatchCheck.py with no errors
> 
> =====================================================
> 
> The following are some additional pre-commit check
> ideas that could be quickly added once the initial
> version using PatchCheck.py is fully functional.
> Please provide feedback on the ones you like and
> additional ones you think may improve the quality of
> the commits to the edk2 repository.
> 
> ## Proposed Pre-Commit Checks in Phase 2
> * Verify Reviewed-by and Acked-by tags are present with
>   correct maintainer email addresses
> * Verify no non-ASCII characters in modified files
> * Verify no binary files in set of modified files
> * Verify package dependency rules in modified files
> 
> ## Proposed Pre-Commit Checks in Phase 3
> * Run ECC on modified files
> * Verify modified modules/libs build
> * Run available host based tests (HBFA) against
> modified
>   modules/libs
> 
> Best regards,
> 
> Mike
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines.  There are some tools
that need to be installed for each of these environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed?  Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to use.
These configurations should be documented in a Wiki page
and updated as new tools are released and adopted by EDK II.
The inventory of tools used to validate a release should 
Also be documented in a release notes for a stable tag.

Here are the YML files that install the additional tools 
required to support EDK II builds.  I need the source and
versions of these tools to be reviewed and approved.

https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/WindowsPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/UbuntuPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/MacOsPrerequisites.yml

Thanks,

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Sean via Groups.Io
> Sent: Thursday, August 29, 2019 7:22 PM
> To: rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> 
> Mike, as you mentioned we have been working towards
> enabling a practical and extensible CI for Edk2 using
> Azure dev ops and the recently added edk2-pytool
> infrastructure.  We have been using similar CI for
> Project Mu for the last few years.
> 
> Our approach is a little different in that we focus on
> validating the whole code base rather than just the
> incoming patch.  We do this because we have found
> unexpected consequences of patches and overall we want
> all code to be compliant not just new additions.  We
> have found the time to test the whole tree is not much
> longer than only the parts impacted by a code change
> (except maybe when running the entire compile test on
> every package).  This obviously comes with an initial
> tax of needing to get the codebase into compliant form.
> Anyway we have prepared an RFC in addition to yours and
> would like to see these two efforts merged together.
> 
> We are still working on making a few optimizations.
> Currently if the full set of tests are run we take
> about 20 minutes.  This is because compiling
> MdeModulePkg for debug, release, and host based tests
> take a while.  Most other packages are in the 10 minute
> range.  We do have easy ways to disable or limit
> certain tests as well as expand the matrix to leverage
> more cloud resources (more parallel builds).
> 
> 
> Content is best viewed online with links to helpful
> content but is also attached below:
> https://github.com/spbrogan/edk2-staging/blob/edk2-
> stuart-ci-latest/Readme-CI-RFC.md
> 
> # CI and PR Gates
> 
> ## Background
> 
> Historically, while the TianoCore maintainers and
> stewards have done a fantastic job of keeping
> contribution policies consistent and contributions
> clean and well-documented, there have been few
> processes that ran to verify the sanity, cleanliness,
> and efficacy of the codebase, and even fewer that
> publicly published their results for the community at
> large. This has caused inconsistancies and issues
> within the codebase from time to time.
> 
> Adding continuous integration (and potentially PR
> gates) to the checkin process ensures that simple
> errors like these are caught and can be fixed on a
> regular basis.
> 
> ## Strategy
> 
> While a number of CI solutions exist, this proposal
> will focus on the usage of Azure Dev Ops and Build
> Pipelines. For demonstration, a sample [TianoCore
> repo](https://github.com/spbrogan/edk2-staging.git)
> (branch edk2-stuart-ci-latest) and [Dev Ops
> Pipeline](https://dev.azure.com/tianocore/edk2-ci-
> play/_build?definitionId=12) have been set up.
> 
> Furthermore, this proposal will leverage the TianoCore
> python tools PIP modules:
> [library](https://pypi.org/project/edk2-pytool-
> library/) and
> [extensions](https://pypi.org/project/edk2-pytool-
> extensions/) (with repos located
> [here](https://github.com/tianocore/edk2-pytool-
> library) and [here](https://github.com/tianocore/edk2-
> pytool-extensions)).
> 
> The primary execution flows can be found in the `azure-
> pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
> linux.yml` files. These YAML files are consumed by the
> Azure Dev Ops Build Pipeline and dictate what server
> resources should be used, how they should be
> configured, and what processes should be run on them.
> An overview of this schema can be found
> [here](https://docs.microsoft.com/en-
> us/azure/devops/pipelines/yaml-schema?view=azure-
> devops&tabs=schema).
> 
> Inspection of these files reveals the EDKII Tools
> commands that make up the primary processes for the CI
> build: 'stuart_setup', 'stuart_update', and
> 'stuart_ci_build'. These commands come from the EDKII
> Tools PIP modules and are configured as described
> below. More documentation on the stuart tools can be
> found [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/using.md) and
> [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/features/feature_invocables
> .md).
> 
> ## Configuration
> 
> Configuration of the CI process consists of (in order
> of precedence):
> * command-line arguments passed in via the Pipeline
> YAML
> * a per-package configuration file (e.g. `<package-
> name>.mu.yaml`) that is detected by the CI system in
> EDKII Tools.
> * a global configuration Python module (e.g.
> `CISetting.py`) passed in via the command-line
> 
> The global configuration file is described in [this
> readme](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/usability/using_settings_ma
> nager.md) from the EDKII Tools documentation. This
> configuration is written as a Python module so that
> decisions can be made dynamically based on command line
> parameters and codebase state.
> 
> The per-package configuration file can override most
> settings in the global configuration file, but is not
> dynamic. This file can be used to skip or customize
> tests that may be incompatible with a specific package.
> By default, the global configuration will try to run
> all tests on all packages.
> 
> ## CI Test Types
> 
> All CI tests are instances of EDKII Tools plugins.
> Documentation on the plugin system can be found
> [here](https://github.com/tianocore/edk2-pytool-
> extensions/blob/master/docs/usability/using_plugin_mana
> ger.md) and [here](https://github.com/tianocore/edk2-
> pytool-
> extensions/blob/master/docs/features/feature_plugin_man
> ager.md). Upon invocation, each plugin will be passed
> the path to the current package under test and a
> dictionary containing its targeted configuration, as
> assembled from the command line, per-package
> configuration, and global configuration.
> 
> Note: CI plugins are considered unique from build
> plugins and helper plugins, even though some CI plugins
> may execute steps of a build.
> 
> In the example, these plugins live alongside the code
> under test (in the `BaseTools` directory), but may be
> moved to the 'edk2-test' repo if that location makes
> more sense for the community.
> 
> ### Module Inclusion Test - DscCompleteCheck
> 
> This test scans all available modules (via INF files)
> and compares them to the package-level DSC file for the
> package each module is contained within. The test
> considers it an error if any module does not appear in
> the `Components` section of at least one package-level
> DSC (indicating that it would not be built if the
> package were built).
> 
> ### Code Compilation Test - CompilerPlugin
> 
> Once the Module Inclusion Test has verified that all
> modules would be built if all package-level DSCs were
> built, the Code Compilation Test simply runs through
> and builds every package-level DSC on every toolchain
> and for every architecture that is supported. Any
> module that fails to build is considered an error.
> 
> ### Host-Based UnitTests - HostUnitTestCompilerPlugin
> and HostUnitTestDscCompleteCheck
> 
> The [Testing RFC doc](Readme-Testing-RFC.md) has much
> more detail on this, but the basic idea is that host-
> based unit tests can be compiled against individual
> modules and libraries and run on the build agent (in
> this case, the Dev Ops build server). The successful
> and failing test case results are collected and
> included in the final build report.
> 
> ### GUID Uniqueness Test - GuidCheck
> 
> This test works on the collection of all packages
> rather than an individual package. It looks at all
> FILE_GUIDs and GUIDs declared in DEC files and ensures
> that they are unique for the codebase. This prevents,
> for example, accidental duplication of GUIDs when using
> an existing INF as a template for a new module.
> 
> ### Cross-Package Dependency Test - DependencyCheck
> 
> This test compares the list of all packages used in
> INFs files for a given package against a list of
> "allowed dependencies" in plugin configuration for that
> package. Any module that depends on a disallowed
> package will cause a test failure.
> 
> ### Library Declaration Test - LibraryClassCheck
> 
> This test looks at all library header files found in a
> package's `Include/Library` directory and ensures that
> all files have a matching LibraryClass declaration in
> the DEC file for the package. Any missing declarations
> will cause a failure.
> 
> ### Invalid Character Test - CharEncodingCheck
> 
> This test scans all files in a package to make sure
> that there are no invalid Unicode characters that may
> cause build errors in some character
> sets/localizations.
> 
> ## Next Steps
> 
> * Receive community feedback on RFC.
> * Determine where this phase makes sense given existing
> RFCs from other TianoCore contributors.
> * Optimize testing beharior.
>   * Only run a subset of tests on PRs or individual
> commits.
>   * Run full testing either once per day or once every
> several commits.
> * Add more tests/capabilities.
> * Continue to improve results formatting.
> * Continue to improve CI documentation.
>   * Much of this documentation effort is pending
> community feedback on which parts are needed and what
> phases are priorities.
> 
> Thanks
> 
> 
> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
> Of Michael D Kinney via Groups.Io
> Sent: Thursday, August 29, 2019 1:23 PM
> To: devel@edk2.groups.io; rfc@edk2.groups.io
> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
> Phase 1
> 
> Hello,
> 
> This is a proposal for a first step towards continuous
> integration for all TianoCore repositories to help
> improve to quality of commits and automate testing and
> release processes for all EDK II packages and
> platforms.
> 
> This is based on work from a number of EDK II community
> members that have provide valuable input and
> evaluations.
> 
> * Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
> evaluation
> * Laszlo Ersek <mailto:lersek@redhat.com> GitLab
> evaluation
> * Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
> GitLab evaluation
> * Sean Brogan <mailto:sean.brogan@microsoft.com> Azure
> Pipelines and HBFA
> * Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
> Azure Pipelines and HBFA
> * Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA
> 
> The following link is a link to an EDK II WIKI page
> that contains a summary of the work to date.  Please
> provide feedback in the EDK II mailing lists.  The WIKI
> pages will be updated with input from the entire EDK II
> community.
> 
> 
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-Continuous-
> Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
> com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
> ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
> ;reserved=0
> 
> Proposal
> ========
> Phase 1 of adding continuous integration is limited to
> the
> edk2 repository.  Additional repositories will be added
> later.
> 
> The following changes are proposed:
> * Remove EDK II Maintainers write access to edk2
> repository.
>   Only EDK II Administrators will continue to have
> write
>   access, and that should only be used to handle
> extraordinary
>   events.
> * EDK II Maintainers use a GitHub Pull Request instead
> of push
>   to commit a patch series to the edk2 repository.
> There are
>   no other changes to the development and review
> process.  The
>   patch series is prepared in an EDK II maintainer
> branch with
>   all commit message requirements met on each patch in
> the series.
> * The edk2 repository only accepts Pull Requests from
> members
>   of the EDK II Maintainers team.  Pull Requests from
> anyone else
>   are rejected.
> * Run pre-commit checks using Azure Pipelines
> * If all pre-commit checks pass, then the patch series
> is auto
>   committed.  The result of this commit must match the
> contents
>   and commit history that would have occurred using the
> previous
>   push operation.
> * If any pre-commit checks fail, then notify the
> submitter.
>   A typical reason for a failure would be a merge
> conflict with
>   another pull request that was just processed.
> * Limit pre-commit checks execution time to 10 minutes.
> * Provide on-demand builds to EDK II Maintainers that
> to allow
>   EDK II Maintainers to submit a branch through for the
> same
>   set of pre-commit checks without submitting a pull
> request.
> 
> ## Pre-Commit Checks in Phase 1
> * Run and pass PatchCheck.py with no errors
> 
> =====================================================
> 
> The following are some additional pre-commit check
> ideas that could be quickly added once the initial
> version using PatchCheck.py is fully functional.
> Please provide feedback on the ones you like and
> additional ones you think may improve the quality of
> the commits to the edk2 repository.
> 
> ## Proposed Pre-Commit Checks in Phase 2
> * Verify Reviewed-by and Acked-by tags are present with
>   correct maintainer email addresses
> * Verify no non-ASCII characters in modified files
> * Verify no binary files in set of modified files
> * Verify package dependency rules in modified files
> 
> ## Proposed Pre-Commit Checks in Phase 3
> * Run ECC on modified files
> * Verify modified modules/libs build
> * Run available host based tests (HBFA) against
> modified
>   modules/libs
> 
> Best regards,
> 
> Mike
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Sean]

Currently it is building on Windows with VS2019.  VS2017 would be trivial but not worth it in my perspective given how aligned the two are.   If you really wanted to do a weekly or nightly build it could be added to that but I have been focused on a PR build.
I have a pipeline for GCC on linux.  It doesn't' support host based unit tests but I am working to get the rest of the tests running.
I do not have a pipeline for Mac or LLVM/Clang.  That would be next and I think your files below should help.

To date we use PYTOOLs features to install our extra tools.  This makes it super easy and works for both local and server based builds.  It gives strong versioning and management of that.  It also tracks those versions and you can see them on every build in the Built Tool Report artifact.   It can be downloaded here for html version. https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=803&view=artifacts.   BUILD_TOOL_REPORT.html

Here it is pasted as plain text below.  You can see the iasl and nasm versions here. 

Key	Value	Type
TOOL_CHAIN_TAG	VS2019	TOOL
VC Version	14.22.27905	TOOL
d:\a\1\s\Conf\build_rule.txt	1.04	INFO
d:\a\1\s\Conf\target.txt	1.03	INFO
d:\a\1\s\Conf\tools_def.txt	1.22	INFO
iasl	20190215.0.0	INFO
Mu-Basetools	2019.03.1	INFO
mu_nasm	2.14.02	INFO


Hope that helps. 

I would be happy to queue up this topic for next design meeting and open it up to questions?  I hope others would take a look prior to meeting.

Thanks
Sean



-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com> 
Sent: Thursday, September 19, 2019 2:56 PM
To: devel@edk2.groups.io; Sean Brogan <sean.brogan@microsoft.com>; rfc@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Hi Sean,

Which OS/Compiler configurations are currently enabled for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler configurations in Azure Pipelines.  There are some tools that need to be installed for each of these environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools installed?  Is it in the YML files or in the Python scripts.

One critical task is to identify the tools and their specific versions that the CI system is configured to use.
These configurations should be documented in a Wiki page and updated as new tools are released and adopted by EDK II.
The inventory of tools used to validate a release should Also be documented in a release notes for a stable tag.

Here are the YML files that install the additional tools required to support EDK II builds.  I need the source and versions of these tools to be reviewed and approved.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisites.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=hM4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisites.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisites.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzAmzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;reserved=0

Thanks,

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean 
> via Groups.Io
> Sent: Thursday, August 29, 2019 7:22 PM
> To: rfc@edk2.groups.io; Kinney, Michael D 
> <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
> 
> Mike, as you mentioned we have been working towards enabling a 
> practical and extensible CI for Edk2 using Azure dev ops and the 
> recently added edk2-pytool infrastructure.  We have been using similar 
> CI for Project Mu for the last few years.
> 
> Our approach is a little different in that we focus on validating the 
> whole code base rather than just the incoming patch.  We do this 
> because we have found unexpected consequences of patches and overall 
> we want all code to be compliant not just new additions.  We have 
> found the time to test the whole tree is not much longer than only the 
> parts impacted by a code change (except maybe when running the entire 
> compile test on every package).  This obviously comes with an initial 
> tax of needing to get the codebase into compliant form.
> Anyway we have prepared an RFC in addition to yours and would like to 
> see these two efforts merged together.
> 
> We are still working on making a few optimizations.
> Currently if the full set of tests are run we take about 20 minutes.
> This is because compiling MdeModulePkg for debug, release, and host 
> based tests take a while.  Most other packages are in the 10 minute 
> range.  We do have easy ways to disable or limit certain tests as well 
> as expand the matrix to leverage more cloud resources (more parallel 
> builds).
> 
> 
> Content is best viewed online with links to helpful content but is 
> also attached below:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-&amp;data=02%7C01%7Csea
> n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=xOiPHH
> VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserved=0
> stuart-ci-latest/Readme-CI-RFC.md
> 
> # CI and PR Gates
> 
> ## Background
> 
> Historically, while the TianoCore maintainers and stewards have done a 
> fantastic job of keeping contribution policies consistent and 
> contributions clean and well-documented, there have been few processes 
> that ran to verify the sanity, cleanliness, and efficacy of the 
> codebase, and even fewer that publicly published their results for the 
> community at large. This has caused inconsistancies and issues within 
> the codebase from time to time.
> 
> Adding continuous integration (and potentially PR
> gates) to the checkin process ensures that simple errors like these 
> are caught and can be fixed on a regular basis.
> 
> ## Strategy
> 
> While a number of CI solutions exist, this proposal will focus on the 
> usage of Azure Dev Ops and Build Pipelines. For demonstration, a 
> sample [TianoCore
> repo](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
> 2Fgithub.com%2Fspbrogan%2Fedk2-staging.git&amp;data=02%7C01%7Csean.bro
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=QmzQhemLUNB
> 7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
> (branch edk2-stuart-ci-latest) and [Dev Ops
> Pipeline](https://nam06.safelinks.protection.outlook.com/?url=https%3A
> %2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-&amp;data=02%7C01%7Csean.bro
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=rS%2FrjIFkS
> uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reserved=0
> play/_build?definitionId=12) have been set up.
> 
> Furthermore, this proposal will leverage the TianoCore python tools 
> PIP modules:
> [library](https://nam06.safelinks.protection.outlook.com/?url=https%3A
> %2F%2Fpypi.org%2Fproject%2Fedk2-pytool-&amp;data=02%7C01%7Csean.brogan
> %40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=jzM6MKZFFAEvyR
> 4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> library/) and
> [extensions](https://nam06.safelinks.protection.outlook.com/?url=https
> %3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-&amp;data=02%7C01%7Csean.bro
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=jzM6MKZFFAE
> vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> extensions/) (with repos located
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
> r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> library) and 
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-&amp;data=02%7C01%7Csean.brogan%40mic
> rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVwGwCXXGBgcmHQ%2B
> 0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
> pytool-extensions)).
> 
> The primary execution flows can be found in the `azure- 
> pipelines-pr-gate.yml` and `azure-pipelines-pr-gate- linux.yml` files. 
> These YAML files are consumed by the Azure Dev Ops Build Pipeline and 
> dictate what server resources should be used, how they should be 
> configured, and what processes should be run on them.
> An overview of this schema can be found
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fdocs.microsoft.com%2Fen-&amp;data=02%7C01%7Csean.brogan%40microsoft
> .com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011
> db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3VnizwKFZ8l1BWoLcPy%
> 2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
> us/azure/devops/pipelines/yaml-schema?view=azure-
> devops&tabs=schema).
> 
> Inspection of these files reveals the EDKII Tools commands that make 
> up the primary processes for the CI
> build: 'stuart_setup', 'stuart_update', and 'stuart_ci_build'. These 
> commands come from the EDKII Tools PIP modules and are configured as 
> described below. More documentation on the stuart tools can be found 
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
> r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> extensions/blob/master/docs/using.md) and
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
> r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> extensions/blob/master/docs/features/feature_invocables
> .md).
> 
> ## Configuration
> 
> Configuration of the CI process consists of (in order of precedence):
> * command-line arguments passed in via the Pipeline YAML
> * a per-package configuration file (e.g. `<package-
> name>.mu.yaml`) that is detected by the CI system in
> EDKII Tools.
> * a global configuration Python module (e.g.
> `CISetting.py`) passed in via the command-line
> 
> The global configuration file is described in [this
> readme](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.brog
> an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9R
> vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> extensions/blob/master/docs/usability/using_settings_ma
> nager.md) from the EDKII Tools documentation. This configuration is 
> written as a Python module so that decisions can be made dynamically 
> based on command line parameters and codebase state.
> 
> The per-package configuration file can override most settings in the 
> global configuration file, but is not dynamic. This file can be used 
> to skip or customize tests that may be incompatible with a specific 
> package.
> By default, the global configuration will try to run all tests on all 
> packages.
> 
> ## CI Test Types
> 
> All CI tests are instances of EDKII Tools plugins.
> Documentation on the plugin system can be found
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=1AM4geY%2BEFh
> ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
> extensions/blob/master/docs/usability/using_plugin_mana
> ger.md) and 
> [here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
> %2Fgithub.com%2Ftianocore%2Fedk2-&amp;data=02%7C01%7Csean.brogan%40mic
> rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d
> 7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPMvrfG3V4doNM0%2B
> cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
> pytool-
> extensions/blob/master/docs/features/feature_plugin_man
> ager.md). Upon invocation, each plugin will be passed the path to the 
> current package under test and a dictionary containing its targeted 
> configuration, as assembled from the command line, per-package 
> configuration, and global configuration.
> 
> Note: CI plugins are considered unique from build plugins and helper 
> plugins, even though some CI plugins may execute steps of a build.
> 
> In the example, these plugins live alongside the code under test (in 
> the `BaseTools` directory), but may be moved to the 'edk2-test' repo 
> if that location makes more sense for the community.
> 
> ### Module Inclusion Test - DscCompleteCheck
> 
> This test scans all available modules (via INF files) and compares 
> them to the package-level DSC file for the package each module is 
> contained within. The test considers it an error if any module does 
> not appear in the `Components` section of at least one package-level 
> DSC (indicating that it would not be built if the package were built).
> 
> ### Code Compilation Test - CompilerPlugin
> 
> Once the Module Inclusion Test has verified that all modules would be 
> built if all package-level DSCs were built, the Code Compilation Test 
> simply runs through and builds every package-level DSC on every 
> toolchain and for every architecture that is supported. Any module 
> that fails to build is considered an error.
> 
> ### Host-Based UnitTests - HostUnitTestCompilerPlugin and 
> HostUnitTestDscCompleteCheck
> 
> The [Testing RFC doc](Readme-Testing-RFC.md) has much more detail on 
> this, but the basic idea is that host- based unit tests can be 
> compiled against individual modules and libraries and run on the build 
> agent (in this case, the Dev Ops build server). The successful and 
> failing test case results are collected and included in the final 
> build report.
> 
> ### GUID Uniqueness Test - GuidCheck
> 
> This test works on the collection of all packages rather than an 
> individual package. It looks at all FILE_GUIDs and GUIDs declared in 
> DEC files and ensures that they are unique for the codebase. This 
> prevents, for example, accidental duplication of GUIDs when using an 
> existing INF as a template for a new module.
> 
> ### Cross-Package Dependency Test - DependencyCheck
> 
> This test compares the list of all packages used in INFs files for a 
> given package against a list of "allowed dependencies" in plugin 
> configuration for that package. Any module that depends on a 
> disallowed package will cause a test failure.
> 
> ### Library Declaration Test - LibraryClassCheck
> 
> This test looks at all library header files found in a package's 
> `Include/Library` directory and ensures that all files have a matching 
> LibraryClass declaration in the DEC file for the package. Any missing 
> declarations will cause a failure.
> 
> ### Invalid Character Test - CharEncodingCheck
> 
> This test scans all files in a package to make sure that there are no 
> invalid Unicode characters that may cause build errors in some 
> character sets/localizations.
> 
> ## Next Steps
> 
> * Receive community feedback on RFC.
> * Determine where this phase makes sense given existing RFCs from 
> other TianoCore contributors.
> * Optimize testing beharior.
>   * Only run a subset of tests on PRs or individual commits.
>   * Run full testing either once per day or once every several 
> commits.
> * Add more tests/capabilities.
> * Continue to improve results formatting.
> * Continue to improve CI documentation.
>   * Much of this documentation effort is pending community feedback on 
> which parts are needed and what phases are priorities.
> 
> Thanks
> 
> 
> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Michael D 
> Kinney via Groups.Io
> Sent: Thursday, August 29, 2019 1:23 PM
> To: devel@edk2.groups.io; rfc@edk2.groups.io
> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1
> 
> Hello,
> 
> This is a proposal for a first step towards continuous integration for 
> all TianoCore repositories to help improve to quality of commits and 
> automate testing and release processes for all EDK II packages and 
> platforms.
> 
> This is based on work from a number of EDK II community members that 
> have provide valuable input and evaluations.
> 
> * Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins evaluation
> * Laszlo Ersek <mailto:lersek@redhat.com> GitLab evaluation
> * Philippe Mathieu-Daudé <mailto:philmd@redhat.com> GitLab evaluation
> * Sean Brogan <mailto:sean.brogan@microsoft.com> Azure Pipelines and 
> HBFA
> * Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
> Azure Pipelines and HBFA
> * Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA
> 
> The following link is a link to an EDK II WIKI page that contains a 
> summary of the work to date.  Please provide feedback in the EDK II 
> mailing lists.  The WIKI pages will be updated with input from the 
> entire EDK II community.
> 
> 
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-Continuous-
> Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
> com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
> ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
> ;reserved=0
> 
> Proposal
> ========
> Phase 1 of adding continuous integration is limited to the
> edk2 repository.  Additional repositories will be added later.
> 
> The following changes are proposed:
> * Remove EDK II Maintainers write access to edk2 repository.
>   Only EDK II Administrators will continue to have write
>   access, and that should only be used to handle extraordinary
>   events.
> * EDK II Maintainers use a GitHub Pull Request instead of push
>   to commit a patch series to the edk2 repository.
> There are
>   no other changes to the development and review process.  The
>   patch series is prepared in an EDK II maintainer branch with
>   all commit message requirements met on each patch in the series.
> * The edk2 repository only accepts Pull Requests from members
>   of the EDK II Maintainers team.  Pull Requests from anyone else
>   are rejected.
> * Run pre-commit checks using Azure Pipelines
> * If all pre-commit checks pass, then the patch series is auto
>   committed.  The result of this commit must match the contents
>   and commit history that would have occurred using the previous
>   push operation.
> * If any pre-commit checks fail, then notify the submitter.
>   A typical reason for a failure would be a merge conflict with
>   another pull request that was just processed.
> * Limit pre-commit checks execution time to 10 minutes.
> * Provide on-demand builds to EDK II Maintainers that to allow
>   EDK II Maintainers to submit a branch through for the same
>   set of pre-commit checks without submitting a pull request.
> 
> ## Pre-Commit Checks in Phase 1
> * Run and pass PatchCheck.py with no errors
> 
> =====================================================
> 
> The following are some additional pre-commit check
> ideas that could be quickly added once the initial
> version using PatchCheck.py is fully functional.
> Please provide feedback on the ones you like and
> additional ones you think may improve the quality of
> the commits to the edk2 repository.
> 
> ## Proposed Pre-Commit Checks in Phase 2
> * Verify Reviewed-by and Acked-by tags are present with
>   correct maintainer email addresses
> * Verify no non-ASCII characters in modified files
> * Verify no binary files in set of modified files
> * Verify package dependency rules in modified files
> 
> ## Proposed Pre-Commit Checks in Phase 3
> * Run ECC on modified files
> * Verify modified modules/libs build
> * Run available host based tests (HBFA) against
> modified
>   modules/libs
> 
> Best regards,
> 
> Mike
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Hi Sean,

For host based tests, I agree that VS2017 or VS2019
would be a good choice.  Pick the one with the best
coverage and easiest for developers to get feedback on
the test results and test coverage.  That may be sufficient
for automated CI tests.  Enabling other tool chains for
host based testing will be required to support developers
that implement unit tests and want to test them locally
before adding to the automated CI tests.

For build tests, I think we need VS2015, VS2017, GCC,
and XCODE5.  We can add VS2019 if there is a request to make
that one of the fully validated tool chains for EDK II.
For pre-commit gates, we could choose to do build tests of
only the libs/modules touched by the patch series.  Then 
perform complete package/platform build tests in a
Daily/Weekly/Release scope.

We need to update the RFC with the specific tool chains and
tool versions for both Host Based tests and Code Compilation
Tests.  They are not the same.

Thanks,

Mike

> -----Original Message-----
> From: Sean Brogan <sean.brogan@microsoft.com>
> Sent: Friday, September 20, 2019 2:30 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: RE: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> 
> Currently it is building on Windows with VS2019.
> VS2017 would be trivial but not worth it in my
> perspective given how aligned the two are.   If you
> really wanted to do a weekly or nightly build it could
> be added to that but I have been focused on a PR build.
> I have a pipeline for GCC on linux.  It doesn't'
> support host based unit tests but I am working to get
> the rest of the tests running.
> I do not have a pipeline for Mac or LLVM/Clang.  That
> would be next and I think your files below should help.
> 
> To date we use PYTOOLs features to install our extra
> tools.  This makes it super easy and works for both
> local and server based builds.  It gives strong
> versioning and management of that.  It also tracks
> those versions and you can see them on every build in
> the Built Tool Report artifact.   It can be downloaded
> here for html version.
> https://dev.azure.com/tianocore/edk2-ci-
> play/_build/results?buildId=803&view=artifacts.
> BUILD_TOOL_REPORT.html
> 
> Here it is pasted as plain text below.  You can see the
> iasl and nasm versions here.
> 
> Key	Value	Type
> TOOL_CHAIN_TAG	VS2019	TOOL
> VC Version	14.22.27905	TOOL
> d:\a\1\s\Conf\build_rule.txt	1.04	INFO
> d:\a\1\s\Conf\target.txt	1.03	INFO
> d:\a\1\s\Conf\tools_def.txt	1.22	INFO
> iasl	20190215.0.0	INFO
> Mu-Basetools	2019.03.1	INFO
> mu_nasm	2.14.02	INFO
> 
> 
> Hope that helps.
> 
> I would be happy to queue up this topic for next design
> meeting and open it up to questions?  I hope others
> would take a look prior to meeting.
> 
> Thanks
> Sean
> 
> 
> 
> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Thursday, September 19, 2019 2:56 PM
> To: devel@edk2.groups.io; Sean Brogan
> <sean.brogan@microsoft.com>; rfc@edk2.groups.io;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: RE: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> 
> Hi Sean,
> 
> Which OS/Compiler configurations are currently enabled
> for the Code Compilation Test?
> 
> I have been working on enabling multiple OS/Compiler
> configurations in Azure Pipelines.  There are some
> tools that need to be installed for each of these
> environments.
> Examples include NASM, iASL, Python.
> 
> For the work you have done, how are these extra tools
> installed?  Is it in the YML files or in the Python
> scripts.
> 
> One critical task is to identify the tools and their
> specific versions that the CI system is configured to
> use.
> These configurations should be documented in a Wiki
> page and updated as new tools are released and adopted
> by EDK II.
> The inventory of tools used to validate a release
> should Also be documented in a release notes for a
> stable tag.
> 
> Here are the YML files that install the additional
> tools required to support EDK II builds.  I need the
> source and versions of these tools to be reviewed and
> approved.
> 
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisi
> tes.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%
> 7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91
> ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=h
> M4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;r
> eserved=0
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisit
> es.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7
> C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ
> 2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;rese
> rved=0
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisite
> s.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C
> 9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzA
> mzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;res
> erved=0
> 
> Thanks,
> 
> Mike
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Sean
> > via Groups.Io
> > Sent: Thursday, August 29, 2019 7:22 PM
> > To: rfc@edk2.groups.io; Kinney, Michael D
> > <michael.d.kinney@intel.com>; devel@edk2.groups.io
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Subject: Re: [edk2-devel] [RFC] EDK II Continuous
> Integration Phase 1
> >
> > Mike, as you mentioned we have been working towards
> enabling a
> > practical and extensible CI for Edk2 using Azure dev
> ops and the
> > recently added edk2-pytool infrastructure.  We have
> been using similar
> > CI for Project Mu for the last few years.
> >
> > Our approach is a little different in that we focus
> on validating the
> > whole code base rather than just the incoming patch.
> We do this
> > because we have found unexpected consequences of
> patches and overall
> > we want all code to be compliant not just new
> additions.  We have
> > found the time to test the whole tree is not much
> longer than only the
> > parts impacted by a code change (except maybe when
> running the entire
> > compile test on every package).  This obviously comes
> with an initial
> > tax of needing to get the codebase into compliant
> form.
> > Anyway we have prepared an RFC in addition to yours
> and would like to
> > see these two efforts merged together.
> >
> > We are still working on making a few optimizations.
> > Currently if the full set of tests are run we take
> about 20 minutes.
> > This is because compiling MdeModulePkg for debug,
> release, and host
> > based tests take a while.  Most other packages are in
> the 10 minute
> > range.  We do have easy ways to disable or limit
> certain tests as well
> > as expand the matrix to leverage more cloud resources
> (more parallel
> > builds).
> >
> >
> > Content is best viewed online with links to helpful
> content but is
> > also attached below:
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgith
> > ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-
> &amp;data=02%7C01%7Csea
> >
> n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c
> 1e1b%7C72f988bf
> >
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&a
> mp;sdata=xOiPHH
> >
> VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserv
> ed=0
> > stuart-ci-latest/Readme-CI-RFC.md
> >
> > # CI and PR Gates
> >
> > ## Background
> >
> > Historically, while the TianoCore maintainers and
> stewards have done a
> > fantastic job of keeping contribution policies
> consistent and
> > contributions clean and well-documented, there have
> been few processes
> > that ran to verify the sanity, cleanliness, and
> efficacy of the
> > codebase, and even fewer that publicly published
> their results for the
> > community at large. This has caused inconsistancies
> and issues within
> > the codebase from time to time.
> >
> > Adding continuous integration (and potentially PR
> > gates) to the checkin process ensures that simple
> errors like these
> > are caught and can be fixed on a regular basis.
> >
> > ## Strategy
> >
> > While a number of CI solutions exist, this proposal
> will focus on the
> > usage of Azure Dev Ops and Build Pipelines. For
> demonstration, a
> > sample [TianoCore
> >
> repo](https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%
> > 2Fgithub.com%2Fspbrogan%2Fedk2-
> staging.git&amp;data=02%7C01%7Csean.bro
> >
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> 7C72f988bf86f14
> >
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> ata=QmzQhemLUNB
> > 7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
> > (branch edk2-stuart-ci-latest) and [Dev Ops
> >
> Pipeline](https://nam06.safelinks.protection.outlook.co
> m/?url=https%3A
> > %2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-
> &amp;data=02%7C01%7Csean.bro
> >
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> 7C72f988bf86f14
> >
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> ata=rS%2FrjIFkS
> >
> uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reser
> ved=0
> > play/_build?definitionId=12) have been set up.
> >
> > Furthermore, this proposal will leverage the
> TianoCore python tools
> > PIP modules:
> >
> [library](https://nam06.safelinks.protection.outlook.co
> m/?url=https%3A
> > %2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.brogan
> >
> %40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C7
> 2f988bf86f141af
> >
> 91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata
> =jzM6MKZFFAEvyR
> > 4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> > library/) and
> >
> [extensions](https://nam06.safelinks.protection.outlook
> .com/?url=https
> > %3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.bro
> >
> gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> 7C72f988bf86f14
> >
> 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> ata=jzM6MKZFFAE
> > vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> > extensions/) (with repos located
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.broga
> >
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> 72f988bf86f141a
> >
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> a=OwT%2BiNTN9Rv
> > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > library) and
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-
> &amp;data=02%7C01%7Csean.brogan%40mic
> >
> rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
> f86f141af91ab2d
> >
> 7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVw
> GwCXXGBgcmHQ%2B
> > 0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
> > pytool-extensions)).
> >
> > The primary execution flows can be found in the
> `azure-
> > pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
> linux.yml` files.
> > These YAML files are consumed by the Azure Dev Ops
> Build Pipeline and
> > dictate what server resources should be used, how
> they should be
> > configured, and what processes should be run on them.
> > An overview of this schema can be found
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fdocs.microsoft.com%2Fen-
> &amp;data=02%7C01%7Csean.brogan%40microsoft
> >
> .com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
> 1af91ab2d7cd011
> >
> db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3Vniz
> wKFZ8l1BWoLcPy%
> > 2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
> > us/azure/devops/pipelines/yaml-schema?view=azure-
> > devops&tabs=schema).
> >
> > Inspection of these files reveals the EDKII Tools
> commands that make
> > up the primary processes for the CI
> > build: 'stuart_setup', 'stuart_update', and
> 'stuart_ci_build'. These
> > commands come from the EDKII Tools PIP modules and
> are configured as
> > described below. More documentation on the stuart
> tools can be found
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.broga
> >
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> 72f988bf86f141a
> >
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> a=OwT%2BiNTN9Rv
> > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > extensions/blob/master/docs/using.md) and
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.broga
> >
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> 72f988bf86f141a
> >
> f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> a=OwT%2BiNTN9Rv
> > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> >
> extensions/blob/master/docs/features/feature_invocables
> > .md).
> >
> > ## Configuration
> >
> > Configuration of the CI process consists of (in order
> of precedence):
> > * command-line arguments passed in via the Pipeline
> YAML
> > * a per-package configuration file (e.g. `<package-
> > name>.mu.yaml`) that is detected by the CI system in
> > EDKII Tools.
> > * a global configuration Python module (e.g.
> > `CISetting.py`) passed in via the command-line
> >
> > The global configuration file is described in [this
> >
> readme](https://nam06.safelinks.protection.outlook.com/
> ?url=https%3A%2
> > F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.brog
> >
> an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7
> C72f988bf86f141
> >
> af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sda
> ta=OwT%2BiNTN9R
> > vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> >
> extensions/blob/master/docs/usability/using_settings_ma
> > nager.md) from the EDKII Tools documentation. This
> configuration is
> > written as a Python module so that decisions can be
> made dynamically
> > based on command line parameters and codebase state.
> >
> > The per-package configuration file can override most
> settings in the
> > global configuration file, but is not dynamic. This
> file can be used
> > to skip or customize tests that may be incompatible
> with a specific
> > package.
> > By default, the global configuration will try to run
> all tests on all
> > packages.
> >
> > ## CI Test Types
> >
> > All CI tests are instances of EDKII Tools plugins.
> > Documentation on the plugin system can be found
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> &amp;data=02%7C01%7Csean.broga
> >
> n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> 72f988bf86f141a
> >
> f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdat
> a=1AM4geY%2BEFh
> > ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
> >
> extensions/blob/master/docs/usability/using_plugin_mana
> > ger.md) and
> >
> [here](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F
> > %2Fgithub.com%2Ftianocore%2Fedk2-
> &amp;data=02%7C01%7Csean.brogan%40mic
> >
> rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
> f86f141af91ab2d
> >
> 7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPM
> vrfG3V4doNM0%2B
> > cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
> > pytool-
> >
> extensions/blob/master/docs/features/feature_plugin_man
> > ager.md). Upon invocation, each plugin will be passed
> the path to the
> > current package under test and a dictionary
> containing its targeted
> > configuration, as assembled from the command line,
> per-package
> > configuration, and global configuration.
> >
> > Note: CI plugins are considered unique from build
> plugins and helper
> > plugins, even though some CI plugins may execute
> steps of a build.
> >
> > In the example, these plugins live alongside the code
> under test (in
> > the `BaseTools` directory), but may be moved to the
> 'edk2-test' repo
> > if that location makes more sense for the community.
> >
> > ### Module Inclusion Test - DscCompleteCheck
> >
> > This test scans all available modules (via INF files)
> and compares
> > them to the package-level DSC file for the package
> each module is
> > contained within. The test considers it an error if
> any module does
> > not appear in the `Components` section of at least
> one package-level
> > DSC (indicating that it would not be built if the
> package were built).
> >
> > ### Code Compilation Test - CompilerPlugin
> >
> > Once the Module Inclusion Test has verified that all
> modules would be
> > built if all package-level DSCs were built, the Code
> Compilation Test
> > simply runs through and builds every package-level
> DSC on every
> > toolchain and for every architecture that is
> supported. Any module
> > that fails to build is considered an error.
> >
> > ### Host-Based UnitTests - HostUnitTestCompilerPlugin
> and
> > HostUnitTestDscCompleteCheck
> >
> > The [Testing RFC doc](Readme-Testing-RFC.md) has much
> more detail on
> > this, but the basic idea is that host- based unit
> tests can be
> > compiled against individual modules and libraries and
> run on the build
> > agent (in this case, the Dev Ops build server). The
> successful and
> > failing test case results are collected and included
> in the final
> > build report.
> >
> > ### GUID Uniqueness Test - GuidCheck
> >
> > This test works on the collection of all packages
> rather than an
> > individual package. It looks at all FILE_GUIDs and
> GUIDs declared in
> > DEC files and ensures that they are unique for the
> codebase. This
> > prevents, for example, accidental duplication of
> GUIDs when using an
> > existing INF as a template for a new module.
> >
> > ### Cross-Package Dependency Test - DependencyCheck
> >
> > This test compares the list of all packages used in
> INFs files for a
> > given package against a list of "allowed
> dependencies" in plugin
> > configuration for that package. Any module that
> depends on a
> > disallowed package will cause a test failure.
> >
> > ### Library Declaration Test - LibraryClassCheck
> >
> > This test looks at all library header files found in
> a package's
> > `Include/Library` directory and ensures that all
> files have a matching
> > LibraryClass declaration in the DEC file for the
> package. Any missing
> > declarations will cause a failure.
> >
> > ### Invalid Character Test - CharEncodingCheck
> >
> > This test scans all files in a package to make sure
> that there are no
> > invalid Unicode characters that may cause build
> errors in some
> > character sets/localizations.
> >
> > ## Next Steps
> >
> > * Receive community feedback on RFC.
> > * Determine where this phase makes sense given
> existing RFCs from
> > other TianoCore contributors.
> > * Optimize testing beharior.
> >   * Only run a subset of tests on PRs or individual
> commits.
> >   * Run full testing either once per day or once
> every several
> > commits.
> > * Add more tests/capabilities.
> > * Continue to improve results formatting.
> > * Continue to improve CI documentation.
> >   * Much of this documentation effort is pending
> community feedback on
> > which parts are needed and what phases are
> priorities.
> >
> > Thanks
> >
> >
> > -----Original Message-----
> > From: rfc@edk2.groups.io <rfc@edk2.groups.io> On
> Behalf Of Michael D
> > Kinney via Groups.Io
> > Sent: Thursday, August 29, 2019 1:23 PM
> > To: devel@edk2.groups.io; rfc@edk2.groups.io
> > Subject: [edk2-rfc] [RFC] EDK II Continuous
> Integration Phase 1
> >
> > Hello,
> >
> > This is a proposal for a first step towards
> continuous integration for
> > all TianoCore repositories to help improve to quality
> of commits and
> > automate testing and release processes for all EDK II
> packages and
> > platforms.
> >
> > This is based on work from a number of EDK II
> community members that
> > have provide valuable input and evaluations.
> >
> > * Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
> evaluation
> > * Laszlo Ersek <mailto:lersek@redhat.com> GitLab
> evaluation
> > * Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
> GitLab evaluation
> > * Sean Brogan <mailto:sean.brogan@microsoft.com>
> Azure Pipelines and
> > HBFA
> > * Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
> > Azure Pipelines and HBFA
> > * Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA
> >
> > The following link is a link to an EDK II WIKI page
> that contains a
> > summary of the work to date.  Please provide feedback
> in the EDK II
> > mailing lists.  The WIKI pages will be updated with
> input from the
> > entire EDK II community.
> >
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> >
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> > %2Fwiki%2FEDK-II-Continuous-
> >
> Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
> >
> com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
> >
> af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
> >
> ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
> > ;reserved=0
> >
> > Proposal
> > ========
> > Phase 1 of adding continuous integration is limited
> to the
> > edk2 repository.  Additional repositories will be
> added later.
> >
> > The following changes are proposed:
> > * Remove EDK II Maintainers write access to edk2
> repository.
> >   Only EDK II Administrators will continue to have
> write
> >   access, and that should only be used to handle
> extraordinary
> >   events.
> > * EDK II Maintainers use a GitHub Pull Request
> instead of push
> >   to commit a patch series to the edk2 repository.
> > There are
> >   no other changes to the development and review
> process.  The
> >   patch series is prepared in an EDK II maintainer
> branch with
> >   all commit message requirements met on each patch
> in the series.
> > * The edk2 repository only accepts Pull Requests from
> members
> >   of the EDK II Maintainers team.  Pull Requests from
> anyone else
> >   are rejected.
> > * Run pre-commit checks using Azure Pipelines
> > * If all pre-commit checks pass, then the patch
> series is auto
> >   committed.  The result of this commit must match
> the contents
> >   and commit history that would have occurred using
> the previous
> >   push operation.
> > * If any pre-commit checks fail, then notify the
> submitter.
> >   A typical reason for a failure would be a merge
> conflict with
> >   another pull request that was just processed.
> > * Limit pre-commit checks execution time to 10
> minutes.
> > * Provide on-demand builds to EDK II Maintainers that
> to allow
> >   EDK II Maintainers to submit a branch through for
> the same
> >   set of pre-commit checks without submitting a pull
> request.
> >
> > ## Pre-Commit Checks in Phase 1
> > * Run and pass PatchCheck.py with no errors
> >
> > =====================================================
> >
> > The following are some additional pre-commit check
> ideas that could be
> > quickly added once the initial version using
> PatchCheck.py is fully
> > functional.
> > Please provide feedback on the ones you like and
> additional ones you
> > think may improve the quality of the commits to the
> edk2 repository.
> >
> > ## Proposed Pre-Commit Checks in Phase 2
> > * Verify Reviewed-by and Acked-by tags are present
> with
> >   correct maintainer email addresses
> > * Verify no non-ASCII characters in modified files
> > * Verify no binary files in set of modified files
> > * Verify package dependency rules in modified files
> >
> > ## Proposed Pre-Commit Checks in Phase 3
> > * Run ECC on modified files
> > * Verify modified modules/libs build
> > * Run available host based tests (HBFA) against
> modified
> >   modules/libs
> >
> > Best regards,
> >
> > Mike
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Liming Gao]

Mike:

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D Kinney
> Sent: Tuesday, September 24, 2019 1:44 AM
> To: Sean Brogan <sean.brogan@microsoft.com>; devel@edk2.groups.io; rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
> 
> Hi Sean,
> 
> For host based tests, I agree that VS2017 or VS2019
> would be a good choice.  Pick the one with the best
> coverage and easiest for developers to get feedback on
> the test results and test coverage.  That may be sufficient
> for automated CI tests.  Enabling other tool chains for
> host based testing will be required to support developers
> that implement unit tests and want to test them locally
> before adding to the automated CI tests.
> 
> For build tests, I think we need VS2015, VS2017, GCC,
> and XCODE5.  We can add VS2019 if there is a request to make
> that one of the fully validated tool chains for EDK II.
VS2019 tool chain has been added for IA32/X64/ARM/AARH64/EBC archs. 
CLANG9 tool chain for IA32/X64 will be added. I request to add CLANG9 build test. 

Thanks
Liming
> For pre-commit gates, we could choose to do build tests of
> only the libs/modules touched by the patch series.  Then
> perform complete package/platform build tests in a
> Daily/Weekly/Release scope.
> 
> We need to update the RFC with the specific tool chains and
> tool versions for both Host Based tests and Code Compilation
> Tests.  They are not the same.
> 
> Thanks,
> 
> Mike
> 
> > -----Original Message-----
> > From: Sean Brogan <sean.brogan@microsoft.com>
> > Sent: Friday, September 20, 2019 2:30 PM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
> > Michael D <michael.d.kinney@intel.com>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Subject: RE: [edk2-devel] [RFC] EDK II Continuous
> > Integration Phase 1
> >
> > Currently it is building on Windows with VS2019.
> > VS2017 would be trivial but not worth it in my
> > perspective given how aligned the two are.   If you
> > really wanted to do a weekly or nightly build it could
> > be added to that but I have been focused on a PR build.
> > I have a pipeline for GCC on linux.  It doesn't'
> > support host based unit tests but I am working to get
> > the rest of the tests running.
> > I do not have a pipeline for Mac or LLVM/Clang.  That
> > would be next and I think your files below should help.
> >
> > To date we use PYTOOLs features to install our extra
> > tools.  This makes it super easy and works for both
> > local and server based builds.  It gives strong
> > versioning and management of that.  It also tracks
> > those versions and you can see them on every build in
> > the Built Tool Report artifact.   It can be downloaded
> > here for html version.
> > https://dev.azure.com/tianocore/edk2-ci-
> > play/_build/results?buildId=803&view=artifacts.
> > BUILD_TOOL_REPORT.html
> >
> > Here it is pasted as plain text below.  You can see the
> > iasl and nasm versions here.
> >
> > Key	Value	Type
> > TOOL_CHAIN_TAG	VS2019	TOOL
> > VC Version	14.22.27905	TOOL
> > d:\a\1\s\Conf\build_rule.txt	1.04	INFO
> > d:\a\1\s\Conf\target.txt	1.03	INFO
> > d:\a\1\s\Conf\tools_def.txt	1.22	INFO
> > iasl	20190215.0.0	INFO
> > Mu-Basetools	2019.03.1	INFO
> > mu_nasm	2.14.02	INFO
> >
> >
> > Hope that helps.
> >
> > I would be happy to queue up this topic for next design
> > meeting and open it up to questions?  I hope others
> > would take a look prior to meeting.
> >
> > Thanks
> > Sean
> >
> >
> >
> > -----Original Message-----
> > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > Sent: Thursday, September 19, 2019 2:56 PM
> > To: devel@edk2.groups.io; Sean Brogan
> > <sean.brogan@microsoft.com>; rfc@edk2.groups.io;
> > Kinney, Michael D <michael.d.kinney@intel.com>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Subject: RE: [edk2-devel] [RFC] EDK II Continuous
> > Integration Phase 1
> >
> > Hi Sean,
> >
> > Which OS/Compiler configurations are currently enabled
> > for the Code Compilation Test?
> >
> > I have been working on enabling multiple OS/Compiler
> > configurations in Azure Pipelines.  There are some
> > tools that need to be installed for each of these
> > environments.
> > Examples include NASM, iASL, Python.
> >
> > For the work you have done, how are these extra tools
> > installed?  Is it in the YML files or in the Python
> > scripts.
> >
> > One critical task is to identify the tools and their
> > specific versions that the CI system is configured to
> > use.
> > These configurations should be documented in a Wiki
> > page and updated as new tools are released and adopted
> > by EDK II.
> > The inventory of tools used to validate a release
> > should Also be documented in a release notes for a
> > stable tag.
> >
> > Here are the YML files that install the additional
> > tools required to support EDK II builds.  I need the
> > source and versions of these tools to be reviewed and
> > approved.
> >
> > https://nam06.safelinks.protection.outlook.com/?url=htt
> > ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> > ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisi
> > tes.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%
> > 7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91
> > ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=h
> > M4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;r
> > eserved=0
> > https://nam06.safelinks.protection.outlook.com/?url=htt
> > ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> > ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisit
> > es.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7
> > C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91a
> > b2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ
> > 2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;rese
> > rved=0
> > https://nam06.safelinks.protection.outlook.com/?url=htt
> > ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
> > ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisite
> > s.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C
> > 9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab
> > 2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzA
> > mzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;res
> > erved=0
> >
> > Thanks,
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On
> > Behalf Of Sean
> > > via Groups.Io
> > > Sent: Thursday, August 29, 2019 7:22 PM
> > > To: rfc@edk2.groups.io; Kinney, Michael D
> > > <michael.d.kinney@intel.com>; devel@edk2.groups.io
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > Subject: Re: [edk2-devel] [RFC] EDK II Continuous
> > Integration Phase 1
> > >
> > > Mike, as you mentioned we have been working towards
> > enabling a
> > > practical and extensible CI for Edk2 using Azure dev
> > ops and the
> > > recently added edk2-pytool infrastructure.  We have
> > been using similar
> > > CI for Project Mu for the last few years.
> > >
> > > Our approach is a little different in that we focus
> > on validating the
> > > whole code base rather than just the incoming patch.
> > We do this
> > > because we have found unexpected consequences of
> > patches and overall
> > > we want all code to be compliant not just new
> > additions.  We have
> > > found the time to test the whole tree is not much
> > longer than only the
> > > parts impacted by a code change (except maybe when
> > running the entire
> > > compile test on every package).  This obviously comes
> > with an initial
> > > tax of needing to get the codebase into compliant
> > form.
> > > Anyway we have prepared an RFC in addition to yours
> > and would like to
> > > see these two efforts merged together.
> > >
> > > We are still working on making a few optimizations.
> > > Currently if the full set of tests are run we take
> > about 20 minutes.
> > > This is because compiling MdeModulePkg for debug,
> > release, and host
> > > based tests take a while.  Most other packages are in
> > the 10 minute
> > > range.  We do have easy ways to disable or limit
> > certain tests as well
> > > as expand the matrix to leverage more cloud resources
> > (more parallel
> > > builds).
> > >
> > >
> > > Content is best viewed online with links to helpful
> > content but is
> > > also attached below:
> > >
> > https://nam06.safelinks.protection.outlook.com/?url=htt
> > ps%3A%2F%2Fgith
> > > ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-
> > &amp;data=02%7C01%7Csea
> > >
> > n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c
> > 1e1b%7C72f988bf
> > >
> > 86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&a
> > mp;sdata=xOiPHH
> > >
> > VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserv
> > ed=0
> > > stuart-ci-latest/Readme-CI-RFC.md
> > >
> > > # CI and PR Gates
> > >
> > > ## Background
> > >
> > > Historically, while the TianoCore maintainers and
> > stewards have done a
> > > fantastic job of keeping contribution policies
> > consistent and
> > > contributions clean and well-documented, there have
> > been few processes
> > > that ran to verify the sanity, cleanliness, and
> > efficacy of the
> > > codebase, and even fewer that publicly published
> > their results for the
> > > community at large. This has caused inconsistancies
> > and issues within
> > > the codebase from time to time.
> > >
> > > Adding continuous integration (and potentially PR
> > > gates) to the checkin process ensures that simple
> > errors like these
> > > are caught and can be fixed on a regular basis.
> > >
> > > ## Strategy
> > >
> > > While a number of CI solutions exist, this proposal
> > will focus on the
> > > usage of Azure Dev Ops and Build Pipelines. For
> > demonstration, a
> > > sample [TianoCore
> > >
> > repo](https://nam06.safelinks.protection.outlook.com/?u
> > rl=https%3A%2F%
> > > 2Fgithub.com%2Fspbrogan%2Fedk2-
> > staging.git&amp;data=02%7C01%7Csean.bro
> > >
> > gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> > 7C72f988bf86f14
> > >
> > 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> > ata=QmzQhemLUNB
> > > 7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
> > > (branch edk2-stuart-ci-latest) and [Dev Ops
> > >
> > Pipeline](https://nam06.safelinks.protection.outlook.co
> > m/?url=https%3A
> > > %2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-
> > &amp;data=02%7C01%7Csean.bro
> > >
> > gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> > 7C72f988bf86f14
> > >
> > 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> > ata=rS%2FrjIFkS
> > >
> > uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reser
> > ved=0
> > > play/_build?definitionId=12) have been set up.
> > >
> > > Furthermore, this proposal will leverage the
> > TianoCore python tools
> > > PIP modules:
> > >
> > [library](https://nam06.safelinks.protection.outlook.co
> > m/?url=https%3A
> > > %2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.brogan
> > >
> > %40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C7
> > 2f988bf86f141af
> > >
> > 91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata
> > =jzM6MKZFFAEvyR
> > > 4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> > > library/) and
> > >
> > [extensions](https://nam06.safelinks.protection.outlook
> > .com/?url=https
> > > %3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.bro
> > >
> > gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
> > 7C72f988bf86f14
> > >
> > 1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
> > ata=jzM6MKZFFAE
> > > vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
> > > extensions/) (with repos located
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.broga
> > >
> > n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> > 72f988bf86f141a
> > >
> > f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> > a=OwT%2BiNTN9Rv
> > > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > > library) and
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-
> > &amp;data=02%7C01%7Csean.brogan%40mic
> > >
> > rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
> > f86f141af91ab2d
> > >
> > 7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVw
> > GwCXXGBgcmHQ%2B
> > > 0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
> > > pytool-extensions)).
> > >
> > > The primary execution flows can be found in the
> > `azure-
> > > pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
> > linux.yml` files.
> > > These YAML files are consumed by the Azure Dev Ops
> > Build Pipeline and
> > > dictate what server resources should be used, how
> > they should be
> > > configured, and what processes should be run on them.
> > > An overview of this schema can be found
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fdocs.microsoft.com%2Fen-
> > &amp;data=02%7C01%7Csean.brogan%40microsoft
> > >
> > .com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
> > 1af91ab2d7cd011
> > >
> > db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3Vniz
> > wKFZ8l1BWoLcPy%
> > > 2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
> > > us/azure/devops/pipelines/yaml-schema?view=azure-
> > > devops&tabs=schema).
> > >
> > > Inspection of these files reveals the EDKII Tools
> > commands that make
> > > up the primary processes for the CI
> > > build: 'stuart_setup', 'stuart_update', and
> > 'stuart_ci_build'. These
> > > commands come from the EDKII Tools PIP modules and
> > are configured as
> > > described below. More documentation on the stuart
> > tools can be found
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.broga
> > >
> > n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> > 72f988bf86f141a
> > >
> > f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> > a=OwT%2BiNTN9Rv
> > > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > > extensions/blob/master/docs/using.md) and
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.broga
> > >
> > n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> > 72f988bf86f141a
> > >
> > f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
> > a=OwT%2BiNTN9Rv
> > > r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > >
> > extensions/blob/master/docs/features/feature_invocables
> > > .md).
> > >
> > > ## Configuration
> > >
> > > Configuration of the CI process consists of (in order
> > of precedence):
> > > * command-line arguments passed in via the Pipeline
> > YAML
> > > * a per-package configuration file (e.g. `<package-
> > > name>.mu.yaml`) that is detected by the CI system in
> > > EDKII Tools.
> > > * a global configuration Python module (e.g.
> > > `CISetting.py`) passed in via the command-line
> > >
> > > The global configuration file is described in [this
> > >
> > readme](https://nam06.safelinks.protection.outlook.com/
> > ?url=https%3A%2
> > > F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.brog
> > >
> > an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7
> > C72f988bf86f141
> > >
> > af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sda
> > ta=OwT%2BiNTN9R
> > > vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
> > >
> > extensions/blob/master/docs/usability/using_settings_ma
> > > nager.md) from the EDKII Tools documentation. This
> > configuration is
> > > written as a Python module so that decisions can be
> > made dynamically
> > > based on command line parameters and codebase state.
> > >
> > > The per-package configuration file can override most
> > settings in the
> > > global configuration file, but is not dynamic. This
> > file can be used
> > > to skip or customize tests that may be incompatible
> > with a specific
> > > package.
> > > By default, the global configuration will try to run
> > all tests on all
> > > packages.
> > >
> > > ## CI Test Types
> > >
> > > All CI tests are instances of EDKII Tools plugins.
> > > Documentation on the plugin system can be found
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-pytool-
> > &amp;data=02%7C01%7Csean.broga
> > >
> > n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
> > 72f988bf86f141a
> > >
> > f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdat
> > a=1AM4geY%2BEFh
> > > ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
> > >
> > extensions/blob/master/docs/usability/using_plugin_mana
> > > ger.md) and
> > >
> > [here](https://nam06.safelinks.protection.outlook.com/?
> > url=https%3A%2F
> > > %2Fgithub.com%2Ftianocore%2Fedk2-
> > &amp;data=02%7C01%7Csean.brogan%40mic
> > >
> > rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
> > f86f141af91ab2d
> > >
> > 7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPM
> > vrfG3V4doNM0%2B
> > > cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
> > > pytool-
> > >
> > extensions/blob/master/docs/features/feature_plugin_man
> > > ager.md). Upon invocation, each plugin will be passed
> > the path to the
> > > current package under test and a dictionary
> > containing its targeted
> > > configuration, as assembled from the command line,
> > per-package
> > > configuration, and global configuration.
> > >
> > > Note: CI plugins are considered unique from build
> > plugins and helper
> > > plugins, even though some CI plugins may execute
> > steps of a build.
> > >
> > > In the example, these plugins live alongside the code
> > under test (in
> > > the `BaseTools` directory), but may be moved to the
> > 'edk2-test' repo
> > > if that location makes more sense for the community.
> > >
> > > ### Module Inclusion Test - DscCompleteCheck
> > >
> > > This test scans all available modules (via INF files)
> > and compares
> > > them to the package-level DSC file for the package
> > each module is
> > > contained within. The test considers it an error if
> > any module does
> > > not appear in the `Components` section of at least
> > one package-level
> > > DSC (indicating that it would not be built if the
> > package were built).
> > >
> > > ### Code Compilation Test - CompilerPlugin
> > >
> > > Once the Module Inclusion Test has verified that all
> > modules would be
> > > built if all package-level DSCs were built, the Code
> > Compilation Test
> > > simply runs through and builds every package-level
> > DSC on every
> > > toolchain and for every architecture that is
> > supported. Any module
> > > that fails to build is considered an error.
> > >
> > > ### Host-Based UnitTests - HostUnitTestCompilerPlugin
> > and
> > > HostUnitTestDscCompleteCheck
> > >
> > > The [Testing RFC doc](Readme-Testing-RFC.md) has much
> > more detail on
> > > this, but the basic idea is that host- based unit
> > tests can be
> > > compiled against individual modules and libraries and
> > run on the build
> > > agent (in this case, the Dev Ops build server). The
> > successful and
> > > failing test case results are collected and included
> > in the final
> > > build report.
> > >
> > > ### GUID Uniqueness Test - GuidCheck
> > >
> > > This test works on the collection of all packages
> > rather than an
> > > individual package. It looks at all FILE_GUIDs and
> > GUIDs declared in
> > > DEC files and ensures that they are unique for the
> > codebase. This
> > > prevents, for example, accidental duplication of
> > GUIDs when using an
> > > existing INF as a template for a new module.
> > >
> > > ### Cross-Package Dependency Test - DependencyCheck
> > >
> > > This test compares the list of all packages used in
> > INFs files for a
> > > given package against a list of "allowed
> > dependencies" in plugin
> > > configuration for that package. Any module that
> > depends on a
> > > disallowed package will cause a test failure.
> > >
> > > ### Library Declaration Test - LibraryClassCheck
> > >
> > > This test looks at all library header files found in
> > a package's
> > > `Include/Library` directory and ensures that all
> > files have a matching
> > > LibraryClass declaration in the DEC file for the
> > package. Any missing
> > > declarations will cause a failure.
> > >
> > > ### Invalid Character Test - CharEncodingCheck
> > >
> > > This test scans all files in a package to make sure
> > that there are no
> > > invalid Unicode characters that may cause build
> > errors in some
> > > character sets/localizations.
> > >
> > > ## Next Steps
> > >
> > > * Receive community feedback on RFC.
> > > * Determine where this phase makes sense given
> > existing RFCs from
> > > other TianoCore contributors.
> > > * Optimize testing beharior.
> > >   * Only run a subset of tests on PRs or individual
> > commits.
> > >   * Run full testing either once per day or once
> > every several
> > > commits.
> > > * Add more tests/capabilities.
> > > * Continue to improve results formatting.
> > > * Continue to improve CI documentation.
> > >   * Much of this documentation effort is pending
> > community feedback on
> > > which parts are needed and what phases are
> > priorities.
> > >
> > > Thanks
> > >
> > >
> > > -----Original Message-----
> > > From: rfc@edk2.groups.io <rfc@edk2.groups.io> On
> > Behalf Of Michael D
> > > Kinney via Groups.Io
> > > Sent: Thursday, August 29, 2019 1:23 PM
> > > To: devel@edk2.groups.io; rfc@edk2.groups.io
> > > Subject: [edk2-rfc] [RFC] EDK II Continuous
> > Integration Phase 1
> > >
> > > Hello,
> > >
> > > This is a proposal for a first step towards
> > continuous integration for
> > > all TianoCore repositories to help improve to quality
> > of commits and
> > > automate testing and release processes for all EDK II
> > packages and
> > > platforms.
> > >
> > > This is based on work from a number of EDK II
> > community members that
> > > have provide valuable input and evaluations.
> > >
> > > * Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
> > evaluation
> > > * Laszlo Ersek <mailto:lersek@redhat.com> GitLab
> > evaluation
> > > * Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
> > GitLab evaluation
> > > * Sean Brogan <mailto:sean.brogan@microsoft.com>
> > Azure Pipelines and
> > > HBFA
> > > * Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
> > > Azure Pipelines and HBFA
> > > * Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA
> > >
> > > The following link is a link to an EDK II WIKI page
> > that contains a
> > > summary of the work to date.  Please provide feedback
> > in the EDK II
> > > mailing lists.  The WIKI pages will be updated with
> > input from the
> > > entire EDK II community.
> > >
> > >
> > >
> > https://nam06.safelinks.protection.outlook.com/?url=htt
> > >
> > ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> > > %2Fwiki%2FEDK-II-Continuous-
> > >
> > Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
> > >
> > com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
> > >
> > af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
> > >
> > ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
> > > ;reserved=0
> > >
> > > Proposal
> > > ========
> > > Phase 1 of adding continuous integration is limited
> > to the
> > > edk2 repository.  Additional repositories will be
> > added later.
> > >
> > > The following changes are proposed:
> > > * Remove EDK II Maintainers write access to edk2
> > repository.
> > >   Only EDK II Administrators will continue to have
> > write
> > >   access, and that should only be used to handle
> > extraordinary
> > >   events.
> > > * EDK II Maintainers use a GitHub Pull Request
> > instead of push
> > >   to commit a patch series to the edk2 repository.
> > > There are
> > >   no other changes to the development and review
> > process.  The
> > >   patch series is prepared in an EDK II maintainer
> > branch with
> > >   all commit message requirements met on each patch
> > in the series.
> > > * The edk2 repository only accepts Pull Requests from
> > members
> > >   of the EDK II Maintainers team.  Pull Requests from
> > anyone else
> > >   are rejected.
> > > * Run pre-commit checks using Azure Pipelines
> > > * If all pre-commit checks pass, then the patch
> > series is auto
> > >   committed.  The result of this commit must match
> > the contents
> > >   and commit history that would have occurred using
> > the previous
> > >   push operation.
> > > * If any pre-commit checks fail, then notify the
> > submitter.
> > >   A typical reason for a failure would be a merge
> > conflict with
> > >   another pull request that was just processed.
> > > * Limit pre-commit checks execution time to 10
> > minutes.
> > > * Provide on-demand builds to EDK II Maintainers that
> > to allow
> > >   EDK II Maintainers to submit a branch through for
> > the same
> > >   set of pre-commit checks without submitting a pull
> > request.
> > >
> > > ## Pre-Commit Checks in Phase 1
> > > * Run and pass PatchCheck.py with no errors
> > >
> > > =====================================================
> > >
> > > The following are some additional pre-commit check
> > ideas that could be
> > > quickly added once the initial version using
> > PatchCheck.py is fully
> > > functional.
> > > Please provide feedback on the ones you like and
> > additional ones you
> > > think may improve the quality of the commits to the
> > edk2 repository.
> > >
> > > ## Proposed Pre-Commit Checks in Phase 2
> > > * Verify Reviewed-by and Acked-by tags are present
> > with
> > >   correct maintainer email addresses
> > > * Verify no non-ASCII characters in modified files
> > > * Verify no binary files in set of modified files
> > > * Verify package dependency rules in modified files
> > >
> > > ## Proposed Pre-Commit Checks in Phase 3
> > > * Run ECC on modified files
> > > * Verify modified modules/libs build
> > > * Run available host based tests (HBFA) against
> > modified
> > >   modules/libs
> > >
> > > Best regards,
> > >
> > > Mike
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> 
> 
> 

Re: [RFC] EDK II Continuous Integration Phase 1

[Liming Gao]

Mike:
  I add my comments. 

>-----Original Message-----
>From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io] On Behalf Of Michael
>D Kinney
>Sent: Friday, August 30, 2019 4:23 AM
>To: devel@edk2.groups.io; rfc@edk2.groups.io
>Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1
>
>Hello,
>
>This is a proposal for a first step towards continuous
>integration for all TianoCore repositories to help
>improve to quality of commits and automate testing and
>release processes for all EDK II packages and platforms.
>
>This is based on work from a number of EDK II community
>members that have provide valuable input and evaluations.
>
>* Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
>* Laszlo Ersek <lersek@redhat.com> GitLab evaluation
>* Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
>* Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
>* Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
>* Jiewen Yao <jiewen.yao@intel.com> HBFA
>
>The following link is a link to an EDK II WIKI page that
>contains a summary of the work to date.  Please provide
>feedback in the EDK II mailing lists.  The WIKI pages will
>be updated with input from the entire EDK II community.
>
>    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-
>Integration
>
>Proposal
>========
>Phase 1 of adding continuous integration is limited to the
>edk2 repository.  Additional repositories will be added later.
>
>The following changes are proposed:
>* Remove EDK II Maintainers write access to edk2 repository.
>  Only EDK II Administrators will continue to have write
>  access, and that should only be used to handle extraordinary
>  events.
>* EDK II Maintainers use a GitHub Pull Request instead of push
>  to commit a patch series to the edk2 repository.  There are
>  no other changes to the development and review process.  The
>  patch series is prepared in an EDK II maintainer branch with
>  all commit message requirements met on each patch in the series.

Will the maintainer manually provide pull request after the patch passes the review?
Can the script scan the mail list and auto trig pull request once the patch gets 
Reviewed-by or Acked-by from Package maintainers?

>* The edk2 repository only accepts Pull Requests from members
>  of the EDK II Maintainers team.  Pull Requests from anyone else
>  are rejected.
>* Run pre-commit checks using Azure Pipelines

The maintainer manually trig pre-commit check or auto trig pre-commit?
By default, pre-commit should be auto trigged based on pull request. 

>* If all pre-commit checks pass, then the patch series is auto
>  committed.  The result of this commit must match the contents
>  and commit history that would have occurred using the previous
>  push operation.
>* If any pre-commit checks fail, then notify the submitter.

Will Pre-commit check fail send the mail to the patch submitter?
The patch submitter need update the patch and go through review process again. 

>  A typical reason for a failure would be a merge conflict with
>  another pull request that was just processed.
>* Limit pre-commit checks execution time to 10 minutes.
>* Provide on-demand builds to EDK II Maintainers that to allow
>  EDK II Maintainers to submit a branch through for the same
>  set of pre-commit checks without submitting a pull request.
>
>## Pre-Commit Checks in Phase 1
>* Run and pass PatchCheck.py with no errors
>
>=====================================================
>
>The following are some additional pre-commit check ideas
>that could be quickly added once the initial version using
>PatchCheck.py is fully functional.  Please provide feedback
>on the ones you like and additional ones you think may
>improve the quality of the commits to the edk2 repository.
>
>## Proposed Pre-Commit Checks in Phase 2
>* Verify Reviewed-by and Acked-by tags are present with
>  correct maintainer email addresses
>* Verify no non-ASCII characters in modified files
>* Verify no binary files in set of modified files

Now, Edk2 has few binary files, like logo.bmp. 
I see one BZ to request logo bmp update. 
(BZ https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
So, we need the exception way for it. 

>* Verify package dependency rules in modified files
>
>## Proposed Pre-Commit Checks in Phase 3
>* Run ECC on modified files
>* Verify modified modules/libs build
>* Run available host based tests (HBFA) against modified
>  modules/libs
>

I request boot test on Emulator and Ovmf in the daily and weekly scope. 
Daily can cover boot to Shell.
Weekly can cover more boot functionality. 

Thanks
Liming
>Best regards,
>
>Mike
>
>
>
>
>
>
>

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

On 08/30/19 10:43, Liming Gao wrote:
> Mike:
>   I add my comments. 
> 
>> -----Original Message-----
>> From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io] On Behalf Of Michael
>> D Kinney
>> Sent: Friday, August 30, 2019 4:23 AM
>> To: devel@edk2.groups.io; rfc@edk2.groups.io
>> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1
>>
>> Hello,
>>
>> This is a proposal for a first step towards continuous
>> integration for all TianoCore repositories to help
>> improve to quality of commits and automate testing and
>> release processes for all EDK II packages and platforms.
>>
>> This is based on work from a number of EDK II community
>> members that have provide valuable input and evaluations.
>>
>> * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
>> * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
>> * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
>> * Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
>> * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
>> * Jiewen Yao <jiewen.yao@intel.com> HBFA
>>
>> The following link is a link to an EDK II WIKI page that
>> contains a summary of the work to date.  Please provide
>> feedback in the EDK II mailing lists.  The WIKI pages will
>> be updated with input from the entire EDK II community.
>>
>>    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-
>> Integration
>>
>> Proposal
>> ========
>> Phase 1 of adding continuous integration is limited to the
>> edk2 repository.  Additional repositories will be added later.
>>
>> The following changes are proposed:
>> * Remove EDK II Maintainers write access to edk2 repository.
>>  Only EDK II Administrators will continue to have write
>>  access, and that should only be used to handle extraordinary
>>  events.
>> * EDK II Maintainers use a GitHub Pull Request instead of push
>>  to commit a patch series to the edk2 repository.  There are
>>  no other changes to the development and review process.  The
>>  patch series is prepared in an EDK II maintainer branch with
>>  all commit message requirements met on each patch in the series.
> 
> Will the maintainer manually provide pull request after the patch passes the review?

Yes. The maintainer will prepare a local branch that is rebased to
master, and has all the mailing list feedback tags (Reviewed-by, etc)
applied. The maintainer also does all the local testing that they
usually do, just before the final "git push origin".

Then, the final "git push origin" action is replaced by:
(1) git push personal-repo topic-branch-pr
(2) manual creation of a GitHub.com Pull Request, for the topic branch
just pushed.

That is, the final "git push origin" action is replaced with the pushing
of a personal (ready-made) topic branch, and a GitHub.com Pull Request
against that branch. The verification and the final merging will be
performed by github.

> Can the script scan the mail list and auto trig pull request once the patch gets 
> Reviewed-by or Acked-by from Package maintainers?

No, it can't. (And, at this stage, it should not.) The coordination
between submitters, maintainers, reviewers, and occasionally, stewards,
for determining when a patch series is ready to go, based on review
discussion, remains the same.

>> * The edk2 repository only accepts Pull Requests from members
>>  of the EDK II Maintainers team.  Pull Requests from anyone else
>>  are rejected.
>> * Run pre-commit checks using Azure Pipelines
> 
> The maintainer manually trig pre-commit check or auto trig pre-commit?

After the maintainer pushes the ready-made branch to their personal
repo, and submits a PR against that, the PR will set off the checks. If
the checks pass, the topic branch is merged.

> By default, pre-commit should be auto trigged based on pull request. 
> 
>> * If all pre-commit checks pass, then the patch series is auto
>>  committed.  The result of this commit must match the contents
>>  and commit history that would have occurred using the previous
>>  push operation.
>> * If any pre-commit checks fail, then notify the submitter.
> 
> Will Pre-commit check fail send the mail to the patch submitter?
> The patch submitter need update the patch and go through review process again. 

My understanding is that, if the testing in GitHub.com fails, the PR
will be rejected and closed. This will generate a notification email for
the maintainer that submitted the PR. The maintainer can then return to
the email thread, and report the CI failure, possibly with a link to the
failed / rejected PR.

Then, indeed, the submitter must rework the series and post a new
version to the list.

It's also possible (and polite) if the maintainer posts the PR link in
the mailing list thread right after submitting the PR. Then the
submitter can monitor the PR too. (Subscribe to it for notifications.)
As I understand it.

Furthermore,

> 
>>  A typical reason for a failure would be a merge conflict with
>>  another pull request that was just processed.
>> * Limit pre-commit checks execution time to 10 minutes.
>> * Provide on-demand builds to EDK II Maintainers that to allow
>>  EDK II Maintainers to submit a branch through for the same
>>  set of pre-commit checks without submitting a pull request.

a maintainer could use this on-demand build & testing facility in the
course of review, to report findings early.

Thanks
Laszlo

>>
>> ## Pre-Commit Checks in Phase 1
>> * Run and pass PatchCheck.py with no errors
>>
>> =====================================================
>>
>> The following are some additional pre-commit check ideas
>> that could be quickly added once the initial version using
>> PatchCheck.py is fully functional.  Please provide feedback
>> on the ones you like and additional ones you think may
>> improve the quality of the commits to the edk2 repository.
>>
>> ## Proposed Pre-Commit Checks in Phase 2
>> * Verify Reviewed-by and Acked-by tags are present with
>>  correct maintainer email addresses
>> * Verify no non-ASCII characters in modified files
>> * Verify no binary files in set of modified files
> 
> Now, Edk2 has few binary files, like logo.bmp. 
> I see one BZ to request logo bmp update. 
> (BZ https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
> So, we need the exception way for it. 
> 
>> * Verify package dependency rules in modified files
>>
>> ## Proposed Pre-Commit Checks in Phase 3
>> * Run ECC on modified files
>> * Verify modified modules/libs build
>> * Run available host based tests (HBFA) against modified
>>  modules/libs
>>
> 
> I request boot test on Emulator and Ovmf in the daily and weekly scope. 
> Daily can cover boot to Shell.
> Weekly can cover more boot functionality. 

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Ni, Ray]

Can someone draw a flow chart of who does what to help clarify?

It sounds like maintainers are going to be the very important bridges between CI system and the patch owners (developers). Important it is I agree but boring it is as well I have to say.

Are maintainers still needed to be picked up/chosen/promoted from developers?

> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Laszlo Ersek
> Sent: Friday, August 30, 2019 5:58 AM
> To: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
> 
> On 08/30/19 10:43, Liming Gao wrote:
> > Mike:
> >   I add my comments.
> >
> >> -----Original Message-----
> >> From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io] On Behalf Of Michael
> >> D Kinney
> >> Sent: Friday, August 30, 2019 4:23 AM
> >> To: devel@edk2.groups.io; rfc@edk2.groups.io
> >> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1
> >>
> >> Hello,
> >>
> >> This is a proposal for a first step towards continuous
> >> integration for all TianoCore repositories to help
> >> improve to quality of commits and automate testing and
> >> release processes for all EDK II packages and platforms.
> >>
> >> This is based on work from a number of EDK II community
> >> members that have provide valuable input and evaluations.
> >>
> >> * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
> >> * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
> >> * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
> >> * Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
> >> * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
> >> * Jiewen Yao <jiewen.yao@intel.com> HBFA
> >>
> >> The following link is a link to an EDK II WIKI page that
> >> contains a summary of the work to date.  Please provide
> >> feedback in the EDK II mailing lists.  The WIKI pages will
> >> be updated with input from the entire EDK II community.
> >>
> >>    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-
> >> Integration
> >>
> >> Proposal
> >> ========
> >> Phase 1 of adding continuous integration is limited to the
> >> edk2 repository.  Additional repositories will be added later.
> >>
> >> The following changes are proposed:
> >> * Remove EDK II Maintainers write access to edk2 repository.
> >>  Only EDK II Administrators will continue to have write
> >>  access, and that should only be used to handle extraordinary
> >>  events.
> >> * EDK II Maintainers use a GitHub Pull Request instead of push
> >>  to commit a patch series to the edk2 repository.  There are
> >>  no other changes to the development and review process.  The
> >>  patch series is prepared in an EDK II maintainer branch with
> >>  all commit message requirements met on each patch in the series.
> >
> > Will the maintainer manually provide pull request after the patch passes the review?
> 
> Yes. The maintainer will prepare a local branch that is rebased to
> master, and has all the mailing list feedback tags (Reviewed-by, etc)
> applied. The maintainer also does all the local testing that they
> usually do, just before the final "git push origin".
> 
> Then, the final "git push origin" action is replaced by:
> (1) git push personal-repo topic-branch-pr
> (2) manual creation of a GitHub.com Pull Request, for the topic branch
> just pushed.
> 
> That is, the final "git push origin" action is replaced with the pushing
> of a personal (ready-made) topic branch, and a GitHub.com Pull Request
> against that branch. The verification and the final merging will be
> performed by github.
> 
> > Can the script scan the mail list and auto trig pull request once the patch gets
> > Reviewed-by or Acked-by from Package maintainers?
> 
> No, it can't. (And, at this stage, it should not.) The coordination
> between submitters, maintainers, reviewers, and occasionally, stewards,
> for determining when a patch series is ready to go, based on review
> discussion, remains the same.
> 
> >> * The edk2 repository only accepts Pull Requests from members
> >>  of the EDK II Maintainers team.  Pull Requests from anyone else
> >>  are rejected.
> >> * Run pre-commit checks using Azure Pipelines
> >
> > The maintainer manually trig pre-commit check or auto trig pre-commit?
> 
> After the maintainer pushes the ready-made branch to their personal
> repo, and submits a PR against that, the PR will set off the checks. If
> the checks pass, the topic branch is merged.
> 
> > By default, pre-commit should be auto trigged based on pull request.
> >
> >> * If all pre-commit checks pass, then the patch series is auto
> >>  committed.  The result of this commit must match the contents
> >>  and commit history that would have occurred using the previous
> >>  push operation.
> >> * If any pre-commit checks fail, then notify the submitter.
> >
> > Will Pre-commit check fail send the mail to the patch submitter?
> > The patch submitter need update the patch and go through review process again.
> 
> My understanding is that, if the testing in GitHub.com fails, the PR
> will be rejected and closed. This will generate a notification email for
> the maintainer that submitted the PR. The maintainer can then return to
> the email thread, and report the CI failure, possibly with a link to the
> failed / rejected PR.
> 
> Then, indeed, the submitter must rework the series and post a new
> version to the list.
> 
> It's also possible (and polite) if the maintainer posts the PR link in
> the mailing list thread right after submitting the PR. Then the
> submitter can monitor the PR too. (Subscribe to it for notifications.)
> As I understand it.
> 
> Furthermore,
> 
> >
> >>  A typical reason for a failure would be a merge conflict with
> >>  another pull request that was just processed.
> >> * Limit pre-commit checks execution time to 10 minutes.
> >> * Provide on-demand builds to EDK II Maintainers that to allow
> >>  EDK II Maintainers to submit a branch through for the same
> >>  set of pre-commit checks without submitting a pull request.
> 
> a maintainer could use this on-demand build & testing facility in the
> course of review, to report findings early.
> 
> Thanks
> Laszlo
> 
> >>
> >> ## Pre-Commit Checks in Phase 1
> >> * Run and pass PatchCheck.py with no errors
> >>
> >> =====================================================
> >>
> >> The following are some additional pre-commit check ideas
> >> that could be quickly added once the initial version using
> >> PatchCheck.py is fully functional.  Please provide feedback
> >> on the ones you like and additional ones you think may
> >> improve the quality of the commits to the edk2 repository.
> >>
> >> ## Proposed Pre-Commit Checks in Phase 2
> >> * Verify Reviewed-by and Acked-by tags are present with
> >>  correct maintainer email addresses
> >> * Verify no non-ASCII characters in modified files
> >> * Verify no binary files in set of modified files
> >
> > Now, Edk2 has few binary files, like logo.bmp.
> > I see one BZ to request logo bmp update.
> > (BZ https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
> > So, we need the exception way for it.
> >
> >> * Verify package dependency rules in modified files
> >>
> >> ## Proposed Pre-Commit Checks in Phase 3
> >> * Run ECC on modified files
> >> * Verify modified modules/libs build
> >> * Run available host based tests (HBFA) against modified
> >>  modules/libs
> >>
> >
> > I request boot test on Emulator and Ovmf in the daily and weekly scope.
> > Daily can cover boot to Shell.
> > Weekly can cover more boot functionality.
> 
> 

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

On 09/03/19 05:39, Ni, Ray wrote:
> Can someone draw a flow chart of who does what to help clarify?

I don't see this as a huge change, relative to the current process.

Before, it's always been up to the subsys maintainer to apply & rebase
the patches locally, pick up the feedback tags, and run at least *some*
build tests before pushing.

After, the final git push is not to "origin" but to a personal branch on
github.com, and then a github PR is needed. If that fails, notify the
submitter. That's all, as far as I can see.

> It sounds like maintainers are going to be the very important bridges between CI system and the patch owners (developers). Important it is I agree but boring it is as well I have to say.

Oh, it *is* absolutely boring.

> Are maintainers still needed to be picked up/chosen/promoted from developers?

Would you trust a person to apply, build-test, and push your UefiCpuPkg
patches, if that person had *zero* UefiCpuPkg development experience?
(Or even zero edk2 development experience?)

Maintainers don't have to be intimately familiar with the subsystem that
they are pushing a given set of patches for, but every bit of knowledge
helps.

So the answer is "technically no, but you'll regret it".

Maintainership is *always* a chore. It always takes away resources from
development activities, for the maintainer.

In most projects, people alternate in a given maintainer role -- they
keep replacing each other. The variance is mostly in how frequent this
alternation is.

In some projects, the role belongs to a group, and people cover the
maintainer responsibilities in very quick succession. In other projects,
people replace each other in maintainer roles when the current
maintainer gets tired of the work, and steps down. Then someone
volunteers to assume the role.

Either way, the new subsys maintainer is almost always a seasoned
developer in the subsystem.

Thanks,
Laszlo

> 
>> -----Original Message-----
>> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Laszlo Ersek
>> Sent: Friday, August 30, 2019 5:58 AM
>> To: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; rfc@edk2.groups.io; Kinney, Michael D
>> <michael.d.kinney@intel.com>
>> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
>>
>> On 08/30/19 10:43, Liming Gao wrote:
>>> Mike:
>>>   I add my comments.
>>>
>>>> -----Original Message-----
>>>> From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io] On Behalf Of Michael
>>>> D Kinney
>>>> Sent: Friday, August 30, 2019 4:23 AM
>>>> To: devel@edk2.groups.io; rfc@edk2.groups.io
>>>> Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1
>>>>
>>>> Hello,
>>>>
>>>> This is a proposal for a first step towards continuous
>>>> integration for all TianoCore repositories to help
>>>> improve to quality of commits and automate testing and
>>>> release processes for all EDK II packages and platforms.
>>>>
>>>> This is based on work from a number of EDK II community
>>>> members that have provide valuable input and evaluations.
>>>>
>>>> * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
>>>> * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
>>>> * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
>>>> * Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
>>>> * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
>>>> * Jiewen Yao <jiewen.yao@intel.com> HBFA
>>>>
>>>> The following link is a link to an EDK II WIKI page that
>>>> contains a summary of the work to date.  Please provide
>>>> feedback in the EDK II mailing lists.  The WIKI pages will
>>>> be updated with input from the entire EDK II community.
>>>>
>>>>    https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-
>>>> Integration
>>>>
>>>> Proposal
>>>> ========
>>>> Phase 1 of adding continuous integration is limited to the
>>>> edk2 repository.  Additional repositories will be added later.
>>>>
>>>> The following changes are proposed:
>>>> * Remove EDK II Maintainers write access to edk2 repository.
>>>>  Only EDK II Administrators will continue to have write
>>>>  access, and that should only be used to handle extraordinary
>>>>  events.
>>>> * EDK II Maintainers use a GitHub Pull Request instead of push
>>>>  to commit a patch series to the edk2 repository.  There are
>>>>  no other changes to the development and review process.  The
>>>>  patch series is prepared in an EDK II maintainer branch with
>>>>  all commit message requirements met on each patch in the series.
>>>
>>> Will the maintainer manually provide pull request after the patch passes the review?
>>
>> Yes. The maintainer will prepare a local branch that is rebased to
>> master, and has all the mailing list feedback tags (Reviewed-by, etc)
>> applied. The maintainer also does all the local testing that they
>> usually do, just before the final "git push origin".
>>
>> Then, the final "git push origin" action is replaced by:
>> (1) git push personal-repo topic-branch-pr
>> (2) manual creation of a GitHub.com Pull Request, for the topic branch
>> just pushed.
>>
>> That is, the final "git push origin" action is replaced with the pushing
>> of a personal (ready-made) topic branch, and a GitHub.com Pull Request
>> against that branch. The verification and the final merging will be
>> performed by github.
>>
>>> Can the script scan the mail list and auto trig pull request once the patch gets
>>> Reviewed-by or Acked-by from Package maintainers?
>>
>> No, it can't. (And, at this stage, it should not.) The coordination
>> between submitters, maintainers, reviewers, and occasionally, stewards,
>> for determining when a patch series is ready to go, based on review
>> discussion, remains the same.
>>
>>>> * The edk2 repository only accepts Pull Requests from members
>>>>  of the EDK II Maintainers team.  Pull Requests from anyone else
>>>>  are rejected.
>>>> * Run pre-commit checks using Azure Pipelines
>>>
>>> The maintainer manually trig pre-commit check or auto trig pre-commit?
>>
>> After the maintainer pushes the ready-made branch to their personal
>> repo, and submits a PR against that, the PR will set off the checks. If
>> the checks pass, the topic branch is merged.
>>
>>> By default, pre-commit should be auto trigged based on pull request.
>>>
>>>> * If all pre-commit checks pass, then the patch series is auto
>>>>  committed.  The result of this commit must match the contents
>>>>  and commit history that would have occurred using the previous
>>>>  push operation.
>>>> * If any pre-commit checks fail, then notify the submitter.
>>>
>>> Will Pre-commit check fail send the mail to the patch submitter?
>>> The patch submitter need update the patch and go through review process again.
>>
>> My understanding is that, if the testing in GitHub.com fails, the PR
>> will be rejected and closed. This will generate a notification email for
>> the maintainer that submitted the PR. The maintainer can then return to
>> the email thread, and report the CI failure, possibly with a link to the
>> failed / rejected PR.
>>
>> Then, indeed, the submitter must rework the series and post a new
>> version to the list.
>>
>> It's also possible (and polite) if the maintainer posts the PR link in
>> the mailing list thread right after submitting the PR. Then the
>> submitter can monitor the PR too. (Subscribe to it for notifications.)
>> As I understand it.
>>
>> Furthermore,
>>
>>>
>>>>  A typical reason for a failure would be a merge conflict with
>>>>  another pull request that was just processed.
>>>> * Limit pre-commit checks execution time to 10 minutes.
>>>> * Provide on-demand builds to EDK II Maintainers that to allow
>>>>  EDK II Maintainers to submit a branch through for the same
>>>>  set of pre-commit checks without submitting a pull request.
>>
>> a maintainer could use this on-demand build & testing facility in the
>> course of review, to report findings early.
>>
>> Thanks
>> Laszlo
>>
>>>>
>>>> ## Pre-Commit Checks in Phase 1
>>>> * Run and pass PatchCheck.py with no errors
>>>>
>>>> =====================================================
>>>>
>>>> The following are some additional pre-commit check ideas
>>>> that could be quickly added once the initial version using
>>>> PatchCheck.py is fully functional.  Please provide feedback
>>>> on the ones you like and additional ones you think may
>>>> improve the quality of the commits to the edk2 repository.
>>>>
>>>> ## Proposed Pre-Commit Checks in Phase 2
>>>> * Verify Reviewed-by and Acked-by tags are present with
>>>>  correct maintainer email addresses
>>>> * Verify no non-ASCII characters in modified files
>>>> * Verify no binary files in set of modified files
>>>
>>> Now, Edk2 has few binary files, like logo.bmp.
>>> I see one BZ to request logo bmp update.
>>> (BZ https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
>>> So, we need the exception way for it.
>>>
>>>> * Verify package dependency rules in modified files
>>>>
>>>> ## Proposed Pre-Commit Checks in Phase 3
>>>> * Run ECC on modified files
>>>> * Verify modified modules/libs build
>>>> * Run available host based tests (HBFA) against modified
>>>>  modules/libs
>>>>
>>>
>>> I request boot test on Emulator and Ovmf in the daily and weekly scope.
>>> Daily can cover boot to Shell.
>>> Weekly can cover more boot functionality.
>>
>> 
> 

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Ni, Ray]

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
> Sent: Tuesday, September 3, 2019 6:20 AM
> To: Ni, Ray <ray.ni@intel.com>; rfc@edk2.groups.io; devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
> 
> On 09/03/19 05:39, Ni, Ray wrote:
> > Can someone draw a flow chart of who does what to help clarify?
> 
> I don't see this as a huge change, relative to the current process.
> 
> Before, it's always been up to the subsys maintainer to apply & rebase
> the patches locally, pick up the feedback tags, and run at least *some*
> build tests before pushing.
> 
> After, the final git push is not to "origin" but to a personal branch on
> github.com, and then a github PR is needed. If that fails, notify the
> submitter. That's all, as far as I can see.
> 
> > It sounds like maintainers are going to be the very important bridges between CI system and the patch owners (developers).
> Important it is I agree but boring it is as well I have to say.
> 
> Oh, it *is* absolutely boring.
> 
> > Are maintainers still needed to be picked up/chosen/promoted from developers?
> 
> Would you trust a person to apply, build-test, and push your UefiCpuPkg
> patches, if that person had *zero* UefiCpuPkg development experience?
> (Or even zero edk2 development experience?)

Can we change the process a bit?
1. maintainers created pull requests on behave of the patch owners
2. patch owners can be notified automatically if pull requests fail
3. patch owners update the pull requests
    (I am not familiar to pull requests. I assume the rights of modifying pull requests and creating pull requests are separated. Are they?)

So, maintainers only need to initiate the pull requests. It assumes when pull requests are initiated, everyone at least agrees the justifications of the changes are valid and the ways the changes are done.

Thanks,
Ray

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

On 09/03/19 18:41, Ni, Ray wrote:
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
>> Sent: Tuesday, September 3, 2019 6:20 AM
>> To: Ni, Ray <ray.ni@intel.com>; rfc@edk2.groups.io; devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Kinney,
>> Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1
>>
>> On 09/03/19 05:39, Ni, Ray wrote:
>>> Can someone draw a flow chart of who does what to help clarify?
>>
>> I don't see this as a huge change, relative to the current process.
>>
>> Before, it's always been up to the subsys maintainer to apply & rebase
>> the patches locally, pick up the feedback tags, and run at least *some*
>> build tests before pushing.
>>
>> After, the final git push is not to "origin" but to a personal branch on
>> github.com, and then a github PR is needed. If that fails, notify the
>> submitter. That's all, as far as I can see.
>>
>>> It sounds like maintainers are going to be the very important bridges between CI system and the patch owners (developers).
>> Important it is I agree but boring it is as well I have to say.
>>
>> Oh, it *is* absolutely boring.
>>
>>> Are maintainers still needed to be picked up/chosen/promoted from developers?
>>
>> Would you trust a person to apply, build-test, and push your UefiCpuPkg
>> patches, if that person had *zero* UefiCpuPkg development experience?
>> (Or even zero edk2 development experience?)
> 
> Can we change the process a bit?
> 1. maintainers created pull requests on behave of the patch owners
> 2. patch owners can be notified automatically if pull requests fail

I believe this can work if:
- the submitter has a github account
- the maintainer knows the submitter's account name
- the maintainer manually subscribes the submitter to the PR
  (I have never tried this myself)

> 3. patch owners update the pull requests
>     (I am not familiar to pull requests. I assume the rights of modifying pull requests and creating pull requests are separated. Are they?)

PRs should not be updated. If there is a CI failure, the PR should be
rejected. If that happens automatically, that's great. If not, then
someone must close the PR manually. If the patch series submitter can do
that, that's great (I have no idea if that works!) If only the PR owner
(= maintainer) can close the PR, then they should close it.

Either way, the patch author will have to submit a v2 to the list. When
that is sufficiently reviewed, the same process starts (new PR opened by
maintainer).

> So, maintainers only need to initiate the pull requests.

I hope this can actually work. We need to test this out first.

> It assumes when pull requests are initiated, everyone at least agrees the justifications of the changes are valid and the ways the changes are done.

I agree fully about this. If the CI check completes, then the changes
are pushed to master automatically! (Only if the push is a fast-forward
-- otherwise the PR is rejected again. GitHub will not be allowed to
merge or rebase withoout supervision.)

Thanks
Laszlo

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Sean]

Laszlo/Mike,

The idea that the maintainer must create the PR is fighting the optimized github PR flow.  Github and PRs process is optimized for letting everyone contribute from "their" fork while still protecting and putting process in place for the "upstream".  

Why not use github to assign maintainers to each package or filepath and then let contributors submit their own PR to edk2 after the mailing list has approved. Then the PR has a policy that someone that is the maintainer of all files changed must approve before the PR can be completed  (or you could even set it so that maintainer must complete PR).   This would have the benefit of less monotonous work for the maintainers and on rejected PRs the contributor could easily update their branch and resubmit their PR.  

Thanks
Sean

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Laszlo Ersek via Groups.Io
Sent: Tuesday, September 3, 2019 9:55 AM
To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io; rfc@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

On 09/03/19 18:41, Ni, Ray wrote:
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo 
>> Ersek
>> Sent: Tuesday, September 3, 2019 6:20 AM
>> To: Ni, Ray <ray.ni@intel.com>; rfc@edk2.groups.io; 
>> devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>; Kinney, 
>> Michael D <michael.d.kinney@intel.com>
>> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous 
>> Integration Phase 1
>>
>> On 09/03/19 05:39, Ni, Ray wrote:
>>> Can someone draw a flow chart of who does what to help clarify?
>>
>> I don't see this as a huge change, relative to the current process.
>>
>> Before, it's always been up to the subsys maintainer to apply & 
>> rebase the patches locally, pick up the feedback tags, and run at 
>> least *some* build tests before pushing.
>>
>> After, the final git push is not to "origin" but to a personal branch 
>> on github.com, and then a github PR is needed. If that fails, notify 
>> the submitter. That's all, as far as I can see.
>>
>>> It sounds like maintainers are going to be the very important bridges between CI system and the patch owners (developers).
>> Important it is I agree but boring it is as well I have to say.
>>
>> Oh, it *is* absolutely boring.
>>
>>> Are maintainers still needed to be picked up/chosen/promoted from developers?
>>
>> Would you trust a person to apply, build-test, and push your 
>> UefiCpuPkg patches, if that person had *zero* UefiCpuPkg development experience?
>> (Or even zero edk2 development experience?)
> 
> Can we change the process a bit?
> 1. maintainers created pull requests on behave of the patch owners 2. 
> patch owners can be notified automatically if pull requests fail

I believe this can work if:
- the submitter has a github account
- the maintainer knows the submitter's account name
- the maintainer manually subscribes the submitter to the PR
  (I have never tried this myself)

> 3. patch owners update the pull requests
>     (I am not familiar to pull requests. I assume the rights of 
> modifying pull requests and creating pull requests are separated. Are 
> they?)

PRs should not be updated. If there is a CI failure, the PR should be rejected. If that happens automatically, that's great. If not, then someone must close the PR manually. If the patch series submitter can do that, that's great (I have no idea if that works!) If only the PR owner (= maintainer) can close the PR, then they should close it.

Either way, the patch author will have to submit a v2 to the list. When that is sufficiently reviewed, the same process starts (new PR opened by maintainer).

> So, maintainers only need to initiate the pull requests.

I hope this can actually work. We need to test this out first.

> It assumes when pull requests are initiated, everyone at least agrees the justifications of the changes are valid and the ways the changes are done.

I agree fully about this. If the CI check completes, then the changes are pushed to master automatically! (Only if the push is a fast-forward
-- otherwise the PR is rejected again. GitHub will not be allowed to merge or rebase withoout supervision.)

Thanks
Laszlo

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Laszlo Ersek]

On 09/03/19 19:09, Sean Brogan wrote:
> Laszlo/Mike,
> 
> The idea that the maintainer must create the PR is fighting the
> optimized github PR flow.  Github and PRs process is optimized for
> letting everyone contribute from "their" fork while still protecting
> and putting process in place for the "upstream".
> 
> Why not use github to assign maintainers to each package or filepath
> and then let contributors submit their own PR to edk2 after the
> mailing list has approved. Then the PR has a policy that someone that
> is the maintainer of all files changed must approve before the PR can
> be completed  (or you could even set it so that maintainer must
> complete PR).   This would have the benefit of less monotonous work
> for the maintainers and on rejected PRs the contributor could easily
> update their branch and resubmit their PR.

I'll let Mike respond.

Thanks
Laszlo

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Sean,

There may be many ways to improve the process and reduce
the work maintainers perform.  So these are ideas we can
explore further going forward.  I will add the concept
of a non-maintainer opening the PR to the Wiki.

In order to get the improvements to code quality from CI
enabled as quickly as possible, I recommend we limit PRs
to maintainers.

Thanks,

Mike

> -----Original Message-----
> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Tuesday, September 3, 2019 10:46 AM
> To: Sean Brogan <sean.brogan@microsoft.com>;
> rfc@edk2.groups.io; Ni, Ray <ray.ni@intel.com>;
> devel@edk2.groups.io; Gao, Liming
> <liming.gao@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II
> Continuous Integration Phase 1
> 
> On 09/03/19 19:09, Sean Brogan wrote:
> > Laszlo/Mike,
> >
> > The idea that the maintainer must create the PR is
> fighting the
> > optimized github PR flow.  Github and PRs process is
> optimized for
> > letting everyone contribute from "their" fork while
> still protecting
> > and putting process in place for the "upstream".
> >
> > Why not use github to assign maintainers to each
> package or filepath
> > and then let contributors submit their own PR to edk2
> after the
> > mailing list has approved. Then the PR has a policy
> that someone that
> > is the maintainer of all files changed must approve
> before the PR can
> > be completed  (or you could even set it so that
> maintainer must
> > complete PR).   This would have the benefit of less
> monotonous work
> > for the maintainers and on rejected PRs the
> contributor could easily
> > update their branch and resubmit their PR.
> 
> I'll let Mike respond.
> 
> Thanks
> Laszlo

Re: [edk2-rfc] [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[rebecca]

On 2019-09-03 10:41, Ni, Ray wrote:
> Can we change the process a bit?
> 1. maintainers created pull requests on behave of the patch owners
> 2. patch owners can be notified automatically if pull requests fail
> 3. patch owners update the pull requests
>     (I am not familiar to pull requests. I assume the rights of modifying pull requests and creating pull requests are separated. Are they?)
>
> So, maintainers only need to initiate the pull requests. It assumes when pull requests are initiated, everyone at least agrees the justifications of the changes are valid and the ways the changes are done.

In other projects I've worked on, patch owners initiate the pre-commit
build/test, and either comment on the review (ideally with link to the
results) about its status, or have the CI system comment automatically
on the review page/thread.

Once the maintainer has signed off on the patch, it can then be sent for
gatekeeping, which is either a manual process whereby the repo owner (or
subsystem maintainer) checks the review, runs any additional tests they
want etc. - or via a 'land' command (for example with Review Board it's
"rbt land") which automates the checks and pushes the changeset.


I understand that with Azure Pipelines and how it integrates with
Github, along with the fact that this project doesn't use pull requests
or any other review automation, means things are probably going to be
pretty different to most other projects for now.


-- 

Rebecca Cran

Re: [RFC] EDK II Continuous Integration Phase 1

[Michael D Kinney]

Responses below.

Mike

> -----Original Message-----
> From: Gao, Liming <liming.gao@intel.com>
> Sent: Friday, August 30, 2019 1:44 AM
> To: rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Subject: RE: [RFC] EDK II Continuous Integration Phase
> 1
> 
> Mike:
>   I add my comments.
> 
> >-----Original Message-----
> >From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io]
> On Behalf Of
> >Michael D Kinney
> >Sent: Friday, August 30, 2019 4:23 AM
> >To: devel@edk2.groups.io; rfc@edk2.groups.io
> >Subject: [edk2-rfc] [RFC] EDK II Continuous
> Integration Phase 1
> >
> >Hello,
> >
> >This is a proposal for a first step towards continuous
> integration for
> >all TianoCore repositories to help improve to quality
> of commits and
> >automate testing and release processes for all EDK II
> packages and
> >platforms.
> >
> >This is based on work from a number of EDK II
> community members that
> >have provide valuable input and evaluations.
> >
> >* Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
> >* Laszlo Ersek <lersek@redhat.com> GitLab evaluation
> >* Philippe Mathieu-Daudé <philmd@redhat.com> GitLab
> evaluation
> >* Sean Brogan <sean.brogan@microsoft.com> Azure
> Pipelines and HBFA
> >* Bret Barkelew <Bret.Barkelew@microsoft.com> Azure
> Pipelines and HBFA
> >* Jiewen Yao <jiewen.yao@intel.com> HBFA
> >
> >The following link is a link to an EDK II WIKI page
> that contains a
> >summary of the work to date.  Please provide feedback
> in the EDK II
> >mailing lists.  The WIKI pages will be updated with
> input from the
> >entire EDK II community.
> >
> >
> >https://github.com/tianocore/tianocore.github.io/wiki/
> EDK-II-Continuous
> >-
> >Integration
> >
> >Proposal
> >========
> >Phase 1 of adding continuous integration is limited to
> the
> >edk2 repository.  Additional repositories will be
> added later.
> >
> >The following changes are proposed:
> >* Remove EDK II Maintainers write access to edk2
> repository.
> >  Only EDK II Administrators will continue to have
> write
> >  access, and that should only be used to handle
> extraordinary
> >  events.
> >* EDK II Maintainers use a GitHub Pull Request instead
> of push
> >  to commit a patch series to the edk2 repository.
> There are
> >  no other changes to the development and review
> process.  The
> >  patch series is prepared in an EDK II maintainer
> branch with
> >  all commit message requirements met on each patch in
> the series.
> 
> Will the maintainer manually provide pull request after
> the patch passes the review?
> Can the script scan the mail list and auto trig pull
> request once the patch gets Reviewed-by or Acked-by
> from Package maintainers?

The maintainer will do this step manually.  After the 
required Reviewed-by and Acked-by responses have been
received, the commit messages in the patch series must
be updated to include the Reviewed-by and Acked-by tags.
I do not see an easy way to automate the editing of the
commit messages and guarantee it is always correct.

> 
> >* The edk2 repository only accepts Pull Requests from
> members
> >  of the EDK II Maintainers team.  Pull Requests from
> anyone else
> >  are rejected.
> >* Run pre-commit checks using Azure Pipelines
> 
> The maintainer manually trig pre-commit check or auto
> trig pre-commit?
> By default, pre-commit should be auto trigged based on
> pull request.

Automatically triggered.

I would be good to provide an on-demand service for maintainers
to run the exact same pre-commit checks on a branch to see
if there are issues that need to be resolved. 

> 
> >* If all pre-commit checks pass, then the patch series
> is auto
> >  committed.  The result of this commit must match the
> contents
> >  and commit history that would have occurred using
> the previous
> >  push operation.
> >* If any pre-commit checks fail, then notify the
> submitter.
> 
> Will Pre-commit check fail send the mail to the patch
> submitter?

Yes.

> The patch submitter need update the patch and go
> through review process again.

Potentially.  If it is a very simple merge conflict with
no functional impacts, then the maintainer should be able
to resolve and resubmit without additional code review.

If there are functional changes to resolve merge conflict,
then another review cycle would be required.

> 
> >  A typical reason for a failure would be a merge
> conflict with
> >  another pull request that was just processed.
> >* Limit pre-commit checks execution time to 10
> minutes.
> >* Provide on-demand builds to EDK II Maintainers that
> to allow
> >  EDK II Maintainers to submit a branch through for
> the same
> >  set of pre-commit checks without submitting a pull
> request.
> >
> >## Pre-Commit Checks in Phase 1
> >* Run and pass PatchCheck.py with no errors
> >
> >=====================================================
> >
> >The following are some additional pre-commit check
> ideas that could be
> >quickly added once the initial version using
> PatchCheck.py is fully
> >functional.  Please provide feedback on the ones you
> like and
> >additional ones you think may improve the quality of
> the commits to the
> >edk2 repository.
> >
> >## Proposed Pre-Commit Checks in Phase 2
> >* Verify Reviewed-by and Acked-by tags are present
> with
> >  correct maintainer email addresses
> >* Verify no non-ASCII characters in modified files
> >* Verify no binary files in set of modified files
> 
> Now, Edk2 has few binary files, like logo.bmp.
> I see one BZ to request logo bmp update.
> (BZ
> https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
> So, we need the exception way for it.
> 
> >* Verify package dependency rules in modified files
> >
> >## Proposed Pre-Commit Checks in Phase 3
> >* Run ECC on modified files
> >* Verify modified modules/libs build
> >* Run available host based tests (HBFA) against
> modified
> >  modules/libs
> >
> 
> I request boot test on Emulator and Ovmf in the daily
> and weekly scope.
> Daily can cover boot to Shell.
> Weekly can cover more boot functionality.
> 
> Thanks
> Liming
> >Best regards,
> >
> >Mike
> >
> >
> >
> >
> >
> >
> >

Re: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1

[rebecca]

On 2019-08-29 14:22, Michael D Kinney wrote:
> This is based on work from a number of EDK II community
> members that have provide valuable input and evaluations.
>
> * Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
> * Laszlo Ersek <lersek@redhat.com> GitLab evaluation
> * Philippe Mathieu-Daudé <philmd@redhat.com> GitLab evaluation
> * Sean Brogan <sean.brogan@microsoft.com> Azure Pipelines and HBFA
> * Bret Barkelew <Bret.Barkelew@microsoft.com> Azure Pipelines and HBFA
> * Jiewen Yao <jiewen.yao@intel.com> HBFA
>
> The following link is a link to an EDK II WIKI page that
> contains a summary of the work to date.  Please provide
> feedback in the EDK II mailing lists.  The WIKI pages will
> be updated with input from the entire EDK II community.
>
>     https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration



Sorry, I was thinking the evaluation I was doing was having the entire
CI run in Jenkins, not only the boot testing. I don't have the array of
hardware you'd want for boot testing, just mostly large x86 systems. 

So I think I'll drop out of the evaluation.


-- 

Rebecca Cran

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

[rebecca]

On 8/29/19 2:22 PM, Michael D Kinney wrote:
> The following link is a link to an EDK II WIKI page that
> contains a summary of the work to date.  Please provide
> feedback in the EDK II mailing lists.  The WIKI pages will
> be updated with input from the entire EDK II community.
>
>      https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration


In that wiki page, it says:

"To work with this branch and run tests immediately, all you need to do is:

|pip install --upgrade -r requirements.txt stuart_setup -c 
.\CISettings.py stuart_update -c .\CISettings.py stuart_ci_build -c 
.\CISettings.py --Tool_Chain VS2017" |


I tried doing that in my KUbuntu system, but it was wanting to run 
NuGet. Once I installed the needed libmono packages via apt, it was 
wanting to install mu-nasm and then mu-Basetools. I thought it would be 
pushing changes to Azure to test, not trying to build it locally?


-- 

Rebecca Cran