[edk2-rfc] GitHub Pull Request based Code Review Process

[edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Hello,

This is a proposal to change from the current email-based code review process to
a GitHub pull request-based code review process for all repositories maintained
in TianoCore.  The current email-based code review process and commit message
requirements are documented in Readme.md or Readme.rst at the root of
repositories along with a few Wiki pages:

* https://github.com/tianocore/edk2/blob/master/ReadMe.rst
* https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Development-Process
* https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers
* https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format
* https://github.com/tianocore/tianocore.github.io/wiki/Commit-Signature-Format

The goal is to post changes by opening a GitHub pull request and perform all
code review activity using the GitHub web interface.  This proposal does not
change any licenses or commit message requirements.  It does require all
developers, maintainers, and reviewers to have GitHub accounts.

One requirement that was collected from previous discussions on this topic is
the need for an email archive of all patches and code review activities.  The
existing GitHub features to produce an email archive were deemed insufficient.
A proof of concept of a GitHub webhook has been implemented to provide the email
archive service.  This email archive is read-only.  You will not be able to send
emails to this archive or reply to emails in the archive.

The sections below provide more details on the proposed GitHub pull request
based code review process, details on the email archive service, and a set of
remaining tasks make the email archive service production quality.  It does not
make sense to support both the existing email-based code review and the GitHub
pull request-based code review at the same time.  Instead, this proposal is to
switch to the GitHub pull request-based code review and retire the email based
code review process on the same date.

The edk2 repository is using GitHub pull requests today to run automated
CI checks on the code changes and allows a maintainer to set the `push` label to
request the changes to be merged if all CI checks pass.  With this proposal,
once the code review is complete and the commit messages have been updated, the
same pull request can be used to perform a final set of CI checks and merge the
changes into the master branch.

I would like to collect feedback on this proposal and the email archive service
over the next two weeks with close of comments on Friday May 22, 2020.  If all
issues and concerns can be addressed, then I would like to see the community
agree to make this change as soon as all remaining tasks are completed.

# TianoCore Repositories to enable

* [edk2](https://github.com/tianocore/edk2)
* [edk2-platforms](https://github.com/tianocore/edk2-platforms)
* [edk2-non-osi](https://github.com/tianocore/edk2-non-osi)
* [edk2-test](https://github.com/tianocore/edk2-test)
* [edk2-libc](https://github.com/tianocore/edk2-libc)
* [edk2-staging](https://github.com/tianocore/edk2-staging)

# GitHub Pull Request Code Review Process

**NOTE**: All steps below use [edk2](https://github.com/tianocore/edk2) as an
example.  Several repositories are supported.

## Author/Developer Steps
  * Create a personal fork of [edk2](https://github.com/tianocore/edk2)

    https://help.github.com/en/github/getting-started-with-github/fork-a-repo

  * Create a new branch from edk2/master in personal fork of edk2 repository.

  * Add set of commits for new feature or bug fix to new branch.  Make sure to
    follow the commit message format requirements.  The only change with this
    RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
    The Cc: lines are still supported, but they should only be used to add
    reviewers that do not have GitHub IDs or are not members of TianoCore.

  * Push branch with new commits to personal fork
  * Create a pull request against TianoCore edk2/master

    https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request

  * If pull request has more than 1 commit, then fill in the pull request title
    and decryption information for Patch #0.  Do not leave defaults.

  * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
    the pull request and each commit in the pull request.

  * If maintainers/reviewers provide feedback that requires changes, then make
    add commits to the current branch with the requested changes.  Once all
    changes are accepted on the current branch, reformulate the patch series and
    commit comments as needed for perform a forced push to the branch in the
    personal fork of the edk2 repository.  This step may be repeated if multiple
    versions of the patch series are required to address all code review
    feedback.

  **OPEN**: How should minimum review period be set?  Labels?

## TianoCore GitHub Email Archive Webhook Service Steps
  * Receive an event that a new pull request was opened
  * Evaluate the files modified by the entire pull request and each commit in
    the pull request and cross references against `Maintainters.txt` in the root
    of the repository to assign maintainers/reviewers to the pull request and
    each commit in the pull request. Individual commit assignments are performed
    by adding a commit comment of the following form:

    [CodeReview] Review-request @mdkinney

  * Generate and sends git patch review emails to the email archive.  Emails
    are also sent to any Cc: tags in the commit messages.

  * If the author/developer performs a forced push to the branch in their
    personal fork of the edk2 repository, then a new set of patch review emails
    with patch series Vx is sent to the email archive and any Cc: tags in commit
    messages.

  * Receive events associated with all code review activities and generate
    and send emails to the email archive that shows all review comments and
    all responses closely matching the email contents seen in the current email
    based code review process.

  * Generate and send email when pull request is merged or closed.

## Maintainer/Reviewer Steps

  * Make sure GitHub configuration is setup to 'Watch' the repositories that
    you have maintainer ship or review responsibilities and that email
    notifications from GitHub are enabled.  This enables email notifications
    when a maintainer/reviewer is assigned to a pull request and individual
    commits.

    https://help.github.com/en/github/managing-subscriptions-and-notifications-on-github/configuring-notifications

  * Subscribe to the email archive associated with the TianoCore GitHub Email
    Archive Webhook Service.

    https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc

  * Review pull requests and commits assigned by the TianoCore GitHub Email
    Archive Webhook Service and use the GitHub web UI to provide all review
    feedback.

    https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-changes-in-pull-requests

  * Wait for Author/Developer to respond to all feedback and add commits with
    code changes as needed to resolve all feedback.  This step may be repeated
    if the developer/author need to produce multiple versions of the patch
    series to address all feedback.

  * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
    responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
    or Series-Tested-by responses to the entire pull request.

  * Wait for Developer/Author to add tags to commit messages in the pull request.

  * Perform final review of patches and commit message tags.  If there are not
    issues, set the `push` label to run final set of CI checks and auto merge
    the pull request into master.

# Maintainers.txt Format Changes

Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
in [].  For example:

    M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]

# TianoCore GitHub Email Archive Webhook Service

Assign reviewers to commits in a GitHub pull request based on assignments
documented in Maintainers.txt and generates an email archive of all pull request
and code review activities.

https://github.com/mdkinney/edk2-email-archive-webhook

# Email Archive Subscription Service

The emails are being delivered to the following RedHat email subscription
service.  Please subscribe to receive the emails and to be able to view the
email archives.

https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc

The email archives are at this link:

https://www.redhat.com/mailman/private/tianocore-code-review-poc/index.html

The following sections show some example pull requests and code reviews to
help review the generated emails, their contents, and threading.

## Email Achieve Thread View

https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289

## Example patch series with 1 patch

https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00340

## Example patch series with < 10 patches

* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00289.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00030.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00018.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00008.html

## Example patch series with > 80 patches

* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00198.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00116.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00035.html

# Tasks to Complete

* Create edk2-codereview repository for evaluation of new code review process.
* Add GitHub IDs to Maintainers.txt in edk2-codereview repository
* Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
  the end of M: and R: statements
* Update webhook to use Rabbit MQ to manage requests and emails
* Determine if webhook requests must be serialized?  Current POC is serialized.
* Make sure webhook has error handling for all unexpected events/states.
* Add logging of all events and emails to webhook
* Add admin interface to webhook
* Deploy webhook on a production server with 24/7 support

# Ideas for Future Enhancements

* Run PatchCheck.py before assigning maintainers/reviewers.
* Add a simple check that fails if a single patch spans more than one package.
* Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
  Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
  Once all commits have required tags, auto update commit messages in the
  branch and wait for maintainer to set the `Push` label to run CI and auto
  merge if all CI checks pass.

Best regards,

Mike

Re: [edk2-rfc] GitHub Pull Request based Code Review Process

[Ni, Ray]

Mike,
It's a huge improvement to me as an Outlook user if pull-request-based review is enabled!

Please help me to understand: The pull-request-based review has been enabled naturally when edk2
was migrated to Github. People don't use it because it's not accepted by community. Your process
tries to meet community's needs of achieving all review comments in mails so pull-request-based
review can be accepted by community. Right?

I just subscribed at https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc with
empty password.
I received the confirmation mail and clicked the link in the mail to confirm.
But I waited for ~15 minutes and didn't receive the additional mail containing the auto-generated password.
I went to https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289.
However, the page requires me to enter password.
Can you please change the setting so that viewing the mail achieve doesn't need password?

Please advise me what else I can try. I am happy to try as many steps as I can before all the process are enabled.

Thanks,
Ray


> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Michael D Kinney
> Sent: Saturday, May 9, 2020 11:00 AM
> To: devel@edk2.groups.io; rfc@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: [edk2-rfc] GitHub Pull Request based Code Review Process
> 
> Hello,
> 
> This is a proposal to change from the current email-based code review process to
> a GitHub pull request-based code review process for all repositories maintained
> in TianoCore.  The current email-based code review process and commit message
> requirements are documented in Readme.md or Readme.rst at the root of
> repositories along with a few Wiki pages:
> 
> * https://github.com/tianocore/edk2/blob/master/ReadMe.rst
> * https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Development-Process
> * https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers
> * https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format
> * https://github.com/tianocore/tianocore.github.io/wiki/Commit-Signature-Format
> 
> The goal is to post changes by opening a GitHub pull request and perform all
> code review activity using the GitHub web interface.  This proposal does not
> change any licenses or commit message requirements.  It does require all
> developers, maintainers, and reviewers to have GitHub accounts.
> 
> One requirement that was collected from previous discussions on this topic is
> the need for an email archive of all patches and code review activities.  The
> existing GitHub features to produce an email archive were deemed insufficient.
> A proof of concept of a GitHub webhook has been implemented to provide the email
> archive service.  This email archive is read-only.  You will not be able to send
> emails to this archive or reply to emails in the archive.
> 
> The sections below provide more details on the proposed GitHub pull request
> based code review process, details on the email archive service, and a set of
> remaining tasks make the email archive service production quality.  It does not
> make sense to support both the existing email-based code review and the GitHub
> pull request-based code review at the same time.  Instead, this proposal is to
> switch to the GitHub pull request-based code review and retire the email based
> code review process on the same date.
> 
> The edk2 repository is using GitHub pull requests today to run automated
> CI checks on the code changes and allows a maintainer to set the `push` label to
> request the changes to be merged if all CI checks pass.  With this proposal,
> once the code review is complete and the commit messages have been updated, the
> same pull request can be used to perform a final set of CI checks and merge the
> changes into the master branch.
> 
> I would like to collect feedback on this proposal and the email archive service
> over the next two weeks with close of comments on Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like to see the community
> agree to make this change as soon as all remaining tasks are completed.
> 
> # TianoCore Repositories to enable
> 
> * [edk2](https://github.com/tianocore/edk2)
> * [edk2-platforms](https://github.com/tianocore/edk2-platforms)
> * [edk2-non-osi](https://github.com/tianocore/edk2-non-osi)
> * [edk2-test](https://github.com/tianocore/edk2-test)
> * [edk2-libc](https://github.com/tianocore/edk2-libc)
> * [edk2-staging](https://github.com/tianocore/edk2-staging)
> 
> # GitHub Pull Request Code Review Process
> 
> **NOTE**: All steps below use [edk2](https://github.com/tianocore/edk2) as an
> example.  Several repositories are supported.
> 
> ## Author/Developer Steps
>   * Create a personal fork of [edk2](https://github.com/tianocore/edk2)
> 
>     https://help.github.com/en/github/getting-started-with-github/fork-a-repo
> 
>   * Create a new branch from edk2/master in personal fork of edk2 repository.
> 
>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>     follow the commit message format requirements.  The only change with this
>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>     The Cc: lines are still supported, but they should only be used to add
>     reviewers that do not have GitHub IDs or are not members of TianoCore.
> 
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
> 
>     https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request
> 
>   * If pull request has more than 1 commit, then fill in the pull request title
>     and decryption information for Patch #0.  Do not leave defaults.
> 
>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>     the pull request and each commit in the pull request.
> 
>   * If maintainers/reviewers provide feedback that requires changes, then make
>     add commits to the current branch with the requested changes.  Once all
>     changes are accepted on the current branch, reformulate the patch series and
>     commit comments as needed for perform a forced push to the branch in the
>     personal fork of the edk2 repository.  This step may be repeated if multiple
>     versions of the patch series are required to address all code review
>     feedback.
> 
>   **OPEN**: How should minimum review period be set?  Labels?
> 
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull request and each commit in
>     the pull request and cross references against `Maintainters.txt` in the root
>     of the repository to assign maintainers/reviewers to the pull request and
>     each commit in the pull request. Individual commit assignments are performed
>     by adding a commit comment of the following form:
> 
>     [CodeReview] Review-request @mdkinney
> 
>   * Generate and sends git patch review emails to the email archive.  Emails
>     are also sent to any Cc: tags in the commit messages.
> 
>   * If the author/developer performs a forced push to the branch in their
>     personal fork of the edk2 repository, then a new set of patch review emails
>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>     messages.
> 
>   * Receive events associated with all code review activities and generate
>     and send emails to the email archive that shows all review comments and
>     all responses closely matching the email contents seen in the current email
>     based code review process.
> 
>   * Generate and send email when pull request is merged or closed.
> 
> ## Maintainer/Reviewer Steps
> 
>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>     you have maintainer ship or review responsibilities and that email
>     notifications from GitHub are enabled.  This enables email notifications
>     when a maintainer/reviewer is assigned to a pull request and individual
>     commits.
> 
>     https://help.github.com/en/github/managing-subscriptions-and-notifications-on-github/configuring-notifications
> 
>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>     Archive Webhook Service.
> 
>     https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc
> 
>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI to provide all review
>     feedback.
> 
>     https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-changes-in-pull-requests
> 
>   * Wait for Author/Developer to respond to all feedback and add commits with
>     code changes as needed to resolve all feedback.  This step may be repeated
>     if the developer/author need to produce multiple versions of the patch
>     series to address all feedback.
> 
>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull request.
> 
>   * Wait for Developer/Author to add tags to commit messages in the pull request.
> 
>   * Perform final review of patches and commit message tags.  If there are not
>     issues, set the `push` label to run final set of CI checks and auto merge
>     the pull request into master.
> 
> # Maintainers.txt Format Changes
> 
> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
> in [].  For example:
> 
>     M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]
> 
> # TianoCore GitHub Email Archive Webhook Service
> 
> Assign reviewers to commits in a GitHub pull request based on assignments
> documented in Maintainers.txt and generates an email archive of all pull request
> and code review activities.
> 
> https://github.com/mdkinney/edk2-email-archive-webhook
> 
> # Email Archive Subscription Service
> 
> The emails are being delivered to the following RedHat email subscription
> service.  Please subscribe to receive the emails and to be able to view the
> email archives.
> 
> https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc
> 
> The email archives are at this link:
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/index.html
> 
> The following sections show some example pull requests and code reviews to
> help review the generated emails, their contents, and threading.
> 
> ## Email Achieve Thread View
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289
> 
> ## Example patch series with 1 patch
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00340
> 
> ## Example patch series with < 10 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00289.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00030.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00018.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00008.html
> 
> ## Example patch series with > 80 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00198.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00116.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00035.html
> 
> # Tasks to Complete
> 
> * Create edk2-codereview repository for evaluation of new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests and emails
> * Determine if webhook requests must be serialized?  Current POC is serialized.
> * Make sure webhook has error handling for all unexpected events/states.
> * Add logging of all events and emails to webhook
> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7 support
> 
> # Ideas for Future Enhancements
> 
> * Run PatchCheck.py before assigning maintainers/reviewers.
> * Add a simple check that fails if a single patch spans more than one package.
> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>   Once all commits have required tags, auto update commit messages in the
>   branch and wait for maintainer to set the `Push` label to run CI and auto
>   merge if all CI checks pass.
> 
> Best regards,
> 
> Mike
> 
> 
> 
> 

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Rebecca Cran]

On 5/8/20 8:59 PM, Michael D Kinney wrote:

>    * Perform final review of patches and commit message tags.  If there are not
>      issues, set the `push` label to run final set of CI checks and auto merge
>      the pull request into master.

What's the difference between the CI that runs when a user submits the 
Pull Request, and the final CI checks that run before the request is merged?

Also, I'm wondering why Mergify is being used instead of the maintainer 
hitting the "Merge Pull Request" button, or however it's worded?


-- 
Rebecca Cran

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Rebecca,

There is no difference between CI checks run during code review
and the final CI checks before merge.  I think it is an interesting
conversation to decide how many times those CI checks should be 
run and if they should run automatically on every change during
review or on demand.

Mergify is more flexible.  We want to make sure the git history
is linear with not git merges and supports both single patches
and patch series without squashing.  GitHub merge button by 
default squashes all commits into a single commit.

Thanks,

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Rebecca Cran
> Sent: Saturday, May 9, 2020 11:25 AM
> To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; rfc@edk2.groups.io
> Subject: Re: [edk2-devel] [edk2-rfc] GitHub Pull
> Request based Code Review Process
> 
> On 5/8/20 8:59 PM, Michael D Kinney wrote:
> 
> >    * Perform final review of patches and commit
> message tags.  If there are not
> >      issues, set the `push` label to run final set of
> CI checks and auto merge
> >      the pull request into master.
> 
> What's the difference between the CI that runs when a
> user submits the
> Pull Request, and the final CI checks that run before
> the request is merged?
> 
> Also, I'm wondering why Mergify is being used instead
> of the maintainer
> hitting the "Merge Pull Request" button, or however
> it's worded?
> 
> 
> --
> Rebecca Cran
> 
> 
> 
> 

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Rebecca Cran]

Mike,

On 5/10/20 3:29 PM, Michael D Kinney wrote:

> There is no difference between CI checks run during code review
> and the final CI checks before merge.  I think it is an interesting
> conversation to decide how many times those CI checks should be
> run and if they should run automatically on every change during
> review or on demand.

I'd suggest following what other Github projects do, which I think is to 
run the CI checks automatically on every change that's made in a pull 
request - I don't know if it might also be necessary to run them during 
the merge, if master has changed in the meantime. That gives the 
_submitter_ feedback about any changes they need to make, instead of 
having to wait until the maintainer tells them their change has broken 
something: it speeds up the development process.

> Mergify is more flexible.  We want to make sure the git history
> is linear with not git merges and supports both single patches
> and patch series without squashing.  GitHub merge button by
> default squashes all commits into a single commit.

Wouldn't disabling all but "Allow rebase merging" do the same thing 
without the additional potential failure point? Though it sounds like 
we've resolved the problems with Mergify, so it's not important.

https://help.github.com/en/github/administering-a-repository/configuring-commit-squashing-for-pull-requests


-- 
Rebecca Cran

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Rebecca,

I agree that the first version should rerun CI checks
on every time commits are added to a PR or there is a 
forced push to the PR.

Perhaps we should use Draft Pull Requests as a way
to indicate the content is not ready for code review
or CI checks yet.

https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests#draft-pull-requests

We also want emails added to the email archive when
the pull request is either abandoned or merged.
merify can add comments to a PR that are picked up
by the webhook.

I agree with reducing the number of services required.
There was feedback from Laszlo related to rebase for
pull requests using the current CI process.  I will
do more investigations of GitHub features, webhook
features, and Mergify features to see if there is 
simpler overall solution.

Mike

> -----Original Message-----
> From: Rebecca Cran <rebecca@bsdio.com>
> Sent: Sunday, May 10, 2020 2:44 PM
> To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; rfc@edk2.groups.io
> Subject: Re: [edk2-devel] [edk2-rfc] GitHub Pull
> Request based Code Review Process
> 
> Mike,
> 
> On 5/10/20 3:29 PM, Michael D Kinney wrote:
> 
> > There is no difference between CI checks run during
> code review
> > and the final CI checks before merge.  I think it is
> an interesting
> > conversation to decide how many times those CI checks
> should be
> > run and if they should run automatically on every
> change during
> > review or on demand.
> 
> I'd suggest following what other Github projects do,
> which I think is to
> run the CI checks automatically on every change that's
> made in a pull
> request - I don't know if it might also be necessary to
> run them during
> the merge, if master has changed in the meantime. That
> gives the
> _submitter_ feedback about any changes they need to
> make, instead of
> having to wait until the maintainer tells them their
> change has broken
> something: it speeds up the development process.
> 
> > Mergify is more flexible.  We want to make sure the
> git history
> > is linear with not git merges and supports both
> single patches
> > and patch series without squashing.  GitHub merge
> button by
> > default squashes all commits into a single commit.
> 
> Wouldn't disabling all but "Allow rebase merging" do
> the same thing
> without the additional potential failure point? Though
> it sounds like
> we've resolved the problems with Mergify, so it's not
> important.
> 
> https://help.github.com/en/github/administering-a-
> repository/configuring-commit-squashing-for-pull-
> requests
> 
> 
> --
> Rebecca Cran
> 

Re: [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Hello,

I have added the following repository to TianoCore to
support the evaluation of the GitHub pull request based
code review process and the email archive webbook.  This
is a copy of tianocore/edk2 repo as of May 10, 2020. 

	https://github.com/tianocore/edk2-codereview

I have updated Maintainers.txt in this repo to add
GitHub IDs for the maintainers and reviewers.  Please
review these updates to make sure they are correct.

    https://github.com/tianocore/edk2-codereview/blob/master/Maintainers.txt

There are a few maintainers and reviewers that I need
GitHub IDs.  Please send me your GitHub IDs and I will
complete the update of Maintainers.txt.

    M: Chao Zhang <chao.b.zhang@intel.com>
    R: Julien Grall <julien@xen.org>
    R: Marc-André Lureau <marcandre.lureau@redhat.com>
    R: Stefan Berger <stefanb@linux.ibm.com>
    R: Liran Alon <liran.alon@oracle.com>
    R: Nikita Leshenko <nikita.leshchenko@oracle.com>

Thanks,

Mike


> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Friday, May 8, 2020 8:00 PM
> To: devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Subject: [edk2-rfc] GitHub Pull Request based Code
> Review Process
> 
> Hello,
> 
> This is a proposal to change from the current email-
> based code review process to
> a GitHub pull request-based code review process for all
> repositories maintained
> in TianoCore.  The current email-based code review
> process and commit message
> requirements are documented in Readme.md or Readme.rst
> at the root of
> repositories along with a few Wiki pages:
> 
> *
> https://github.com/tianocore/edk2/blob/master/ReadMe.rs
> t
> *
> https://github.com/tianocore/tianocore.github.io/wiki/E
> DK-II-Development-Process
> *
> https://github.com/tianocore/tianocore.github.io/wiki/L
> aszlo's-unkempt-git-guide-for-edk2-contributors-and-
> maintainers
> *
> https://github.com/tianocore/tianocore.github.io/wiki/C
> ommit-Message-Format
> *
> https://github.com/tianocore/tianocore.github.io/wiki/C
> ommit-Signature-Format
> 
> The goal is to post changes by opening a GitHub pull
> request and perform all
> code review activity using the GitHub web interface.
> This proposal does not
> change any licenses or commit message requirements.  It
> does require all
> developers, maintainers, and reviewers to have GitHub
> accounts.
> 
> One requirement that was collected from previous
> discussions on this topic is
> the need for an email archive of all patches and code
> review activities.  The
> existing GitHub features to produce an email archive
> were deemed insufficient.
> A proof of concept of a GitHub webhook has been
> implemented to provide the email
> archive service.  This email archive is read-only.  You
> will not be able to send
> emails to this archive or reply to emails in the
> archive.
> 
> The sections below provide more details on the proposed
> GitHub pull request
> based code review process, details on the email archive
> service, and a set of
> remaining tasks make the email archive service
> production quality.  It does not
> make sense to support both the existing email-based
> code review and the GitHub
> pull request-based code review at the same time.
> Instead, this proposal is to
> switch to the GitHub pull request-based code review and
> retire the email based
> code review process on the same date.
> 
> The edk2 repository is using GitHub pull requests today
> to run automated
> CI checks on the code changes and allows a maintainer
> to set the `push` label to
> request the changes to be merged if all CI checks pass.
> With this proposal,
> once the code review is complete and the commit
> messages have been updated, the
> same pull request can be used to perform a final set of
> CI checks and merge the
> changes into the master branch.
> 
> I would like to collect feedback on this proposal and
> the email archive service
> over the next two weeks with close of comments on
> Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like
> to see the community
> agree to make this change as soon as all remaining
> tasks are completed.
> 
> # TianoCore Repositories to enable
> 
> * [edk2](https://github.com/tianocore/edk2)
> * [edk2-platforms](https://github.com/tianocore/edk2-
> platforms)
> * [edk2-non-osi](https://github.com/tianocore/edk2-non-
> osi)
> * [edk2-test](https://github.com/tianocore/edk2-test)
> * [edk2-libc](https://github.com/tianocore/edk2-libc)
> * [edk2-staging](https://github.com/tianocore/edk2-
> staging)
> 
> # GitHub Pull Request Code Review Process
> 
> **NOTE**: All steps below use
> [edk2](https://github.com/tianocore/edk2) as an
> example.  Several repositories are supported.
> 
> ## Author/Developer Steps
>   * Create a personal fork of
> [edk2](https://github.com/tianocore/edk2)
> 
>     https://help.github.com/en/github/getting-started-
> with-github/fork-a-repo
> 
>   * Create a new branch from edk2/master in personal
> fork of edk2 repository.
> 
>   * Add set of commits for new feature or bug fix to
> new branch.  Make sure to
>     follow the commit message format requirements.  The
> only change with this
>     RFC is that the Cc: lines to maintainers/reviewers
> should **not** be added.
>     The Cc: lines are still supported, but they should
> only be used to add
>     reviewers that do not have GitHub IDs or are not
> members of TianoCore.
> 
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
> 
>     https://help.github.com/en/github/collaborating-
> with-issues-and-pull-requests/creating-a-pull-request
> 
>   * If pull request has more than 1 commit, then fill
> in the pull request title
>     and decryption information for Patch #0.  Do not
> leave defaults.
> 
>   * Do not assign reviewers.  The webhook assigns
> maintainers and reviewers to
>     the pull request and each commit in the pull
> request.
> 
>   * If maintainers/reviewers provide feedback that
> requires changes, then make
>     add commits to the current branch with the
> requested changes.  Once all
>     changes are accepted on the current branch,
> reformulate the patch series and
>     commit comments as needed for perform a forced push
> to the branch in the
>     personal fork of the edk2 repository.  This step
> may be repeated if multiple
>     versions of the patch series are required to
> address all code review
>     feedback.
> 
>   **OPEN**: How should minimum review period be set?
> Labels?
> 
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull
> request and each commit in
>     the pull request and cross references against
> `Maintainters.txt` in the root
>     of the repository to assign maintainers/reviewers
> to the pull request and
>     each commit in the pull request. Individual commit
> assignments are performed
>     by adding a commit comment of the following form:
> 
>     [CodeReview] Review-request @mdkinney
> 
>   * Generate and sends git patch review emails to the
> email archive.  Emails
>     are also sent to any Cc: tags in the commit
> messages.
> 
>   * If the author/developer performs a forced push to
> the branch in their
>     personal fork of the edk2 repository, then a new
> set of patch review emails
>     with patch series Vx is sent to the email archive
> and any Cc: tags in commit
>     messages.
> 
>   * Receive events associated with all code review
> activities and generate
>     and send emails to the email archive that shows all
> review comments and
>     all responses closely matching the email contents
> seen in the current email
>     based code review process.
> 
>   * Generate and send email when pull request is merged
> or closed.
> 
> ## Maintainer/Reviewer Steps
> 
>   * Make sure GitHub configuration is setup to 'Watch'
> the repositories that
>     you have maintainer ship or review responsibilities
> and that email
>     notifications from GitHub are enabled.  This
> enables email notifications
>     when a maintainer/reviewer is assigned to a pull
> request and individual
>     commits.
> 
>     https://help.github.com/en/github/managing-
> subscriptions-and-notifications-on-github/configuring-
> notifications
> 
>   * Subscribe to the email archive associated with the
> TianoCore GitHub Email
>     Archive Webhook Service.
> 
>     https://www.redhat.com/mailman/listinfo/tianocore-
> code-review-poc
> 
>   * Review pull requests and commits assigned by the
> TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI
> to provide all review
>     feedback.
> 
>     https://help.github.com/en/github/collaborating-
> with-issues-and-pull-requests/reviewing-changes-in-
> pull-requests
> 
>   * Wait for Author/Developer to respond to all
> feedback and add commits with
>     code changes as needed to resolve all feedback.
> This step may be repeated
>     if the developer/author need to produce multiple
> versions of the patch
>     series to address all feedback.
> 
>   * Once all feedback is addressed, add Reviewed-by,
> Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-
> reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull
> request.
> 
>   * Wait for Developer/Author to add tags to commit
> messages in the pull request.
> 
>   * Perform final review of patches and commit message
> tags.  If there are not
>     issues, set the `push` label to run final set of CI
> checks and auto merge
>     the pull request into master.
> 
> # Maintainers.txt Format Changes
> 
> Add GitHub IDs of all maintainers and reviewers at the
> end of M: and R: lines
> in [].  For example:
> 
>     M: Michael D Kinney <michael.d.kinney@intel.com>
> [mdkinney]
> 
> # TianoCore GitHub Email Archive Webhook Service
> 
> Assign reviewers to commits in a GitHub pull request
> based on assignments
> documented in Maintainers.txt and generates an email
> archive of all pull request
> and code review activities.
> 
> https://github.com/mdkinney/edk2-email-archive-webhook
> 
> # Email Archive Subscription Service
> 
> The emails are being delivered to the following RedHat
> email subscription
> service.  Please subscribe to receive the emails and to
> be able to view the
> email archives.
> 
> https://www.redhat.com/mailman/listinfo/tianocore-code-
> review-poc
> 
> The email archives are at this link:
> 
> https://www.redhat.com/mailman/private/tianocore-code-
> review-poc/index.html
> 
> The following sections show some example pull requests
> and code reviews to
> help review the generated emails, their contents, and
> threading.
> 
> ## Email Achieve Thread View
> 
> https://www.redhat.com/mailman/private/tianocore-code-
> review-poc/2020-May/thread.html#00289
> 
> ## Example patch series with 1 patch
> 
> https://www.redhat.com/mailman/private/tianocore-code-
> review-poc/2020-May/thread.html#00340
> 
> ## Example patch series with < 10 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00289.html
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00030.html
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00018.html
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00008.html
> 
> ## Example patch series with > 80 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00198.html
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00116.html
> * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00035.html
> 
> # Tasks to Complete
> 
> * Create edk2-codereview repository for evaluation of
> new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview
> repository
> * Update BaseTools/Scripts/GetMaintainer.py to be
> compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests
> and emails
> * Determine if webhook requests must be serialized?
> Current POC is serialized.
> * Make sure webhook has error handling for all
> unexpected events/states.
> * Add logging of all events and emails to webhook
> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7
> support
> 
> # Ideas for Future Enhancements
> 
> * Run PatchCheck.py before assigning
> maintainers/reviewers.
> * Add a simple check that fails if a single patch spans
> more than one package.
> * Monitor comments for Reviewed-by, Acked-by, Tested-
> by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned
> maintainers/reviewers.
>   Once all commits have required tags, auto update
> commit messages in the
>   branch and wait for maintainer to set the `Push`
> label to run CI and auto
>   merge if all CI checks pass.
> 
> Best regards,
> 
> Mike
> 

Re: [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Hi Ray,

Comments below.

Mike

> -----Original Message-----
> From: Ni, Ray <ray.ni@intel.com>
> Sent: Friday, May 8, 2020 9:23 PM
> To: rfc@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Subject: RE: [edk2-rfc] GitHub Pull Request based Code
> Review Process
> 
> Mike,
> It's a huge improvement to me as an Outlook user if
> pull-request-based review is enabled!
> 
> Please help me to understand: The pull-request-based
> review has been enabled naturally when edk2
> was migrated to Github. People don't use it because
> it's not accepted by community. Your process
> tries to meet community's needs of achieving all review
> comments in mails so pull-request-based
> review can be accepted by community. Right?

Yes.

> 
> I just subscribed at
> https://www.redhat.com/mailman/listinfo/tianocore-code-
> review-poc with
> empty password.
> I received the confirmation mail and clicked the link
> in the mail to confirm.
> But I waited for ~15 minutes and didn't receive the
> additional mail containing the auto-generated password.
> I went to
> https://www.redhat.com/mailman/private/tianocore-code-
> review-poc/2020-May/thread.html#00289.
> However, the page requires me to enter password.
> Can you please change the setting so that viewing the
> mail achieve doesn't need password?

Laszlo has graciously volunteered to help setup this email
subscription service to help evaluate the POC.  I will let 
him comment on the settings available.  He should also be
able to approve your subscription so you can see the archive
and receive any emails generated by the webhook.

> 
> Please advise me what else I can try. I am happy to try
> as many steps as I can before all the process are
> enabled.
> 
> Thanks,
> Ray
> 
> 
> > -----Original Message-----
> > From: rfc@edk2.groups.io <rfc@edk2.groups.io> On
> Behalf Of Michael D Kinney
> > Sent: Saturday, May 9, 2020 11:00 AM
> > To: devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
> Michael D <michael.d.kinney@intel.com>
> > Subject: [edk2-rfc] GitHub Pull Request based Code
> Review Process
> >
> > Hello,
> >
> > This is a proposal to change from the current email-
> based code review process to
> > a GitHub pull request-based code review process for
> all repositories maintained
> > in TianoCore.  The current email-based code review
> process and commit message
> > requirements are documented in Readme.md or
> Readme.rst at the root of
> > repositories along with a few Wiki pages:
> >
> > *
> https://github.com/tianocore/edk2/blob/master/ReadMe.rs
> t
> > *
> https://github.com/tianocore/tianocore.github.io/wiki/E
> DK-II-Development-Process
> > *
> https://github.com/tianocore/tianocore.github.io/wiki/L
> aszlo's-unkempt-git-guide-for-edk2-contributors-and-
> maintainers
> > *
> https://github.com/tianocore/tianocore.github.io/wiki/C
> ommit-Message-Format
> > *
> https://github.com/tianocore/tianocore.github.io/wiki/C
> ommit-Signature-Format
> >
> > The goal is to post changes by opening a GitHub pull
> request and perform all
> > code review activity using the GitHub web interface.
> This proposal does not
> > change any licenses or commit message requirements.
> It does require all
> > developers, maintainers, and reviewers to have GitHub
> accounts.
> >
> > One requirement that was collected from previous
> discussions on this topic is
> > the need for an email archive of all patches and code
> review activities.  The
> > existing GitHub features to produce an email archive
> were deemed insufficient.
> > A proof of concept of a GitHub webhook has been
> implemented to provide the email
> > archive service.  This email archive is read-only.
> You will not be able to send
> > emails to this archive or reply to emails in the
> archive.
> >
> > The sections below provide more details on the
> proposed GitHub pull request
> > based code review process, details on the email
> archive service, and a set of
> > remaining tasks make the email archive service
> production quality.  It does not
> > make sense to support both the existing email-based
> code review and the GitHub
> > pull request-based code review at the same time.
> Instead, this proposal is to
> > switch to the GitHub pull request-based code review
> and retire the email based
> > code review process on the same date.
> >
> > The edk2 repository is using GitHub pull requests
> today to run automated
> > CI checks on the code changes and allows a maintainer
> to set the `push` label to
> > request the changes to be merged if all CI checks
> pass.  With this proposal,
> > once the code review is complete and the commit
> messages have been updated, the
> > same pull request can be used to perform a final set
> of CI checks and merge the
> > changes into the master branch.
> >
> > I would like to collect feedback on this proposal and
> the email archive service
> > over the next two weeks with close of comments on
> Friday May 22, 2020.  If all
> > issues and concerns can be addressed, then I would
> like to see the community
> > agree to make this change as soon as all remaining
> tasks are completed.
> >
> > # TianoCore Repositories to enable
> >
> > * [edk2](https://github.com/tianocore/edk2)
> > * [edk2-platforms](https://github.com/tianocore/edk2-
> platforms)
> > * [edk2-non-osi](https://github.com/tianocore/edk2-
> non-osi)
> > * [edk2-test](https://github.com/tianocore/edk2-test)
> > * [edk2-libc](https://github.com/tianocore/edk2-libc)
> > * [edk2-staging](https://github.com/tianocore/edk2-
> staging)
> >
> > # GitHub Pull Request Code Review Process
> >
> > **NOTE**: All steps below use
> [edk2](https://github.com/tianocore/edk2) as an
> > example.  Several repositories are supported.
> >
> > ## Author/Developer Steps
> >   * Create a personal fork of
> [edk2](https://github.com/tianocore/edk2)
> >
> >     https://help.github.com/en/github/getting-
> started-with-github/fork-a-repo
> >
> >   * Create a new branch from edk2/master in personal
> fork of edk2 repository.
> >
> >   * Add set of commits for new feature or bug fix to
> new branch.  Make sure to
> >     follow the commit message format requirements.
> The only change with this
> >     RFC is that the Cc: lines to
> maintainers/reviewers should **not** be added.
> >     The Cc: lines are still supported, but they
> should only be used to add
> >     reviewers that do not have GitHub IDs or are not
> members of TianoCore.
> >
> >   * Push branch with new commits to personal fork
> >   * Create a pull request against TianoCore
> edk2/master
> >
> >     https://help.github.com/en/github/collaborating-
> with-issues-and-pull-requests/creating-a-pull-request
> >
> >   * If pull request has more than 1 commit, then fill
> in the pull request title
> >     and decryption information for Patch #0.  Do not
> leave defaults.
> >
> >   * Do not assign reviewers.  The webhook assigns
> maintainers and reviewers to
> >     the pull request and each commit in the pull
> request.
> >
> >   * If maintainers/reviewers provide feedback that
> requires changes, then make
> >     add commits to the current branch with the
> requested changes.  Once all
> >     changes are accepted on the current branch,
> reformulate the patch series and
> >     commit comments as needed for perform a forced
> push to the branch in the
> >     personal fork of the edk2 repository.  This step
> may be repeated if multiple
> >     versions of the patch series are required to
> address all code review
> >     feedback.
> >
> >   **OPEN**: How should minimum review period be set?
> Labels?
> >
> > ## TianoCore GitHub Email Archive Webhook Service
> Steps
> >   * Receive an event that a new pull request was
> opened
> >   * Evaluate the files modified by the entire pull
> request and each commit in
> >     the pull request and cross references against
> `Maintainters.txt` in the root
> >     of the repository to assign maintainers/reviewers
> to the pull request and
> >     each commit in the pull request. Individual
> commit assignments are performed
> >     by adding a commit comment of the following form:
> >
> >     [CodeReview] Review-request @mdkinney
> >
> >   * Generate and sends git patch review emails to the
> email archive.  Emails
> >     are also sent to any Cc: tags in the commit
> messages.
> >
> >   * If the author/developer performs a forced push to
> the branch in their
> >     personal fork of the edk2 repository, then a new
> set of patch review emails
> >     with patch series Vx is sent to the email archive
> and any Cc: tags in commit
> >     messages.
> >
> >   * Receive events associated with all code review
> activities and generate
> >     and send emails to the email archive that shows
> all review comments and
> >     all responses closely matching the email contents
> seen in the current email
> >     based code review process.
> >
> >   * Generate and send email when pull request is
> merged or closed.
> >
> > ## Maintainer/Reviewer Steps
> >
> >   * Make sure GitHub configuration is setup to
> 'Watch' the repositories that
> >     you have maintainer ship or review
> responsibilities and that email
> >     notifications from GitHub are enabled.  This
> enables email notifications
> >     when a maintainer/reviewer is assigned to a pull
> request and individual
> >     commits.
> >
> >     https://help.github.com/en/github/managing-
> subscriptions-and-notifications-on-github/configuring-
> notifications
> >
> >   * Subscribe to the email archive associated with
> the TianoCore GitHub Email
> >     Archive Webhook Service.
> >
> >
> https://www.redhat.com/mailman/listinfo/tianocore-code-
> review-poc
> >
> >   * Review pull requests and commits assigned by the
> TianoCore GitHub Email
> >     Archive Webhook Service and use the GitHub web UI
> to provide all review
> >     feedback.
> >
> >     https://help.github.com/en/github/collaborating-
> with-issues-and-pull-requests/reviewing-changes-in-
> pull-requests
> >
> >   * Wait for Author/Developer to respond to all
> feedback and add commits with
> >     code changes as needed to resolve all feedback.
> This step may be repeated
> >     if the developer/author need to produce multiple
> versions of the patch
> >     series to address all feedback.
> >
> >   * Once all feedback is addressed, add Reviewed-by,
> Acked-by, and Tested-by
> >     responses on individual commits.  Or add Series-
> reviewed-by, Series-acked-by,
> >     or Series-Tested-by responses to the entire pull
> request.
> >
> >   * Wait for Developer/Author to add tags to commit
> messages in the pull request.
> >
> >   * Perform final review of patches and commit
> message tags.  If there are not
> >     issues, set the `push` label to run final set of
> CI checks and auto merge
> >     the pull request into master.
> >
> > # Maintainers.txt Format Changes
> >
> > Add GitHub IDs of all maintainers and reviewers at
> the end of M: and R: lines
> > in [].  For example:
> >
> >     M: Michael D Kinney <michael.d.kinney@intel.com>
> [mdkinney]
> >
> > # TianoCore GitHub Email Archive Webhook Service
> >
> > Assign reviewers to commits in a GitHub pull request
> based on assignments
> > documented in Maintainers.txt and generates an email
> archive of all pull request
> > and code review activities.
> >
> > https://github.com/mdkinney/edk2-email-archive-
> webhook
> >
> > # Email Archive Subscription Service
> >
> > The emails are being delivered to the following
> RedHat email subscription
> > service.  Please subscribe to receive the emails and
> to be able to view the
> > email archives.
> >
> > https://www.redhat.com/mailman/listinfo/tianocore-
> code-review-poc
> >
> > The email archives are at this link:
> >
> > https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/index.html
> >
> > The following sections show some example pull
> requests and code reviews to
> > help review the generated emails, their contents, and
> threading.
> >
> > ## Email Achieve Thread View
> >
> > https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/thread.html#00289
> >
> > ## Example patch series with 1 patch
> >
> > https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/thread.html#00340
> >
> > ## Example patch series with < 10 patches
> >
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00289.html
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00030.html
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00018.html
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00008.html
> >
> > ## Example patch series with > 80 patches
> >
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00198.html
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00116.html
> > * https://www.redhat.com/mailman/private/tianocore-
> code-review-poc/2020-May/msg00035.html
> >
> > # Tasks to Complete
> >
> > * Create edk2-codereview repository for evaluation of
> new code review process.
> > * Add GitHub IDs to Maintainers.txt in edk2-
> codereview repository
> > * Update BaseTools/Scripts/GetMaintainer.py to be
> compatible with GitHub IDs at
> >   the end of M: and R: statements
> > * Update webhook to use Rabbit MQ to manage requests
> and emails
> > * Determine if webhook requests must be serialized?
> Current POC is serialized.
> > * Make sure webhook has error handling for all
> unexpected events/states.
> > * Add logging of all events and emails to webhook
> > * Add admin interface to webhook
> > * Deploy webhook on a production server with 24/7
> support
> >
> > # Ideas for Future Enhancements
> >
> > * Run PatchCheck.py before assigning
> maintainers/reviewers.
> > * Add a simple check that fails if a single patch
> spans more than one package.
> > * Monitor comments for Reviewed-by, Acked-by, Tested-
> by, Series-Reviewed-by,
> >   Series-Acked-by, Series-Tested-by made by assigned
> maintainers/reviewers.
> >   Once all commits have required tags, auto update
> commit messages in the
> >   branch and wait for maintainer to set the `Push`
> label to run CI and auto
> >   merge if all CI checks pass.
> >
> > Best regards,
> >
> > Mike
> >
> >
> >
> > 
> 

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/09/20 04:59, Michael D Kinney wrote:
> Hello,
> 
> This is a proposal to change from the current email-based code review process to
> a GitHub pull request-based code review process for all repositories maintained
> in TianoCore.  The current email-based code review process and commit message
> requirements are documented in Readme.md or Readme.rst at the root of
> repositories along with a few Wiki pages:
> 
> * https://github.com/tianocore/edk2/blob/master/ReadMe.rst
> * https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Development-Process
> * https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers
> * https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format
> * https://github.com/tianocore/tianocore.github.io/wiki/Commit-Signature-Format
> 
> The goal is to post changes by opening a GitHub pull request and perform all
> code review activity using the GitHub web interface.  This proposal does not
> change any licenses or commit message requirements.  It does require all
> developers, maintainers, and reviewers to have GitHub accounts.
> 
> One requirement that was collected from previous discussions on this topic is
> the need for an email archive of all patches and code review activities.  The
> existing GitHub features to produce an email archive were deemed insufficient.
> A proof of concept of a GitHub webhook has been implemented to provide the email
> archive service.  This email archive is read-only.  You will not be able to send
> emails to this archive or reply to emails in the archive.
> 
> The sections below provide more details on the proposed GitHub pull request
> based code review process, details on the email archive service, and a set of
> remaining tasks make the email archive service production quality.  It does not
> make sense to support both the existing email-based code review and the GitHub
> pull request-based code review at the same time.  Instead, this proposal is to
> switch to the GitHub pull request-based code review and retire the email based
> code review process on the same date.
> 
> The edk2 repository is using GitHub pull requests today to run automated
> CI checks on the code changes and allows a maintainer to set the `push` label to
> request the changes to be merged if all CI checks pass.  With this proposal,
> once the code review is complete and the commit messages have been updated, the
> same pull request can be used to perform a final set of CI checks and merge the
> changes into the master branch.
> 
> I would like to collect feedback on this proposal and the email archive service
> over the next two weeks with close of comments on Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like to see the community
> agree to make this change as soon as all remaining tasks are completed.
> 
> # TianoCore Repositories to enable
> 
> * [edk2](https://github.com/tianocore/edk2)
> * [edk2-platforms](https://github.com/tianocore/edk2-platforms)
> * [edk2-non-osi](https://github.com/tianocore/edk2-non-osi)
> * [edk2-test](https://github.com/tianocore/edk2-test)
> * [edk2-libc](https://github.com/tianocore/edk2-libc)
> * [edk2-staging](https://github.com/tianocore/edk2-staging)
> 
> # GitHub Pull Request Code Review Process
> 
> **NOTE**: All steps below use [edk2](https://github.com/tianocore/edk2) as an
> example.  Several repositories are supported.
> 
> ## Author/Developer Steps
>   * Create a personal fork of [edk2](https://github.com/tianocore/edk2)
> 
>     https://help.github.com/en/github/getting-started-with-github/fork-a-repo
> 
>   * Create a new branch from edk2/master in personal fork of edk2 repository.
> 
>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>     follow the commit message format requirements.  The only change with this
>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>     The Cc: lines are still supported, but they should only be used to add
>     reviewers that do not have GitHub IDs or are not members of TianoCore.
> 
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
> 
>     https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request
> 
>   * If pull request has more than 1 commit, then fill in the pull request title
>     and decryption information for Patch #0.  Do not leave defaults.

s/decryption/description/

(Because I'm assuming this will turn into a wiki article at some point.)

> 
>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>     the pull request and each commit in the pull request.
> 
>   * If maintainers/reviewers provide feedback that requires changes, then make
>     add commits to the current branch with the requested changes.  Once all

s/make add/add/

>     changes are accepted on the current branch, reformulate the patch series and
>     commit comments as needed for perform a forced push to the branch in the
>     personal fork of the edk2 repository.  This step may be repeated if multiple
>     versions of the patch series are required to address all code review
>     feedback.

Do I understand correctly that this recommends the contributor first
push incremental patches on top of the series, then do a rebase
(squashing updates as necessary) and finally do a force-push, for the
next round of review?

To me as a reviewer, that's extra work. I'm used to locally comparing
the v(n) patch set to v(n+1) with git-range-diff, and/or with some
personal scripts. I wouldn't encourage incremental patches appended --
even temporarily -- to the branch, because (a) it's extra review work
(it requires me to review something that has zero chance to get into the
git history as-is), and (b) it superficially resembles the
github.com-specific bad practice called "squash on merge", and (c) it
runs the risk that the maintainer responsible for ultimately merging the
series ends up actually merging the incremental (= "fixup") patches in
isolation (without squashing them).

> 
>   **OPEN**: How should minimum review period be set?  Labels?

Not sure about the best tooling. My recommendation would be to require
reviewers to start providing their feedback within one week.

One thing that I find important is that a maintainer can signal "I got
your work in my queue, but I may need more time". And a special case of
that are automated out-of-office responses. I think they are very
helpful (when a contributor feels they are bottlenecked on review), but
I'm not sure how one can configure that via github. I certainly would
not share my out-of-office times with github. (I set the start/end dates
in my email infrastructure, at the moment, but the out-of-office
messages it sends do not contain the dates either, on purpose.)

> 
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull request and each commit in
>     the pull request and cross references against `Maintainters.txt` in the root

s/cross references/cross reference them/ ?

>     of the repository to assign maintainers/reviewers to the pull request and
>     each commit in the pull request. Individual commit assignments are performed
>     by adding a commit comment of the following form:
> 
>     [CodeReview] Review-request @mdkinney
> 
>   * Generate and sends git patch review emails to the email archive.  Emails

s/sends/send/

>     are also sent to any Cc: tags in the commit messages.
> 
>   * If the author/developer performs a forced push to the branch in their
>     personal fork of the edk2 repository, then a new set of patch review emails
>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>     messages.
> 
>   * Receive events associated with all code review activities and generate
>     and send emails to the email archive that shows all review comments and
>     all responses closely matching the email contents seen in the current email
>     based code review process.
> 
>   * Generate and send email when pull request is merged or closed.
> 
> ## Maintainer/Reviewer Steps
> 
>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>     you have maintainer ship or review responsibilities and that email

s/maintainer ship/maintainership/

>     notifications from GitHub are enabled.  This enables email notifications
>     when a maintainer/reviewer is assigned to a pull request and individual
>     commits.
> 
>     https://help.github.com/en/github/managing-subscriptions-and-notifications-on-github/configuring-notifications
> 
>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>     Archive Webhook Service.
> 
>     https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc

Important: as the name says ("-poc"), this is a Proof of Concept list,
for now. Once we're ready to switch over, I'll file an internal ticket
at RH to either rename the list, or (which is probably better) to create
a new list (no "-poc" suffix).

The second option seems more useful because then the webhook development
/ bugfixing (if any) could perhaps occur in parallel to the normal edk2
workflow.

> 
>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI to provide all review
>     feedback.
> 
>     https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-changes-in-pull-requests
> 
>   * Wait for Author/Developer to respond to all feedback and add commits with
>     code changes as needed to resolve all feedback.  This step may be repeated
>     if the developer/author need to produce multiple versions of the patch
>     series to address all feedback.

(same question about the incremental fixup patches as above)

> 
>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull request.
> 
>   * Wait for Developer/Author to add tags to commit messages in the pull request.
> 
>   * Perform final review of patches and commit message tags.  If there are not
>     issues, set the `push` label to run final set of CI checks and auto merge
>     the pull request into master.
> 
> # Maintainers.txt Format Changes
> 
> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
> in [].  For example:
> 
>     M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]
> 
> # TianoCore GitHub Email Archive Webhook Service
> 
> Assign reviewers to commits in a GitHub pull request based on assignments
> documented in Maintainers.txt and generates an email archive of all pull request
> and code review activities.

s/generates/generate/

(or s/Assign/Assigns/)

> 
> https://github.com/mdkinney/edk2-email-archive-webhook
> 
> # Email Archive Subscription Service
> 
> The emails are being delivered to the following RedHat email subscription
> service.  Please subscribe to receive the emails and to be able to view the
> email archives.
> 
> https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc
> 
> The email archives are at this link:
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/index.html
> 
> The following sections show some example pull requests and code reviews to
> help review the generated emails, their contents, and threading.
> 
> ## Email Achieve Thread View
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289
> 
> ## Example patch series with 1 patch
> 
> https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00340
> 
> ## Example patch series with < 10 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00289.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00030.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00018.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00008.html
> 
> ## Example patch series with > 80 patches
> 
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00198.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00116.html
> * https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00035.html
> 
> # Tasks to Complete
> 
> * Create edk2-codereview repository for evaluation of new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests and emails
> * Determine if webhook requests must be serialized?  Current POC is serialized.
> * Make sure webhook has error handling for all unexpected events/states.
> * Add logging of all events and emails to webhook

The logging sounds very useful, thank you.

Whenever a log message relates to an email, please consider logging the
message-id of that email, if possible.

> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7 support
> 
> # Ideas for Future Enhancements
> 
> * Run PatchCheck.py before assigning maintainers/reviewers.
> * Add a simple check that fails if a single patch spans more than one package.

Hmmm, good idea in general, but there have been valid exceptions to this
rule.

> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>   Once all commits have required tags, auto update commit messages in the
>   branch and wait for maintainer to set the `Push` label to run CI and auto
>   merge if all CI checks pass.

Thank you for writing this up (and for implementing the webhook)!
Laszlo

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/09/20 06:22, Ni, Ray wrote:
> Mike,
> It's a huge improvement to me as an Outlook user if pull-request-based review is enabled!
> 
> Please help me to understand: The pull-request-based review has been enabled naturally when edk2
> was migrated to Github. People don't use it because it's not accepted by community. Your process
> tries to meet community's needs of achieving all review comments in mails so pull-request-based
> review can be accepted by community. Right?
> 
> I just subscribed at https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc with
> empty password.
> I received the confirmation mail and clicked the link in the mail to confirm.
> But I waited for ~15 minutes and didn't receive the additional mail containing the auto-generated password.

That's because the proof-of-concept list is currently subscriber-only,
and subscription requests have to be manually approved -- by Phil, or by
me. The PoC list contains a bunch of webhook test messages, and while
they are not secret, they are not useful to the grand public (and
arguably shouldn't be indexed by web search engines either).

Once we go live, the intent is that production list be publicly visible.
(Of course spam could become a problem; we'll see.)

Importantly, I totally don't "insist" that the email archive be hosted
on redhat.com (in fact it's extra moderation work for me, which I don't
necessarily welcome); I just offered because Red Hat associates can
request such public-facing mailing lists if they support relevant open
source development efforts.

The traffic should be federated to multiple lists, preferably, and the
redhat.com-hosted list need not be the primary archive address. Wherever
the primary list will live, we can subscribe the mail-archive.com daemon
to it, too.

> I went to https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289.
> However, the page requires me to enter password.
> Can you please change the setting so that viewing the mail achieve doesn't need password?

The password protection should remain in place for now, I think. I've
approved your subscription request; sorry about the delay. (I avoid
reading work email on the weekend.)

Thanks!
Laszlo

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/10/20 23:29, Michael D Kinney wrote:
> Rebecca,
> 
> There is no difference between CI checks run during code review
> and the final CI checks before merge.  I think it is an interesting
> conversation to decide how many times those CI checks should be 
> run and if they should run automatically on every change during
> review or on demand.
> 
> Mergify is more flexible.  We want to make sure the git history
> is linear with not git merges and supports both single patches
> and patch series without squashing.  GitHub merge button by 
> default squashes all commits into a single commit.

(

Wow, "squash-on-merge" is even the *default* now? That's terrible.
Unfortunately, github.com sets a very bad example with this, which is
made worse by github's popularity.

How can we expect developers to think about bisectability and patch
series structuring as first class traits of their contributions if
github.com actively educates them to ignore those aspects? Shaking my head.

)

Laszlo

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/10/20 23:43, Rebecca Cran wrote:
> Mike,
> 
> On 5/10/20 3:29 PM, Michael D Kinney wrote:
> 
>> There is no difference between CI checks run during code review
>> and the final CI checks before merge.  I think it is an interesting
>> conversation to decide how many times those CI checks should be
>> run and if they should run automatically on every change during
>> review or on demand.
> 
> I'd suggest following what other Github projects do, which I think is to
> run the CI checks automatically on every change that's made in a pull
> request - I don't know if it might also be necessary to run them during
> the merge, if master has changed in the meantime. That gives the
> _submitter_ feedback about any changes they need to make, instead of
> having to wait until the maintainer tells them their change has broken
> something: it speeds up the development process.

Build-testing at every stage through a patch series is important for
ensuring bisectability.

But there's a critical ingredient to that: based on the assumption that
our build system / build rules are good, the builds mentioned above
should be *incremental*.

That is, if we have a patch set with 10 patches, then then the first
patch in the series should trigger a complete build, and the 9 later
patches should trigger only incremental builds.

(During a bisection, the same commits wouldn't be visited in that same
order of course, but that's where the sanity of the build system / build
rules comes in! Basically, if your builds succeed with a linear
progression through the series, then the build system / build rules
ought to *guarantee* that the same "tree states" will build
incrementally just fine when visited in any particular order. "git
checkout" updates the relevant files, and the build system should be
able to derive the minimum set of necessary actions.

Anyway, digression ends.)

The incremental nature of builds is important for saving energy, and
also for saving developer time. The above 10-part example series should
not take 10 times as long to build as 10 independent patches, submitted
in isolation. Patches#2 through #10 should only rebuild a few modules
each (unless lib class headers, protocol headers and such are modified).


> 
>> Mergify is more flexible.  We want to make sure the git history
>> is linear with not git merges and supports both single patches
>> and patch series without squashing.  GitHub merge button by
>> default squashes all commits into a single commit.
> 
> Wouldn't disabling all but "Allow rebase merging" do the same thing
> without the additional potential failure point? Though it sounds like
> we've resolved the problems with Mergify, so it's not important.
> 
> https://help.github.com/en/github/administering-a-repository/configuring-commit-squashing-for-pull-requests

mergify has been pretty stable for me!

Thanks,
Laszlo

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/11/20 03:37, Michael D Kinney wrote:

> There was feedback from Laszlo related to rebase for
> pull requests using the current CI process.

To clarify, I don't think we should allow any github-side automatism to
auto-rebase pull requests. I think such rebases need to occur on
personal developer machines, under human oversight, and then resubmitted
(likely: force-pushed). My request is that the build costs (time,
energy) associated with such force-pushes be reduced somehow.

For example, on a local machine, the following sequence:

$ git checkout master
$ git pull
$ git rebase -i master my_topic_branch
$ build ...

would trigger an incremental build. *.c files not touched by either
operation would not have to be re-built (assuming their dependencies
didn't change either, such as lib class headers, protocol headers, ...)

Thanks,
Laszlo

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 23706 bytes --]

As a counterpoint: if we force a new branch or force push on every tweak, we lose the “thread” of discussion on what caused the change, what changed as a result, and the easy hook for the original change requester to reply directly to the change as is.

- Bret

From: Laszlo Ersek via groups.io<mailto:lersek=redhat.com@groups.io>
Sent: Monday, May 11, 2020 12:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>; rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 05/09/20 04:59, Michael D Kinney wrote:
> Hello,
>
> This is a proposal to change from the current email-based code review process to
> a GitHub pull request-based code review process for all repositories maintained
> in TianoCore.  The current email-based code review process and commit message
> requirements are documented in Readme.md or Readme.rst at the root of
> repositories along with a few Wiki pages:
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaster%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Development-Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-contributors-and-maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Message-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Signature-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;reserved=0
>
> The goal is to post changes by opening a GitHub pull request and perform all
> code review activity using the GitHub web interface.  This proposal does not
> change any licenses or commit message requirements.  It does require all
> developers, maintainers, and reviewers to have GitHub accounts.
>
> One requirement that was collected from previous discussions on this topic is
> the need for an email archive of all patches and code review activities.  The
> existing GitHub features to produce an email archive were deemed insufficient.
> A proof of concept of a GitHub webhook has been implemented to provide the email
> archive service.  This email archive is read-only.  You will not be able to send
> emails to this archive or reply to emails in the archive.
>
> The sections below provide more details on the proposed GitHub pull request
> based code review process, details on the email archive service, and a set of
> remaining tasks make the email archive service production quality.  It does not
> make sense to support both the existing email-based code review and the GitHub
> pull request-based code review at the same time.  Instead, this proposal is to
> switch to the GitHub pull request-based code review and retire the email based
> code review process on the same date.
>
> The edk2 repository is using GitHub pull requests today to run automated
> CI checks on the code changes and allows a maintainer to set the `push` label to
> request the changes to be merged if all CI checks pass.  With this proposal,
> once the code review is complete and the commit messages have been updated, the
> same pull request can be used to perform a final set of CI checks and merge the
> changes into the master branch.
>
> I would like to collect feedback on this proposal and the email archive service
> over the next two weeks with close of comments on Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like to see the community
> agree to make this change as soon as all remaining tasks are completed.
>
> # TianoCore Repositories to enable
>
> * [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
> * [edk2-platforms](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp;reserved=0)
> * [edk2-non-osi](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lrEsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reserved=0)
> * [edk2-test](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserved=0)
> * [edk2-libc](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tzt293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;reserved=0)
> * [edk2-staging](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;reserved=0)
>
> # GitHub Pull Request Code Review Process
>
> **NOTE**: All steps below use [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0) as an
> example.  Several repositories are supported.
>
> ## Author/Developer Steps
>   * Create a personal fork of [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-started-with-github%2Ffork-a-repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=umI3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;reserved=0
>
>   * Create a new branch from edk2/master in personal fork of edk2 repository.
>
>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>     follow the commit message format requirements.  The only change with this
>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>     The Cc: lines are still supported, but they should only be used to add
>     reviewers that do not have GitHub IDs or are not members of TianoCore.
>
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Fcreating-a-pull-request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;reserved=0
>
>   * If pull request has more than 1 commit, then fill in the pull request title
>     and decryption information for Patch #0.  Do not leave defaults.

s/decryption/description/

(Because I'm assuming this will turn into a wiki article at some point.)

>
>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>     the pull request and each commit in the pull request.
>
>   * If maintainers/reviewers provide feedback that requires changes, then make
>     add commits to the current branch with the requested changes.  Once all

s/make add/add/

>     changes are accepted on the current branch, reformulate the patch series and
>     commit comments as needed for perform a forced push to the branch in the
>     personal fork of the edk2 repository.  This step may be repeated if multiple
>     versions of the patch series are required to address all code review
>     feedback.

Do I understand correctly that this recommends the contributor first
push incremental patches on top of the series, then do a rebase
(squashing updates as necessary) and finally do a force-push, for the
next round of review?

To me as a reviewer, that's extra work. I'm used to locally comparing
the v(n) patch set to v(n+1) with git-range-diff, and/or with some
personal scripts. I wouldn't encourage incremental patches appended --
even temporarily -- to the branch, because (a) it's extra review work
(it requires me to review something that has zero chance to get into the
git history as-is), and (b) it superficially resembles the
github.com-specific bad practice called "squash on merge", and (c) it
runs the risk that the maintainer responsible for ultimately merging the
series ends up actually merging the incremental (= "fixup") patches in
isolation (without squashing them).

>
>   **OPEN**: How should minimum review period be set?  Labels?

Not sure about the best tooling. My recommendation would be to require
reviewers to start providing their feedback within one week.

One thing that I find important is that a maintainer can signal "I got
your work in my queue, but I may need more time". And a special case of
that are automated out-of-office responses. I think they are very
helpful (when a contributor feels they are bottlenecked on review), but
I'm not sure how one can configure that via github. I certainly would
not share my out-of-office times with github. (I set the start/end dates
in my email infrastructure, at the moment, but the out-of-office
messages it sends do not contain the dates either, on purpose.)

>
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull request and each commit in
>     the pull request and cross references against `Maintainters.txt` in the root

s/cross references/cross reference them/ ?

>     of the repository to assign maintainers/reviewers to the pull request and
>     each commit in the pull request. Individual commit assignments are performed
>     by adding a commit comment of the following form:
>
>     [CodeReview] Review-request @mdkinney
>
>   * Generate and sends git patch review emails to the email archive.  Emails

s/sends/send/

>     are also sent to any Cc: tags in the commit messages.
>
>   * If the author/developer performs a forced push to the branch in their
>     personal fork of the edk2 repository, then a new set of patch review emails
>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>     messages.
>
>   * Receive events associated with all code review activities and generate
>     and send emails to the email archive that shows all review comments and
>     all responses closely matching the email contents seen in the current email
>     based code review process.
>
>   * Generate and send email when pull request is merged or closed.
>
> ## Maintainer/Reviewer Steps
>
>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>     you have maintainer ship or review responsibilities and that email

s/maintainer ship/maintainership/

>     notifications from GitHub are enabled.  This enables email notifications
>     when a maintainer/reviewer is assigned to a pull request and individual
>     commits.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-subscriptions-and-notifications-on-github%2Fconfiguring-notifications&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%3D&amp;reserved=0
>
>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>     Archive Webhook Service.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0

Important: as the name says ("-poc"), this is a Proof of Concept list,
for now. Once we're ready to switch over, I'll file an internal ticket
at RH to either rename the list, or (which is probably better) to create
a new list (no "-poc" suffix).

The second option seems more useful because then the webhook development
/ bugfixing (if any) could perhaps occur in parallel to the normal edk2
workflow.

>
>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI to provide all review
>     feedback.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Freviewing-changes-in-pull-requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;reserved=0
>
>   * Wait for Author/Developer to respond to all feedback and add commits with
>     code changes as needed to resolve all feedback.  This step may be repeated
>     if the developer/author need to produce multiple versions of the patch
>     series to address all feedback.

(same question about the incremental fixup patches as above)

>
>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull request.
>
>   * Wait for Developer/Author to add tags to commit messages in the pull request.
>
>   * Perform final review of patches and commit message tags.  If there are not
>     issues, set the `push` label to run final set of CI checks and auto merge
>     the pull request into master.
>
> # Maintainers.txt Format Changes
>
> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
> in [].  For example:
>
>     M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]
>
> # TianoCore GitHub Email Archive Webhook Service
>
> Assign reviewers to commits in a GitHub pull request based on assignments
> documented in Maintainers.txt and generates an email archive of all pull request
> and code review activities.

s/generates/generate/

(or s/Assign/Assigns/)

>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;reserved=0
>
> # Email Archive Subscription Service
>
> The emails are being delivered to the following RedHat email subscription
> service.  Please subscribe to receive the emails and to be able to view the
> email archives.
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0
>
> The email archives are at this link:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M%3D&amp;reserved=0
>
> The following sections show some example pull requests and code reviews to
> help review the generated emails, their contents, and threading.
>
> ## Email Achieve Thread View
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzhkpI6DkE%3D&amp;reserved=0
>
> ## Example patch series with 1 patch
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtjdFjD%2BY3NA%3D&amp;reserved=0
>
> ## Example patch series with < 10 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4ol%2FyahQVU%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gFpXA%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9lh6hE%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNTzAE%3D&amp;reserved=0
>
> ## Example patch series with > 80 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d4%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV7TDOc%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%2BoX39Y%3D&amp;reserved=0
>
> # Tasks to Complete
>
> * Create edk2-codereview repository for evaluation of new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests and emails
> * Determine if webhook requests must be serialized?  Current POC is serialized.
> * Make sure webhook has error handling for all unexpected events/states.
> * Add logging of all events and emails to webhook

The logging sounds very useful, thank you.

Whenever a log message relates to an email, please consider logging the
message-id of that email, if possible.

> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7 support
>
> # Ideas for Future Enhancements
>
> * Run PatchCheck.py before assigning maintainers/reviewers.
> * Add a simple check that fails if a single patch spans more than one package.

Hmmm, good idea in general, but there have been valid exceptions to this
rule.

> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>   Once all commits have required tags, auto update commit messages in the
>   branch and wait for maintainer to set the `Push` label to run CI and auto
>   merge if all CI checks pass.

Thank you for writing this up (and for implementing the webhook)!
Laszlo





[-- Attachment #2: Type: text/html, Size: 38049 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 25161 bytes --]

Hi Bret,

This is a good point.

What I am proposing is the first version of the patch series submitted as a pull request.  Let the community do a complete review of the content.  The submitter can add patches to the end of the pull request addressing feedback and can even add patches that make changes to previous patches until all feedback/conversations are resolved.  This keeps the conversations complete and the conversations will also be archived to the email archive.

At this point, the developer can reformulate the patch series and do forced push of V2.  Reviewers can review the cleaned up patch series and repeat the process if there is more feedback, or move to final approval.

By doing all the work on a single pull request, we minimize the total number of pull requests in the repo.

An alternative approach would be to open a new pull request for each new version of the series.  This would preserve the GitHub conversations for each version of the pull request.  All the earlier ones would be closed/abandoned, and only the final one would be closed/merged.

Best regards,

Mike

From: Bret Barkelew <Bret.Barkelew@microsoft.com>
Sent: Monday, May 11, 2020 1:10 PM
To: devel@edk2.groups.io; lersek@redhat.com; Kinney, Michael D <michael.d.kinney@intel.com>; rfc@edk2.groups.io
Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

As a counterpoint: if we force a new branch or force push on every tweak, we lose the “thread” of discussion on what caused the change, what changed as a result, and the easy hook for the original change requester to reply directly to the change as is.

- Bret

From: Laszlo Ersek via groups.io<mailto:lersek=redhat.com@groups.io>
Sent: Monday, May 11, 2020 12:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>; rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 05/09/20 04:59, Michael D Kinney wrote:
> Hello,
>
> This is a proposal to change from the current email-based code review process to
> a GitHub pull request-based code review process for all repositories maintained
> in TianoCore.  The current email-based code review process and commit message
> requirements are documented in Readme.md or Readme.rst at the root of
> repositories along with a few Wiki pages:
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaster%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Development-Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-contributors-and-maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Message-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Signature-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;reserved=0
>
> The goal is to post changes by opening a GitHub pull request and perform all
> code review activity using the GitHub web interface.  This proposal does not
> change any licenses or commit message requirements.  It does require all
> developers, maintainers, and reviewers to have GitHub accounts.
>
> One requirement that was collected from previous discussions on this topic is
> the need for an email archive of all patches and code review activities.  The
> existing GitHub features to produce an email archive were deemed insufficient.
> A proof of concept of a GitHub webhook has been implemented to provide the email
> archive service.  This email archive is read-only.  You will not be able to send
> emails to this archive or reply to emails in the archive.
>
> The sections below provide more details on the proposed GitHub pull request
> based code review process, details on the email archive service, and a set of
> remaining tasks make the email archive service production quality.  It does not
> make sense to support both the existing email-based code review and the GitHub
> pull request-based code review at the same time.  Instead, this proposal is to
> switch to the GitHub pull request-based code review and retire the email based
> code review process on the same date.
>
> The edk2 repository is using GitHub pull requests today to run automated
> CI checks on the code changes and allows a maintainer to set the `push` label to
> request the changes to be merged if all CI checks pass.  With this proposal,
> once the code review is complete and the commit messages have been updated, the
> same pull request can be used to perform a final set of CI checks and merge the
> changes into the master branch.
>
> I would like to collect feedback on this proposal and the email archive service
> over the next two weeks with close of comments on Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like to see the community
> agree to make this change as soon as all remaining tasks are completed.
>
> # TianoCore Repositories to enable
>
> * [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
> * [edk2-platforms](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp;reserved=0)
> * [edk2-non-osi](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lrEsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reserved=0)
> * [edk2-test](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserved=0)
> * [edk2-libc](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tzt293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;reserved=0)
> * [edk2-staging](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;reserved=0)
>
> # GitHub Pull Request Code Review Process
>
> **NOTE**: All steps below use [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0) as an
> example.  Several repositories are supported.
>
> ## Author/Developer Steps
>   * Create a personal fork of [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-started-with-github%2Ffork-a-repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=umI3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;reserved=0
>
>   * Create a new branch from edk2/master in personal fork of edk2 repository.
>
>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>     follow the commit message format requirements.  The only change with this
>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>     The Cc: lines are still supported, but they should only be used to add
>     reviewers that do not have GitHub IDs or are not members of TianoCore.
>
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Fcreating-a-pull-request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;reserved=0
>
>   * If pull request has more than 1 commit, then fill in the pull request title
>     and decryption information for Patch #0.  Do not leave defaults.

s/decryption/description/

(Because I'm assuming this will turn into a wiki article at some point.)

>
>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>     the pull request and each commit in the pull request.
>
>   * If maintainers/reviewers provide feedback that requires changes, then make
>     add commits to the current branch with the requested changes.  Once all

s/make add/add/

>     changes are accepted on the current branch, reformulate the patch series and
>     commit comments as needed for perform a forced push to the branch in the
>     personal fork of the edk2 repository.  This step may be repeated if multiple
>     versions of the patch series are required to address all code review
>     feedback.

Do I understand correctly that this recommends the contributor first
push incremental patches on top of the series, then do a rebase
(squashing updates as necessary) and finally do a force-push, for the
next round of review?

To me as a reviewer, that's extra work. I'm used to locally comparing
the v(n) patch set to v(n+1) with git-range-diff, and/or with some
personal scripts. I wouldn't encourage incremental patches appended --
even temporarily -- to the branch, because (a) it's extra review work
(it requires me to review something that has zero chance to get into the
git history as-is), and (b) it superficially resembles the
github.com-specific bad practice called "squash on merge", and (c) it
runs the risk that the maintainer responsible for ultimately merging the
series ends up actually merging the incremental (= "fixup") patches in
isolation (without squashing them).

>
>   **OPEN**: How should minimum review period be set?  Labels?

Not sure about the best tooling. My recommendation would be to require
reviewers to start providing their feedback within one week.

One thing that I find important is that a maintainer can signal "I got
your work in my queue, but I may need more time". And a special case of
that are automated out-of-office responses. I think they are very
helpful (when a contributor feels they are bottlenecked on review), but
I'm not sure how one can configure that via github. I certainly would
not share my out-of-office times with github. (I set the start/end dates
in my email infrastructure, at the moment, but the out-of-office
messages it sends do not contain the dates either, on purpose.)

>
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull request and each commit in
>     the pull request and cross references against `Maintainters.txt` in the root

s/cross references/cross reference them/ ?

>     of the repository to assign maintainers/reviewers to the pull request and
>     each commit in the pull request. Individual commit assignments are performed
>     by adding a commit comment of the following form:
>
>     [CodeReview] Review-request @mdkinney
>
>   * Generate and sends git patch review emails to the email archive.  Emails

s/sends/send/

>     are also sent to any Cc: tags in the commit messages.
>
>   * If the author/developer performs a forced push to the branch in their
>     personal fork of the edk2 repository, then a new set of patch review emails
>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>     messages.
>
>   * Receive events associated with all code review activities and generate
>     and send emails to the email archive that shows all review comments and
>     all responses closely matching the email contents seen in the current email
>     based code review process.
>
>   * Generate and send email when pull request is merged or closed.
>
> ## Maintainer/Reviewer Steps
>
>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>     you have maintainer ship or review responsibilities and that email

s/maintainer ship/maintainership/

>     notifications from GitHub are enabled.  This enables email notifications
>     when a maintainer/reviewer is assigned to a pull request and individual
>     commits.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-subscriptions-and-notifications-on-github%2Fconfiguring-notifications&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%3D&amp;reserved=0
>
>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>     Archive Webhook Service.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0

Important: as the name says ("-poc"), this is a Proof of Concept list,
for now. Once we're ready to switch over, I'll file an internal ticket
at RH to either rename the list, or (which is probably better) to create
a new list (no "-poc" suffix).

The second option seems more useful because then the webhook development
/ bugfixing (if any) could perhaps occur in parallel to the normal edk2
workflow.

>
>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI to provide all review
>     feedback.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Freviewing-changes-in-pull-requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;reserved=0
>
>   * Wait for Author/Developer to respond to all feedback and add commits with
>     code changes as needed to resolve all feedback.  This step may be repeated
>     if the developer/author need to produce multiple versions of the patch
>     series to address all feedback.

(same question about the incremental fixup patches as above)

>
>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull request.
>
>   * Wait for Developer/Author to add tags to commit messages in the pull request.
>
>   * Perform final review of patches and commit message tags.  If there are not
>     issues, set the `push` label to run final set of CI checks and auto merge
>     the pull request into master.
>
> # Maintainers.txt Format Changes
>
> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
> in [].  For example:
>
>     M: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> [mdkinney]
>
> # TianoCore GitHub Email Archive Webhook Service
>
> Assign reviewers to commits in a GitHub pull request based on assignments
> documented in Maintainers.txt and generates an email archive of all pull request
> and code review activities.

s/generates/generate/

(or s/Assign/Assigns/)

>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;reserved=0
>
> # Email Archive Subscription Service
>
> The emails are being delivered to the following RedHat email subscription
> service.  Please subscribe to receive the emails and to be able to view the
> email archives.
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0
>
> The email archives are at this link:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M%3D&amp;reserved=0
>
> The following sections show some example pull requests and code reviews to
> help review the generated emails, their contents, and threading.
>
> ## Email Achieve Thread View
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzhkpI6DkE%3D&amp;reserved=0
>
> ## Example patch series with 1 patch
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtjdFjD%2BY3NA%3D&amp;reserved=0
>
> ## Example patch series with < 10 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4ol%2FyahQVU%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gFpXA%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9lh6hE%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNTzAE%3D&amp;reserved=0
>
> ## Example patch series with > 80 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d4%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV7TDOc%3D&amp;reserved=0
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%2BoX39Y%3D&amp;reserved=0
>
> # Tasks to Complete
>
> * Create edk2-codereview repository for evaluation of new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests and emails
> * Determine if webhook requests must be serialized?  Current POC is serialized.
> * Make sure webhook has error handling for all unexpected events/states.
> * Add logging of all events and emails to webhook

The logging sounds very useful, thank you.

Whenever a log message relates to an email, please consider logging the
message-id of that email, if possible.

> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7 support
>
> # Ideas for Future Enhancements
>
> * Run PatchCheck.py before assigning maintainers/reviewers.
> * Add a simple check that fails if a single patch spans more than one package.

Hmmm, good idea in general, but there have been valid exceptions to this
rule.

> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>   Once all commits have required tags, auto update commit messages in the
>   branch and wait for maintainer to set the `Push` label to run CI and auto
>   merge if all CI checks pass.

Thank you for writing this up (and for implementing the webhook)!
Laszlo





[-- Attachment #2: Type: text/html, Size: 81750 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/11/20 22:09, Bret Barkelew wrote:
> As a counterpoint: if we force a new branch or force push on every tweak, we lose the “thread� of discussion on what caused the change,

This is a github.com limitation.

And the email archive mitigates it.

In the current process, when I review v2 of a 10-part series, I have one
Thunderbird window open with the v1 thread, containing both the v1
patches and my (and others') review comments given for them.

(I open the new window by right-clicking the v1 blurb, and then
selecting "open message in new window". Then I navigate between the
messages of the v1 thread with the "f" and "b" hotkeys. The "scope" of
the new window is set to the v1 thread, recursively, when I open the new
window like that, and so "f" and "b" just do the right thing.)

In another window, to the right side, I run "git-range-diff", to
interdiff the v1 patches (patch by patch) with the v2 patches. (An
interdiff is a diff of diffs.) Importantly, the interdiff also
highlights commit message differences.

I verify that all the feedback comments from the v1 thread have been
addressed (per patch), and also that any otherwise "uncalled-for"
changes in v2 are in fact justified. (The contributor may have
justifiedly implemented further changes than what I requested under v1.)
This is also the reason why I meticulously number my feedback comments,
as I'm going to require a complete (one by one) coverage in the next
version of the patch set. (Except for those comments of course that the
contributor successfully refutes.)

When the v2 series has different structure from v1, then git-range-diff
is not as helpful -- in that case, I compare only a subset of the
patches like described above, and the entirely new patches in v2 I have
to review from zero.

The entire process depends on having unfettered access to comments given
for *any* earlier version of the patch set (it's not uncommon that I
refer back to v(n-3) or v(n-2) when reviewing v(n)), with those comments
being tightly bound (for display and for re-reading) to their subject
patches.

The github webui destroys (at least visually) the comments given before
a force-push. I can't fathom how incremental reviews can work on
github.com *at all*, in other projects. Hence my earlier suggestion to
use new pull requests rather than force-pushes.

But the mailing list archive generated by the webhook will solve this
completely -- I will use that list as a primary review support tool (for
v2, v3, ...), not only as an archive.


... After all, I guess I could reformulate like this: it's not my intent
to prevent people from pushing incremental fixups *temporarily*; I'm
only saying I will ignore those patches, and I will review only the next
full version of the branch.


My concern that does persist is this: "it runs the risk that the
maintainer responsible for ultimately merging the series ends up
actually merging the incremental (= "fixup") patches in isolation
(without squashing them)".

The git history should neither be littered with fixup patches, nor
contain huge squashes. The structure of a patch series is a first class
trait; it is an aspect to iterate upon, when a branch is being
contributed. The tooling should support that. (And the list traffic
generated by the webhook does.)

For instance, the last time I've given feedback regarding patch series
structure was just an hour ago, under the series "[PATCH V4 00/27]
Disabling safe string constraint assertions". I requested moving a hunk
from patch#1 to patch#26. Having the hunk in patch#1 does not break
bisection, and it's irrelevant for the end-state after the whole series
is applied (the end-state is the same). But the hunk still doesn't
*belong* in patch#1 -- wherever we add a new bit to a bitmask PCD
(patch#26), the UNI file (= documentation) udpate belongs in the exact
same patch.

> what changed as a result, and the easy hook for the original change requester to reply directly to the change as is.

No matter what I say about an incremental/fixup patch in isolation,
things can easily go wrong when the contributor squashes the fixup into
the more substantial patch that needs the fixup. Not to mention any
commit message updates on the more substantial patch, as necessitated by
the fixup. So I'll have to review the next full version of the topic
branch anyway, with git-range-diff, and compare the interdiff against my
earlier feedback.

Thanks!
Laszlo


> 
> - Bret
> 
> From: Laszlo Ersek via groups.io<mailto:lersek=redhat.com@groups.io>
> Sent: Monday, May 11, 2020 12:39 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>; rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
> Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process
> 
> On 05/09/20 04:59, Michael D Kinney wrote:
>> Hello,
>>
>> This is a proposal to change from the current email-based code review process to
>> a GitHub pull request-based code review process for all repositories maintained
>> in TianoCore.  The current email-based code review process and commit message
>> requirements are documented in Readme.md or Readme.rst at the root of
>> repositories along with a few Wiki pages:
>>
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaster%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Development-Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-contributors-and-maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Message-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Signature-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;reserved=0
>>
>> The goal is to post changes by opening a GitHub pull request and perform all
>> code review activity using the GitHub web interface.  This proposal does not
>> change any licenses or commit message requirements.  It does require all
>> developers, maintainers, and reviewers to have GitHub accounts.
>>
>> One requirement that was collected from previous discussions on this topic is
>> the need for an email archive of all patches and code review activities.  The
>> existing GitHub features to produce an email archive were deemed insufficient.
>> A proof of concept of a GitHub webhook has been implemented to provide the email
>> archive service.  This email archive is read-only.  You will not be able to send
>> emails to this archive or reply to emails in the archive.
>>
>> The sections below provide more details on the proposed GitHub pull request
>> based code review process, details on the email archive service, and a set of
>> remaining tasks make the email archive service production quality.  It does not
>> make sense to support both the existing email-based code review and the GitHub
>> pull request-based code review at the same time.  Instead, this proposal is to
>> switch to the GitHub pull request-based code review and retire the email based
>> code review process on the same date.
>>
>> The edk2 repository is using GitHub pull requests today to run automated
>> CI checks on the code changes and allows a maintainer to set the `push` label to
>> request the changes to be merged if all CI checks pass.  With this proposal,
>> once the code review is complete and the commit messages have been updated, the
>> same pull request can be used to perform a final set of CI checks and merge the
>> changes into the master branch.
>>
>> I would like to collect feedback on this proposal and the email archive service
>> over the next two weeks with close of comments on Friday May 22, 2020.  If all
>> issues and concerns can be addressed, then I would like to see the community
>> agree to make this change as soon as all remaining tasks are completed.
>>
>> # TianoCore Repositories to enable
>>
>> * [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
>> * [edk2-platforms](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp;reserved=0)
>> * [edk2-non-osi](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lrEsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reserved=0)
>> * [edk2-test](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserved=0)
>> * [edk2-libc](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tzt293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;reserved=0)
>> * [edk2-staging](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;reserved=0)
>>
>> # GitHub Pull Request Code Review Process
>>
>> **NOTE**: All steps below use [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0) as an
>> example.  Several repositories are supported.
>>
>> ## Author/Developer Steps
>>   * Create a personal fork of [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0)
>>
>>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-started-with-github%2Ffork-a-repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=umI3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;reserved=0
>>
>>   * Create a new branch from edk2/master in personal fork of edk2 repository.
>>
>>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>>     follow the commit message format requirements.  The only change with this
>>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>>     The Cc: lines are still supported, but they should only be used to add
>>     reviewers that do not have GitHub IDs or are not members of TianoCore.
>>
>>   * Push branch with new commits to personal fork
>>   * Create a pull request against TianoCore edk2/master
>>
>>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Fcreating-a-pull-request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;reserved=0
>>
>>   * If pull request has more than 1 commit, then fill in the pull request title
>>     and decryption information for Patch #0.  Do not leave defaults.
> 
> s/decryption/description/
> 
> (Because I'm assuming this will turn into a wiki article at some point.)
> 
>>
>>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>>     the pull request and each commit in the pull request.
>>
>>   * If maintainers/reviewers provide feedback that requires changes, then make
>>     add commits to the current branch with the requested changes.  Once all
> 
> s/make add/add/
> 
>>     changes are accepted on the current branch, reformulate the patch series and
>>     commit comments as needed for perform a forced push to the branch in the
>>     personal fork of the edk2 repository.  This step may be repeated if multiple
>>     versions of the patch series are required to address all code review
>>     feedback.
> 
> Do I understand correctly that this recommends the contributor first
> push incremental patches on top of the series, then do a rebase
> (squashing updates as necessary) and finally do a force-push, for the
> next round of review?
> 
> To me as a reviewer, that's extra work. I'm used to locally comparing
> the v(n) patch set to v(n+1) with git-range-diff, and/or with some
> personal scripts. I wouldn't encourage incremental patches appended --
> even temporarily -- to the branch, because (a) it's extra review work
> (it requires me to review something that has zero chance to get into the
> git history as-is), and (b) it superficially resembles the
> github.com-specific bad practice called "squash on merge", and (c) it
> runs the risk that the maintainer responsible for ultimately merging the
> series ends up actually merging the incremental (= "fixup") patches in
> isolation (without squashing them).
> 
>>
>>   **OPEN**: How should minimum review period be set?  Labels?
> 
> Not sure about the best tooling. My recommendation would be to require
> reviewers to start providing their feedback within one week.
> 
> One thing that I find important is that a maintainer can signal "I got
> your work in my queue, but I may need more time". And a special case of
> that are automated out-of-office responses. I think they are very
> helpful (when a contributor feels they are bottlenecked on review), but
> I'm not sure how one can configure that via github. I certainly would
> not share my out-of-office times with github. (I set the start/end dates
> in my email infrastructure, at the moment, but the out-of-office
> messages it sends do not contain the dates either, on purpose.)
> 
>>
>> ## TianoCore GitHub Email Archive Webhook Service Steps
>>   * Receive an event that a new pull request was opened
>>   * Evaluate the files modified by the entire pull request and each commit in
>>     the pull request and cross references against `Maintainters.txt` in the root
> 
> s/cross references/cross reference them/ ?
> 
>>     of the repository to assign maintainers/reviewers to the pull request and
>>     each commit in the pull request. Individual commit assignments are performed
>>     by adding a commit comment of the following form:
>>
>>     [CodeReview] Review-request @mdkinney
>>
>>   * Generate and sends git patch review emails to the email archive.  Emails
> 
> s/sends/send/
> 
>>     are also sent to any Cc: tags in the commit messages.
>>
>>   * If the author/developer performs a forced push to the branch in their
>>     personal fork of the edk2 repository, then a new set of patch review emails
>>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>>     messages.
>>
>>   * Receive events associated with all code review activities and generate
>>     and send emails to the email archive that shows all review comments and
>>     all responses closely matching the email contents seen in the current email
>>     based code review process.
>>
>>   * Generate and send email when pull request is merged or closed.
>>
>> ## Maintainer/Reviewer Steps
>>
>>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>>     you have maintainer ship or review responsibilities and that email
> 
> s/maintainer ship/maintainership/
> 
>>     notifications from GitHub are enabled.  This enables email notifications
>>     when a maintainer/reviewer is assigned to a pull request and individual
>>     commits.
>>
>>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-subscriptions-and-notifications-on-github%2Fconfiguring-notifications&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%3D&amp;reserved=0
>>
>>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>>     Archive Webhook Service.
>>
>>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0
> 
> Important: as the name says ("-poc"), this is a Proof of Concept list,
> for now. Once we're ready to switch over, I'll file an internal ticket
> at RH to either rename the list, or (which is probably better) to create
> a new list (no "-poc" suffix).
> 
> The second option seems more useful because then the webhook development
> / bugfixing (if any) could perhaps occur in parallel to the normal edk2
> workflow.
> 
>>
>>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>>     Archive Webhook Service and use the GitHub web UI to provide all review
>>     feedback.
>>
>>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Freviewing-changes-in-pull-requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;reserved=0
>>
>>   * Wait for Author/Developer to respond to all feedback and add commits with
>>     code changes as needed to resolve all feedback.  This step may be repeated
>>     if the developer/author need to produce multiple versions of the patch
>>     series to address all feedback.
> 
> (same question about the incremental fixup patches as above)
> 
>>
>>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>>     or Series-Tested-by responses to the entire pull request.
>>
>>   * Wait for Developer/Author to add tags to commit messages in the pull request.
>>
>>   * Perform final review of patches and commit message tags.  If there are not
>>     issues, set the `push` label to run final set of CI checks and auto merge
>>     the pull request into master.
>>
>> # Maintainers.txt Format Changes
>>
>> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
>> in [].  For example:
>>
>>     M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]
>>
>> # TianoCore GitHub Email Archive Webhook Service
>>
>> Assign reviewers to commits in a GitHub pull request based on assignments
>> documented in Maintainers.txt and generates an email archive of all pull request
>> and code review activities.
> 
> s/generates/generate/
> 
> (or s/Assign/Assigns/)
> 
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;reserved=0
>>
>> # Email Archive Subscription Service
>>
>> The emails are being delivered to the following RedHat email subscription
>> service.  Please subscribe to receive the emails and to be able to view the
>> email archives.
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0
>>
>> The email archives are at this link:
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M%3D&amp;reserved=0
>>
>> The following sections show some example pull requests and code reviews to
>> help review the generated emails, their contents, and threading.
>>
>> ## Email Achieve Thread View
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzhkpI6DkE%3D&amp;reserved=0
>>
>> ## Example patch series with 1 patch
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtjdFjD%2BY3NA%3D&amp;reserved=0
>>
>> ## Example patch series with < 10 patches
>>
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4ol%2FyahQVU%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gFpXA%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9lh6hE%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNTzAE%3D&amp;reserved=0
>>
>> ## Example patch series with > 80 patches
>>
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d4%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV7TDOc%3D&amp;reserved=0
>> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%2BoX39Y%3D&amp;reserved=0
>>
>> # Tasks to Complete
>>
>> * Create edk2-codereview repository for evaluation of new code review process.
>> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
>> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>>   the end of M: and R: statements
>> * Update webhook to use Rabbit MQ to manage requests and emails
>> * Determine if webhook requests must be serialized?  Current POC is serialized.
>> * Make sure webhook has error handling for all unexpected events/states.
>> * Add logging of all events and emails to webhook
> 
> The logging sounds very useful, thank you.
> 
> Whenever a log message relates to an email, please consider logging the
> message-id of that email, if possible.
> 
>> * Add admin interface to webhook
>> * Deploy webhook on a production server with 24/7 support
>>
>> # Ideas for Future Enhancements
>>
>> * Run PatchCheck.py before assigning maintainers/reviewers.
>> * Add a simple check that fails if a single patch spans more than one package.
> 
> Hmmm, good idea in general, but there have been valid exceptions to this
> rule.
> 
>> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>>   Once all commits have required tags, auto update commit messages in the
>>   branch and wait for maintainer to set the `Push` label to run CI and auto
>>   merge if all CI checks pass.
> 
> Thank you for writing this up (and for implementing the webhook)!
> Laszlo
> 
> 
> 
> 
> 

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 36552 bytes --]

I feel like this process is a good compromise. It’s not perfect (frankly, I’m a fan of enforced squash merges, which can maintain bisectability if managed well), but it allows for rapid iteration, ease of contribution, and approaches the workflow that many who have never used email to maintain a project would be familiar with.

It’s code management for the Instagram generation, and I for one welcome our new insect overlords.

- Bret

From: Kinney, Michael D<mailto:michael.d.kinney@intel.com>
Sent: Monday, May 11, 2020 1:43 PM
To: Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; lersek@redhat.com<mailto:lersek@redhat.com>; rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>
Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

Hi Bret,

This is a good point.

What I am proposing is the first version of the patch series submitted as a pull request.  Let the community do a complete review of the content.  The submitter can add patches to the end of the pull request addressing feedback and can even add patches that make changes to previous patches until all feedback/conversations are resolved.  This keeps the conversations complete and the conversations will also be archived to the email archive.

At this point, the developer can reformulate the patch series and do forced push of V2.  Reviewers can review the cleaned up patch series and repeat the process if there is more feedback, or move to final approval.

By doing all the work on a single pull request, we minimize the total number of pull requests in the repo.

An alternative approach would be to open a new pull request for each new version of the series.  This would preserve the GitHub conversations for each version of the pull request.  All the earlier ones would be closed/abandoned, and only the final one would be closed/merged.

Best regards,

Mike

From: Bret Barkelew <Bret.Barkelew@microsoft.com>
Sent: Monday, May 11, 2020 1:10 PM
To: devel@edk2.groups.io; lersek@redhat.com; Kinney, Michael D <michael.d.kinney@intel.com>; rfc@edk2.groups.io
Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

As a counterpoint: if we force a new branch or force push on every tweak, we lose the “thread” of discussion on what caused the change, what changed as a result, and the easy hook for the original change requester to reply directly to the change as is.

- Bret

From: Laszlo Ersek via groups.io<mailto:lersek=redhat.com@groups.io>
Sent: Monday, May 11, 2020 12:39 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>; rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 05/09/20 04:59, Michael D Kinney wrote:
> Hello,
>
> This is a proposal to change from the current email-based code review process to
> a GitHub pull request-based code review process for all repositories maintained
> in TianoCore.  The current email-based code review process and commit message
> requirements are documented in Readme.md or Readme.rst at the root of
> repositories along with a few Wiki pages:
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaster%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaster%2FReadMe.rst&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951667090&sdata=S%2Fu9iHwOFHKtYL7jeqIVGZLwDRbG%2F8%2BUm6qQxtLpwH0%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Development-Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FEDK-II-Development-Process&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951667090&sdata=l4uZzb0JliBkFXCQ7YfNBXs3Aoky0RYQn5gVT34AlH4%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-contributors-and-maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-contributors-and-maintainers&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951677089&sdata=Wo2qJFt7cHi5zZS96kCml7MZI%2B32v%2FiRqPdICvpTw5c%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Message-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Message-Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951677089&sdata=RjfxykjBSMEU%2BqsYkAmDPl%2FIgvBTPx%2BCvSIOPexpcc8%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Signature-Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io%2Fwiki%2FCommit-Signature-Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951687191&sdata=jPadwqi8wSOKaVOKU3o2JOegzvTqdz8o7bSLyT%2B0El8%3D&reserved=0>
>
> The goal is to post changes by opening a GitHub pull request and perform all
> code review activity using the GitHub web interface.  This proposal does not
> change any licenses or commit message requirements.  It does require all
> developers, maintainers, and reviewers to have GitHub accounts.
>
> One requirement that was collected from previous discussions on this topic is
> the need for an email archive of all patches and code review activities.  The
> existing GitHub features to produce an email archive were deemed insufficient.
> A proof of concept of a GitHub webhook has been implemented to provide the email
> archive service.  This email archive is read-only.  You will not be able to send
> emails to this archive or reply to emails in the archive.
>
> The sections below provide more details on the proposed GitHub pull request
> based code review process, details on the email archive service, and a set of
> remaining tasks make the email archive service production quality.  It does not
> make sense to support both the existing email-based code review and the GitHub
> pull request-based code review at the same time.  Instead, this proposal is to
> switch to the GitHub pull request-based code review and retire the email based
> code review process on the same date.
>
> The edk2 repository is using GitHub pull requests today to run automated
> CI checks on the code changes and allows a maintainer to set the `push` label to
> request the changes to be merged if all CI checks pass.  With this proposal,
> once the code review is complete and the commit messages have been updated, the
> same pull request can be used to perform a final set of CI checks and merge the
> changes into the master branch.
>
> I would like to collect feedback on this proposal and the email archive service
> over the next two weeks with close of comments on Friday May 22, 2020.  If all
> issues and concerns can be addressed, then I would like to see the community
> agree to make this change as soon as all remaining tasks are completed.
>
> # TianoCore Repositories to enable
>
> * [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951697080&sdata=YhFQ9Fxt8Y0kFdOTgY2v9vML9uCOpS9j5cKJj%2FPCToc%3D&reserved=0>)
> * [edk2-platforms](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-platforms&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951697080&sdata=dG8NsNTDjSX05wQWXargPnJydEGQyvZCia%2BSOIUyG6o%3D&reserved=0>)
> * [edk2-non-osi](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lrEsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-osi&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951707067&sdata=Mkz1CUW26UJ2X6wIEgO0UGz0ZJqWhkXMFwc5v5420tU%3D&reserved=0>)
> * [edk2-test](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-test&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951707067&sdata=fzDqTRet6MgfwmVIq8mb5%2BgB3rAiVteCQklombfBMaU%3D&reserved=0>)
> * [edk2-libc](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tzt293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-libc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951717077&sdata=wpaAwogDjzDwRzpoB5vInxeqtMkNoawyPFg1j3m9omw%3D&reserved=0>)
> * [edk2-staging](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-staging&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951717077&sdata=5ehzG40XuHbF56QwzFHyY8krO1NwGDWnFVMZdZ9OUvM%3D&reserved=0>)
>
> # GitHub Pull Request Code Review Process
>
> **NOTE**: All steps below use [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9JzgwbKGP6o%3D&reserved=0>) as an
> example.  Several repositories are supported.
>
> ## Author/Developer Steps
>   * Create a personal fork of [edk2](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9JzgwbKGP6o%3D&reserved=0>)
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-started-with-github%2Ffork-a-repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=umI3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-started-with-github%2Ffork-a-repo&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951737053&sdata=pC%2F9W90ZNlmGQFNy97ee1xQMJ93XC%2Bdo5lmIdZhnRk0%3D&reserved=0>
>
>   * Create a new branch from edk2/master in personal fork of edk2 repository.
>
>   * Add set of commits for new feature or bug fix to new branch.  Make sure to
>     follow the commit message format requirements.  The only change with this
>     RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
>     The Cc: lines are still supported, but they should only be used to add
>     reviewers that do not have GitHub IDs or are not members of TianoCore.
>
>   * Push branch with new commits to personal fork
>   * Create a pull request against TianoCore edk2/master
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Fcreating-a-pull-request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Fcreating-a-pull-request&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951737053&sdata=sZjbyeRye35une4C2y5RQ83ah%2B01o4XhFNXsvIXT1l0%3D&reserved=0>
>
>   * If pull request has more than 1 commit, then fill in the pull request title
>     and decryption information for Patch #0.  Do not leave defaults.

s/decryption/description/

(Because I'm assuming this will turn into a wiki article at some point.)

>
>   * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
>     the pull request and each commit in the pull request.
>
>   * If maintainers/reviewers provide feedback that requires changes, then make
>     add commits to the current branch with the requested changes.  Once all

s/make add/add/

>     changes are accepted on the current branch, reformulate the patch series and
>     commit comments as needed for perform a forced push to the branch in the
>     personal fork of the edk2 repository.  This step may be repeated if multiple
>     versions of the patch series are required to address all code review
>     feedback.

Do I understand correctly that this recommends the contributor first
push incremental patches on top of the series, then do a rebase
(squashing updates as necessary) and finally do a force-push, for the
next round of review?

To me as a reviewer, that's extra work. I'm used to locally comparing
the v(n) patch set to v(n+1) with git-range-diff, and/or with some
personal scripts. I wouldn't encourage incremental patches appended --
even temporarily -- to the branch, because (a) it's extra review work
(it requires me to review something that has zero chance to get into the
git history as-is), and (b) it superficially resembles the
github.com-specific bad practice called "squash on merge", and (c) it
runs the risk that the maintainer responsible for ultimately merging the
series ends up actually merging the incremental (= "fixup") patches in
isolation (without squashing them).

>
>   **OPEN**: How should minimum review period be set?  Labels?

Not sure about the best tooling. My recommendation would be to require
reviewers to start providing their feedback within one week.

One thing that I find important is that a maintainer can signal "I got
your work in my queue, but I may need more time". And a special case of
that are automated out-of-office responses. I think they are very
helpful (when a contributor feels they are bottlenecked on review), but
I'm not sure how one can configure that via github. I certainly would
not share my out-of-office times with github. (I set the start/end dates
in my email infrastructure, at the moment, but the out-of-office
messages it sends do not contain the dates either, on purpose.)

>
> ## TianoCore GitHub Email Archive Webhook Service Steps
>   * Receive an event that a new pull request was opened
>   * Evaluate the files modified by the entire pull request and each commit in
>     the pull request and cross references against `Maintainters.txt` in the root

s/cross references/cross reference them/ ?

>     of the repository to assign maintainers/reviewers to the pull request and
>     each commit in the pull request. Individual commit assignments are performed
>     by adding a commit comment of the following form:
>
>     [CodeReview] Review-request @mdkinney
>
>   * Generate and sends git patch review emails to the email archive.  Emails

s/sends/send/

>     are also sent to any Cc: tags in the commit messages.
>
>   * If the author/developer performs a forced push to the branch in their
>     personal fork of the edk2 repository, then a new set of patch review emails
>     with patch series Vx is sent to the email archive and any Cc: tags in commit
>     messages.
>
>   * Receive events associated with all code review activities and generate
>     and send emails to the email archive that shows all review comments and
>     all responses closely matching the email contents seen in the current email
>     based code review process.
>
>   * Generate and send email when pull request is merged or closed.
>
> ## Maintainer/Reviewer Steps
>
>   * Make sure GitHub configuration is setup to 'Watch' the repositories that
>     you have maintainer ship or review responsibilities and that email

s/maintainer ship/maintainership/

>     notifications from GitHub are enabled.  This enables email notifications
>     when a maintainer/reviewer is assigned to a pull request and individual
>     commits.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-subscriptions-and-notifications-on-github%2Fconfiguring-notifications&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-subscriptions-and-notifications-on-github%2Fconfiguring-notifications&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951747044&sdata=FJwHjpFL7UFtZ9N12pSvQSXci3IqCJJVJXI9dDzlHHU%3D&reserved=0>
>
>   * Subscribe to the email archive associated with the TianoCore GitHub Email
>     Archive Webhook Service.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951747044&sdata=zbDuJd9lW3KoiuzGo6%2BU14Mnude7lZnbCeXQHJnR%2BCg%3D&reserved=0>

Important: as the name says ("-poc"), this is a Proof of Concept list,
for now. Once we're ready to switch over, I'll file an internal ticket
at RH to either rename the list, or (which is probably better) to create
a new list (no "-poc" suffix).

The second option seems more useful because then the webhook development
/ bugfixing (if any) could perhaps occur in parallel to the normal edk2
workflow.

>
>   * Review pull requests and commits assigned by the TianoCore GitHub Email
>     Archive Webhook Service and use the GitHub web UI to provide all review
>     feedback.
>
>     https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Freviewing-changes-in-pull-requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaborating-with-issues-and-pull-requests%2Freviewing-changes-in-pull-requests&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=Iav2d2fP2vw9mS9uA76tN3uQcIbl52D6BT963cD7ZC4%3D&reserved=0>
>
>   * Wait for Author/Developer to respond to all feedback and add commits with
>     code changes as needed to resolve all feedback.  This step may be repeated
>     if the developer/author need to produce multiple versions of the patch
>     series to address all feedback.

(same question about the incremental fixup patches as above)

>
>   * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
>     responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
>     or Series-Tested-by responses to the entire pull request.
>
>   * Wait for Developer/Author to add tags to commit messages in the pull request.
>
>   * Perform final review of patches and commit message tags.  If there are not
>     issues, set the `push` label to run final set of CI checks and auto merge
>     the pull request into master.
>
> # Maintainers.txt Format Changes
>
> Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
> in [].  For example:
>
>     M: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> [mdkinney]
>
> # TianoCore GitHub Email Archive Webhook Service
>
> Assign reviewers to commits in a GitHub pull request based on assignments
> documented in Maintainers.txt and generates an email archive of all pull request
> and code review activities.

s/generates/generate/

(or s/Assign/Assigns/)

>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-webhook&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=FihVwcwfqewohsmqUECWG36TR1iCQVhVHq02iMZkEL8%3D&reserved=0>
>
> # Email Archive Subscription Service
>
> The emails are being delivered to the following RedHat email subscription
> service.  Please subscribe to receive the emails and to be able to view the
> email archives.
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0IuvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianocore-code-review-poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951767033&sdata=482jeizloSfEobAk0F8%2BvP9FZptdF6oC90A%2BYSQZcds%3D&reserved=0>
>
> The email archives are at this link:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2Findex.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951767033&sdata=78HRihY2z%2Fll0qoC05RMh8I%2Bwri9rwIwblxNBYFPecw%3D&reserved=0>
>
> The following sections show some example pull requests and code reviews to
> help review the generated emails, their contents, and threading.
>
> ## Email Achieve Thread View
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzhkpI6DkE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300289&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951777024&sdata=%2BU8aFUi4kPBeJdlyW%2FERNtp5Fum5gZrWkgCTI8pwz14%3D&reserved=0>
>
> ## Example patch series with 1 patch
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtjdFjD%2BY3NA%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fthread.html%2300340&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951777024&sdata=gXn%2B9z8MIdnJyUkk2xR1mK%2FL710jUDEeNhK74EUkb9A%3D&reserved=0>
>
> ## Example patch series with < 10 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4ol%2FyahQVU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00289.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&sdata=yvE8%2F%2FKOJUj%2BD4ZmtT3fZPAb2KBsGcAqxt%2FKwRKa6hA%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gFpXA%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00030.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&sdata=zz5BgISidxyQVI%2BQihn%2Fq%2Fv9%2B5%2Fn2vpaeyvFbIiAigs%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9lh6hE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00018.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&sdata=yN0WkrPdk1EEvyPwlqCG%2B6HTOpa3iT85w2QhXQJ2xYE%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNTzAE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00008.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&sdata=qF2vw8XdQfY0ECclzhbpd60vIgTj2Bzl5RjC%2F4EdbA8%3D&reserved=0>
>
> ## Example patch series with > 80 patches
>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d4%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00198.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&sdata=uXAPx9iTDbQbiv40j3EiqIP51nlFLQ43ghvAsjt4dhs%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV7TDOc%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00116.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&sdata=4ANIbH2YDPjN7H4gHGtbgNkAbM9EeVFOf%2FoWqTiXtiA%3D&reserved=0>
> * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%2BoX39Y%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-May%2Fmsg00035.html&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951817004&sdata=1jX%2FuPSMlRvzN9eZcv5U7k4JmwfJJwaVhDJ%2BrA6mPYc%3D&reserved=0>
>
> # Tasks to Complete
>
> * Create edk2-codereview repository for evaluation of new code review process.
> * Add GitHub IDs to Maintainers.txt in edk2-codereview repository
> * Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
>   the end of M: and R: statements
> * Update webhook to use Rabbit MQ to manage requests and emails
> * Determine if webhook requests must be serialized?  Current POC is serialized.
> * Make sure webhook has error handling for all unexpected events/states.
> * Add logging of all events and emails to webhook

The logging sounds very useful, thank you.

Whenever a log message relates to an email, please consider logging the
message-id of that email, if possible.

> * Add admin interface to webhook
> * Deploy webhook on a production server with 24/7 support
>
> # Ideas for Future Enhancements
>
> * Run PatchCheck.py before assigning maintainers/reviewers.
> * Add a simple check that fails if a single patch spans more than one package.

Hmmm, good idea in general, but there have been valid exceptions to this
rule.

> * Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
>   Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
>   Once all commits have required tags, auto update commit messages in the
>   branch and wait for maintainer to set the `Push` label to run CI and auto
>   merge if all CI checks pass.

Thank you for writing this up (and for implementing the webhook)!
Laszlo






[-- Attachment #2: Type: text/html, Size: 44390 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Rebecca Cran]

On 5/14/20 3:26 PM, Bret Barkelew via groups.io wrote:

> I feel like this process is a good compromise. It’s not perfect (frankly, I’m a fan of enforced squash merges, which can maintain bisectability if managed well), but it allows for rapid iteration, ease of contribution, and approaches the workflow that many who have never used email to maintain a project would be familiar with.
>
> It’s code management for the Instagram generation, and I for one welcome our new insect overlords.

Or at least, that's what Microsoft is betting on! :D

Personally, I remain unconvinced about the usability of Github Pull 
Requests for a project the size of EDK2, but I hope to be proven wrong.


-- 
Rebecca Cran

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Michael D Kinney]

Bret,

If the original submission is a single patch, and the code review
generates changes that are added as additional patches for review,
but the intent in the end is still a single patch, then squashing
them all at the end is correct.

Using the GitHub feature to squash them is a challenge because of 
the EDK II commit message requirements.  In order to make sure the
final commit message for the one commit is correct, the developer
should squash and edit the commit message.

There may be opportunities to automate these operations, but the
developer still needs to do the final review and have the ability
to do further edits of the commit message.

Mike

> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
> Of Bret Barkelew via groups.io
> Sent: Thursday, May 14, 2020 2:27 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io; lersek@redhat.com;
> rfc@edk2.groups.io
> Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
> 
> I feel like this process is a good compromise. It’s not
> perfect (frankly, I’m a fan of enforced squash merges,
> which can maintain bisectability if managed well), but
> it allows for rapid iteration, ease of contribution,
> and approaches the workflow that many who have never
> used email to maintain a project would be familiar
> with.
> 
> It’s code management for the Instagram generation, and
> I for one welcome our new insect overlords.
> 
> - Bret
> 
> From: Kinney, Michael
> D<mailto:michael.d.kinney@intel.com>
> Sent: Monday, May 11, 2020 1:43 PM
> To: Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>;
> devel@edk2.groups.io<mailto:devel@edk2.groups.io>;
> lersek@redhat.com<mailto:lersek@redhat.com>;
> rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Kinney,
> Michael D<mailto:michael.d.kinney@intel.com>
> Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
> 
> Hi Bret,
> 
> This is a good point.
> 
> What I am proposing is the first version of the patch
> series submitted as a pull request.  Let the community
> do a complete review of the content.  The submitter can
> add patches to the end of the pull request addressing
> feedback and can even add patches that make changes to
> previous patches until all feedback/conversations are
> resolved.  This keeps the conversations complete and
> the conversations will also be archived to the email
> archive.
> 
> At this point, the developer can reformulate the patch
> series and do forced push of V2.  Reviewers can review
> the cleaned up patch series and repeat the process if
> there is more feedback, or move to final approval.
> 
> By doing all the work on a single pull request, we
> minimize the total number of pull requests in the repo.
> 
> An alternative approach would be to open a new pull
> request for each new version of the series.  This would
> preserve the GitHub conversations for each version of
> the pull request.  All the earlier ones would be
> closed/abandoned, and only the final one would be
> closed/merged.
> 
> Best regards,
> 
> Mike
> 
> From: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Sent: Monday, May 11, 2020 1:10 PM
> To: devel@edk2.groups.io; lersek@redhat.com; Kinney,
> Michael D <michael.d.kinney@intel.com>;
> rfc@edk2.groups.io
> Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
> 
> As a counterpoint: if we force a new branch or force
> push on every tweak, we lose the “thread” of discussion
> on what caused the change, what changed as a result,
> and the easy hook for the original change requester to
> reply directly to the change as is.
> 
> - Bret
> 
> From: Laszlo Ersek via
> groups.io<mailto:lersek=redhat.com@groups.io>
> Sent: Monday, May 11, 2020 12:39 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>;
> Kinney, Michael D<mailto:michael.d.kinney@intel.com>;
> rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
> Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub
> Pull Request based Code Review Process
> 
> On 05/09/20 04:59, Michael D Kinney wrote:
> > Hello,
> >
> > This is a proposal to change from the current email-
> based code review process to
> > a GitHub pull request-based code review process for
> all repositories maintained
> > in TianoCore.  The current email-based code review
> process and commit message
> > requirements are documented in Readme.md or
> Readme.rst at the root of
> > repositories along with a few Wiki pages:
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaste
> r%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40micro
> soft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&am
> p;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&
> amp;reserved=0<https://nam06.safelinks.protection.outlo
> ok.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2
> %2Fblob%2Fmaster%2FReadMe.rst&data=02%7C01%7CBret.Barke
> lew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482659
> 51667090&sdata=S%2Fu9iHwOFHKtYL7jeqIVGZLwDRbG%2F8%2BUm6
> qQxtLpwH0%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-Development-
> Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata
> =sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp
> ;reserved=0<https://nam06.safelinks.protection.outlook.
> com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianoco
> re.github.io%2Fwiki%2FEDK-II-Development-
> Process&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951667090&sdata=l4uZzb0
> JliBkFXCQ7YfNBXs3Aoky0RYQn5gVT34AlH4%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-
> contributors-and-
> maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsof
> t.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f1
> 41af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;s
> data=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D
> &amp;reserved=0<https://nam06.safelinks.protection.outl
> ook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftia
> nocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-
> for-edk2-contributors-and-
> maintainers&data=02%7C01%7CBret.Barkelew%40microsoft.co
> m%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248265951677089&sdata=Wo2
> qJFt7cHi5zZS96kCml7MZI%2B32v%2FiRqPdICvpTw5c%3D&reserve
> d=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FCommit-Message-
> Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com
> %7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=
> uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.c
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocor
> e.github.io%2Fwiki%2FCommit-Message-
> Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb
> 61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C637248265951677089&sdata=RjfxykjB
> SMEU%2BqsYkAmDPl%2FIgvBTPx%2BCvSIOPexpcc8%3D&reserved=0
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FCommit-Signature-
> Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com
> %7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=
> Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.c
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocor
> e.github.io%2Fwiki%2FCommit-Signature-
> Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb
> 61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C637248265951687191&sdata=jPadwqi8
> wSOKaVOKU3o2JOegzvTqdz8o7bSLyT%2B0El8%3D&reserved=0>
> >
> > The goal is to post changes by opening a GitHub pull
> request and perform all
> > code review activity using the GitHub web interface.
> This proposal does not
> > change any licenses or commit message requirements.
> It does require all
> > developers, maintainers, and reviewers to have GitHub
> accounts.
> >
> > One requirement that was collected from previous
> discussions on this topic is
> > the need for an email archive of all patches and code
> review activities.  The
> > existing GitHub features to produce an email archive
> were deemed insufficient.
> > A proof of concept of a GitHub webhook has been
> implemented to provide the email
> > archive service.  This email archive is read-only.
> You will not be able to send
> > emails to this archive or reply to emails in the
> archive.
> >
> > The sections below provide more details on the
> proposed GitHub pull request
> > based code review process, details on the email
> archive service, and a set of
> > remaining tasks make the email archive service
> production quality.  It does not
> > make sense to support both the existing email-based
> code review and the GitHub
> > pull request-based code review at the same time.
> Instead, this proposal is to
> > switch to the GitHub pull request-based code review
> and retire the email based
> > code review process on the same date.
> >
> > The edk2 repository is using GitHub pull requests
> today to run automated
> > CI checks on the code changes and allows a maintainer
> to set the `push` label to
> > request the changes to be merged if all CI checks
> pass.  With this proposal,
> > once the code review is complete and the commit
> messages have been updated, the
> > same pull request can be used to perform a final set
> of CI checks and merge the
> > changes into the master branch.
> >
> > I would like to collect feedback on this proposal and
> the email archive service
> > over the next two weeks with close of comments on
> Friday May 22, 2020.  If all
> > issues and concerns can be addressed, then I would
> like to see the community
> > agree to make this change as soon as all remaining
> tasks are completed.
> >
> > # TianoCore Repositories to enable
> >
> > *
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951697080&sdata=YhFQ9Fxt8Y0kFdOTgY2v9vML9uCOpS9j5cKJ
> j%2FPCToc%3D&reserved=0>)
> > * [edk2-
> platforms](https://nam06.safelinks.protection.outlook.c
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.
> com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sda
> ta=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp
> ;reserved=0<https://nam06.safelinks.protection.outlook.
> com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> platforms&data=02%7C01%7CBret.Barkelew%40microsoft.com%
> 7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91
> ab2d7cd011db47%7C1%7C0%7C637248265951697080&sdata=dG8Ns
> NTDjSX05wQWXargPnJydEGQyvZCia%2BSOIUyG6o%3D&reserved=0>
> )
> > * [edk2-non-
> osi](https://nam06.safelinks.protection.outlook.com/?ur
> l=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-
> osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lr
> EsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reser
> ved=0<https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-
> osi&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951707067&sdata=Mkz1CUW26UJ
> 2X6wIEgO0UGz0ZJqWhkXMFwc5v5420tU%3D&reserved=0>)
> > * [edk2-
> test](https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v
> 205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserv
> ed=0<https://nam06.safelinks.protection.outlook.com/?ur
> l=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> test&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951707067&sdata=fzDqTRet6M
> gfwmVIq8mb5%2BgB3rAiVteCQklombfBMaU%3D&reserved=0>)
> > * [edk2-
> libc](https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tz
> t293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;rese
> rved=0<https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> libc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951717077&sdata=wpaAwogDjz
> DwRzpoB5vInxeqtMkNoawyPFg1j3m9omw%3D&reserved=0>)
> > * [edk2-
> staging](https://nam06.safelinks.protection.outlook.com
> /?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata
> =bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;r
> eserved=0<https://nam06.safelinks.protection.outlook.co
> m/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> staging&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951717077&sdata=5ehzG40
> XuHbF56QwzFHyY8krO1NwGDWnFVMZdZ9OUvM%3D&reserved=0>)
> >
> > # GitHub Pull Request Code Review Process
> >
> > **NOTE**: All steps below use
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9J
> zgwbKGP6o%3D&reserved=0>) as an
> > example.  Several repositories are supported.
> >
> > ## Author/Developer Steps
> >   * Create a personal fork of
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9J
> zgwbKGP6o%3D&reserved=0>)
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-
> started-with-github%2Ffork-a-
> repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=um
> I3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;rese
> rved=0<https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetti
> ng-started-with-github%2Ffork-a-
> repo&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951737053&sdata=pC%2F9W90Z
> NlmGQFNy97ee1xQMJ93XC%2Bdo5lmIdZhnRk0%3D&reserved=0>
> >
> >   * Create a new branch from edk2/master in personal
> fork of edk2 repository.
> >
> >   * Add set of commits for new feature or bug fix to
> new branch.  Make sure to
> >     follow the commit message format requirements.
> The only change with this
> >     RFC is that the Cc: lines to
> maintainers/reviewers should **not** be added.
> >     The Cc: lines are still supported, but they
> should only be used to add
> >     reviewers that do not have GitHub IDs or are not
> members of TianoCore.
> >
> >   * Push branch with new commits to personal fork
> >   * Create a pull request against TianoCore
> edk2/master
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaboratin
> g-with-issues-and-pull-requests%2Fcreating-a-pull-
> request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata
> =2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;res
> erved=0<https://nam06.safelinks.protection.outlook.com/
> ?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcoll
> aborating-with-issues-and-pull-requests%2Fcreating-a-
> pull-
> request&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951737053&sdata=sZjbyeR
> ye35une4C2y5RQ83ah%2B01o4XhFNXsvIXT1l0%3D&reserved=0>
> >
> >   * If pull request has more than 1 commit, then fill
> in the pull request title
> >     and decryption information for Patch #0.  Do not
> leave defaults.
> 
> s/decryption/description/
> 
> (Because I'm assuming this will turn into a wiki
> article at some point.)
> 
> >
> >   * Do not assign reviewers.  The webhook assigns
> maintainers and reviewers to
> >     the pull request and each commit in the pull
> request.
> >
> >   * If maintainers/reviewers provide feedback that
> requires changes, then make
> >     add commits to the current branch with the
> requested changes.  Once all
> 
> s/make add/add/
> 
> >     changes are accepted on the current branch,
> reformulate the patch series and
> >     commit comments as needed for perform a forced
> push to the branch in the
> >     personal fork of the edk2 repository.  This step
> may be repeated if multiple
> >     versions of the patch series are required to
> address all code review
> >     feedback.
> 
> Do I understand correctly that this recommends the
> contributor first
> push incremental patches on top of the series, then do
> a rebase
> (squashing updates as necessary) and finally do a
> force-push, for the
> next round of review?
> 
> To me as a reviewer, that's extra work. I'm used to
> locally comparing
> the v(n) patch set to v(n+1) with git-range-diff,
> and/or with some
> personal scripts. I wouldn't encourage incremental
> patches appended --
> even temporarily -- to the branch, because (a) it's
> extra review work
> (it requires me to review something that has zero
> chance to get into the
> git history as-is), and (b) it superficially resembles
> the
> github.com-specific bad practice called "squash on
> merge", and (c) it
> runs the risk that the maintainer responsible for
> ultimately merging the
> series ends up actually merging the incremental (=
> "fixup") patches in
> isolation (without squashing them).
> 
> >
> >   **OPEN**: How should minimum review period be set?
> Labels?
> 
> Not sure about the best tooling. My recommendation
> would be to require
> reviewers to start providing their feedback within one
> week.
> 
> One thing that I find important is that a maintainer
> can signal "I got
> your work in my queue, but I may need more time". And a
> special case of
> that are automated out-of-office responses. I think
> they are very
> helpful (when a contributor feels they are bottlenecked
> on review), but
> I'm not sure how one can configure that via github. I
> certainly would
> not share my out-of-office times with github. (I set
> the start/end dates
> in my email infrastructure, at the moment, but the out-
> of-office
> messages it sends do not contain the dates either, on
> purpose.)
> 
> >
> > ## TianoCore GitHub Email Archive Webhook Service
> Steps
> >   * Receive an event that a new pull request was
> opened
> >   * Evaluate the files modified by the entire pull
> request and each commit in
> >     the pull request and cross references against
> `Maintainters.txt` in the root
> 
> s/cross references/cross reference them/ ?
> 
> >     of the repository to assign maintainers/reviewers
> to the pull request and
> >     each commit in the pull request. Individual
> commit assignments are performed
> >     by adding a commit comment of the following form:
> >
> >     [CodeReview] Review-request @mdkinney
> >
> >   * Generate and sends git patch review emails to the
> email archive.  Emails
> 
> s/sends/send/
> 
> >     are also sent to any Cc: tags in the commit
> messages.
> >
> >   * If the author/developer performs a forced push to
> the branch in their
> >     personal fork of the edk2 repository, then a new
> set of patch review emails
> >     with patch series Vx is sent to the email archive
> and any Cc: tags in commit
> >     messages.
> >
> >   * Receive events associated with all code review
> activities and generate
> >     and send emails to the email archive that shows
> all review comments and
> >     all responses closely matching the email contents
> seen in the current email
> >     based code review process.
> >
> >   * Generate and send email when pull request is
> merged or closed.
> >
> > ## Maintainer/Reviewer Steps
> >
> >   * Make sure GitHub configuration is setup to
> 'Watch' the repositories that
> >     you have maintainer ship or review
> responsibilities and that email
> 
> s/maintainer ship/maintainership/
> 
> >     notifications from GitHub are enabled.  This
> enables email notifications
> >     when a maintainer/reviewer is assigned to a pull
> request and individual
> >     commits.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-
> subscriptions-and-notifications-on-
> github%2Fconfiguring-
> notifications&amp;data=02%7C01%7Cbret.barkelew%40micros
> oft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86
> f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp
> ;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%
> 3D&amp;reserved=0<https://nam06.safelinks.protection.ou
> tlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgit
> hub%2Fmanaging-subscriptions-and-notifications-on-
> github%2Fconfiguring-
> notifications&data=02%7C01%7CBret.Barkelew%40microsoft.
> com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637248265951747044&sdata=F
> JwHjpFL7UFtZ9N12pSvQSXci3IqCJJVJXI9dDzlHHU%3D&reserved=
> 0>
> >
> >   * Subscribe to the email archive associated with
> the TianoCore GitHub Email
> >     Archive Webhook Service.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianoc
> ore-code-review-
> poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0I
> uvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserve
> d=0<https://nam06.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Fti
> anocore-code-review-
> poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951747044&sdata=zbDuJd9lW3K
> oiuzGo6%2BU14Mnude7lZnbCeXQHJnR%2BCg%3D&reserved=0>
> 
> Important: as the name says ("-poc"), this is a Proof
> of Concept list,
> for now. Once we're ready to switch over, I'll file an
> internal ticket
> at RH to either rename the list, or (which is probably
> better) to create
> a new list (no "-poc" suffix).
> 
> The second option seems more useful because then the
> webhook development
> / bugfixing (if any) could perhaps occur in parallel to
> the normal edk2
> workflow.
> 
> >
> >   * Review pull requests and commits assigned by the
> TianoCore GitHub Email
> >     Archive Webhook Service and use the GitHub web UI
> to provide all review
> >     feedback.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaboratin
> g-with-issues-and-pull-requests%2Freviewing-changes-in-
> pull-
> requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.c
> om%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdat
> a=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.c
> om/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fc
> ollaborating-with-issues-and-pull-requests%2Freviewing-
> changes-in-pull-
> requests&data=02%7C01%7CBret.Barkelew%40microsoft.com%7
> Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=Iav2d2
> fP2vw9mS9uA76tN3uQcIbl52D6BT963cD7ZC4%3D&reserved=0>
> >
> >   * Wait for Author/Developer to respond to all
> feedback and add commits with
> >     code changes as needed to resolve all feedback.
> This step may be repeated
> >     if the developer/author need to produce multiple
> versions of the patch
> >     series to address all feedback.
> 
> (same question about the incremental fixup patches as
> above)
> 
> >
> >   * Once all feedback is addressed, add Reviewed-by,
> Acked-by, and Tested-by
> >     responses on individual commits.  Or add Series-
> reviewed-by, Series-acked-by,
> >     or Series-Tested-by responses to the entire pull
> request.
> >
> >   * Wait for Developer/Author to add tags to commit
> messages in the pull request.
> >
> >   * Perform final review of patches and commit
> message tags.  If there are not
> >     issues, set the `push` label to run final set of
> CI checks and auto merge
> >     the pull request into master.
> >
> > # Maintainers.txt Format Changes
> >
> > Add GitHub IDs of all maintainers and reviewers at
> the end of M: and R: lines
> > in [].  For example:
> >
> >     M: Michael D Kinney
> <michael.d.kinney@intel.com<mailto:michael.d.kinney@int
> el.com>> [mdkinney]
> >
> > # TianoCore GitHub Email Archive Webhook Service
> >
> > Assign reviewers to commits in a GitHub pull request
> based on assignments
> > documented in Maintainers.txt and generates an email
> archive of all pull request
> > and code review activities.
> 
> s/generates/generate/
> 
> (or s/Assign/Assigns/)
> 
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-
> webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata
> =7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;res
> erved=0<https://nam06.safelinks.protection.outlook.com/
> ?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-
> archive-
> webhook&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=FihVwcw
> fqewohsmqUECWG36TR1iCQVhVHq02iMZkEL8%3D&reserved=0>
> >
> > # Email Archive Subscription Service
> >
> > The emails are being delivered to the following
> RedHat email subscription
> > service.  Please subscribe to receive the emails and
> to be able to view the
> > email archives.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianoc
> ore-code-review-
> poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0I
> uvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserve
> d=0<https://nam06.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Fti
> anocore-code-review-
> poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951767033&sdata=482jeizloSf
> EobAk0F8%2BvP9FZptdF6oC90A%2BYSQZcds%3D&reserved=0>
> >
> > The email archives are at this link:
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-
> poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40mic
> rosoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988b
> f86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&
> amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M
> %3D&amp;reserved=0<https://nam06.safelinks.protection.o
> utlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%
> 2Fprivate%2Ftianocore-code-review-
> poc%2Findex.html&data=02%7C01%7CBret.Barkelew%40microso
> ft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f
> 141af91ab2d7cd011db47%7C1%7C0%7C637248265951767033&sdat
> a=78HRihY2z%2Fll0qoC05RMh8I%2Bwri9rwIwblxNBYFPecw%3D&re
> served=0>
> >
> > The following sections show some example pull
> requests and code reviews to
> > help review the generated emails, their contents, and
> threading.
> >
> > ## Email Achieve Thread View
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barke
> lew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275
> 86678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzh
> kpI6DkE%3D&amp;reserved=0<https://nam06.safelinks.prote
> ction.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fm
> ailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fthread.html%2300289&data=02%7C01%7CBret.Barkelew%
> 40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72
> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63724826595177
> 7024&sdata=%2BU8aFUi4kPBeJdlyW%2FERNtp5Fum5gZrWkgCTI8pw
> z14%3D&reserved=0>
> >
> > ## Example patch series with 1 patch
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barke
> lew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275
> 86678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtj
> dFjD%2BY3NA%3D&amp;reserved=0<https://nam06.safelinks.p
> rotection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com
> %2Fmailman%2Fprivate%2Ftianocore-code-review-
> poc%2F2020-
> May%2Fthread.html%2300340&data=02%7C01%7CBret.Barkelew%
> 40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72
> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63724826595177
> 7024&sdata=gXn%2B9z8MIdnJyUkk2xR1mK%2FL710jUDEeNhK74EUk
> b9A%3D&reserved=0>
> >
> > ## Example patch series with < 10 patches
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4
> ol%2FyahQVU%3D&amp;reserved=0<https://nam06.safelinks.p
> rotection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com
> %2Fmailman%2Fprivate%2Ftianocore-code-review-
> poc%2F2020-
> May%2Fmsg00289.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&s
> data=yvE8%2F%2FKOJUj%2BD4ZmtT3fZPAb2KBsGcAqxt%2FKwRKa6h
> A%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gF
> pXA%3D&amp;reserved=0<https://nam06.safelinks.protectio
> n.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailm
> an%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00030.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&s
> data=zz5BgISidxyQVI%2BQihn%2Fq%2Fv9%2B5%2Fn2vpaeyvFbIiA
> igs%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9
> lh6hE%3D&amp;reserved=0<https://nam06.safelinks.protect
> ion.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmai
> lman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00018.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&s
> data=yN0WkrPdk1EEvyPwlqCG%2B6HTOpa3iT85w2QhXQJ2xYE%3D&r
> eserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNT
> zAE%3D&amp;reserved=0<https://nam06.safelinks.protectio
> n.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailm
> an%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00008.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&s
> data=qF2vw8XdQfY0ECclzhbpd60vIgTj2Bzl5RjC%2F4EdbA8%3D&r
> eserved=0>
> >
> > ## Example patch series with > 80 patches
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d
> 4%3D&amp;reserved=0<https://nam06.safelinks.protection.
> outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman
> %2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00198.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&s
> data=uXAPx9iTDbQbiv40j3EiqIP51nlFLQ43ghvAsjt4dhs%3D&res
> erved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV
> 7TDOc%3D&amp;reserved=0<https://nam06.safelinks.protect
> ion.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmai
> lman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00116.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&s
> data=4ANIbH2YDPjN7H4gHGtbgNkAbM9EeVFOf%2FoWqTiXtiA%3D&r
> eserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%
> 2BoX39Y%3D&amp;reserved=0<https://nam06.safelinks.prote
> ction.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fm
> ailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00035.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951817004&s
> data=1jX%2FuPSMlRvzN9eZcv5U7k4JmwfJJwaVhDJ%2BrA6mPYc%3D
> &reserved=0>
> >
> > # Tasks to Complete
> >
> > * Create edk2-codereview repository for evaluation of
> new code review process.
> > * Add GitHub IDs to Maintainers.txt in edk2-
> codereview repository
> > * Update BaseTools/Scripts/GetMaintainer.py to be
> compatible with GitHub IDs at
> >   the end of M: and R: statements
> > * Update webhook to use Rabbit MQ to manage requests
> and emails
> > * Determine if webhook requests must be serialized?
> Current POC is serialized.
> > * Make sure webhook has error handling for all
> unexpected events/states.
> > * Add logging of all events and emails to webhook
> 
> The logging sounds very useful, thank you.
> 
> Whenever a log message relates to an email, please
> consider logging the
> message-id of that email, if possible.
> 
> > * Add admin interface to webhook
> > * Deploy webhook on a production server with 24/7
> support
> >
> > # Ideas for Future Enhancements
> >
> > * Run PatchCheck.py before assigning
> maintainers/reviewers.
> > * Add a simple check that fails if a single patch
> spans more than one package.
> 
> Hmmm, good idea in general, but there have been valid
> exceptions to this
> rule.
> 
> > * Monitor comments for Reviewed-by, Acked-by, Tested-
> by, Series-Reviewed-by,
> >   Series-Acked-by, Series-Tested-by made by assigned
> maintainers/reviewers.
> >   Once all commits have required tags, auto update
> commit messages in the
> >   branch and wait for maintainer to set the `Push`
> label to run CI and auto
> >   merge if all CI checks pass.
> 
> Thank you for writing this up (and for implementing the
> webhook)!
> Laszlo
> 
> 
> 
> 
> 
> 
> 

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 47417 bytes --]

Agreed. Responsibility to verify the commit message when squashing is always something to be aware of.

With Github, the one who presses the “Close and Merge” (or whatever it’s called) button has the final say on the title and message. We can play with approaches to how the squash merge would work.

I would far prefer the approach of individual PRs for commits to allow for the squash flexibility (and is the strategy I think I would pursue with my PRs). For example, the VarPol PR would be broken up into 9 PRs for each final commit, and we can get them in one by one. Ideally, each one would be a small back and forth and then in. If it had been done that way to begin with, it would be over in a week and a half or so, rather than the multiple months that we’re now verging on.

- Bret

From: Michael D Kinney via groups.io<mailto:michael.d.kinney=intel.com@groups.io>
Sent: Thursday, May 14, 2020 6:19 PM
To: rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; lersek@redhat.com<mailto:lersek@redhat.com>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

Bret,

If the original submission is a single patch, and the code review
generates changes that are added as additional patches for review,
but the intent in the end is still a single patch, then squashing
them all at the end is correct.

Using the GitHub feature to squash them is a challenge because of
the EDK II commit message requirements.  In order to make sure the
final commit message for the one commit is correct, the developer
should squash and edit the commit message.

There may be opportunities to automate these operations, but the
developer still needs to do the final review and have the ability
to do further edits of the commit message.

Mike

> -----Original Message-----
> From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
> Of Bret Barkelew via groups.io
> Sent: Thursday, May 14, 2020 2:27 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io; lersek@redhat.com;
> rfc@edk2.groups.io
> Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
>
> I feel like this process is a good compromise. It’s not
> perfect (frankly, I’m a fan of enforced squash merges,
> which can maintain bisectability if managed well), but
> it allows for rapid iteration, ease of contribution,
> and approaches the workflow that many who have never
> used email to maintain a project would be familiar
> with.
>
> It’s code management for the Instagram generation, and
> I for one welcome our new insect overlords.
>
> - Bret
>
> From: Kinney, Michael
> D<mailto:michael.d.kinney@intel.com>
> Sent: Monday, May 11, 2020 1:43 PM
> To: Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>;
> devel@edk2.groups.io<mailto:devel@edk2.groups.io>;
> lersek@redhat.com<mailto:lersek@redhat.com>;
> rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Kinney,
> Michael D<mailto:michael.d.kinney@intel.com>
> Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
>
> Hi Bret,
>
> This is a good point.
>
> What I am proposing is the first version of the patch
> series submitted as a pull request.  Let the community
> do a complete review of the content.  The submitter can
> add patches to the end of the pull request addressing
> feedback and can even add patches that make changes to
> previous patches until all feedback/conversations are
> resolved.  This keeps the conversations complete and
> the conversations will also be archived to the email
> archive.
>
> At this point, the developer can reformulate the patch
> series and do forced push of V2.  Reviewers can review
> the cleaned up patch series and repeat the process if
> there is more feedback, or move to final approval.
>
> By doing all the work on a single pull request, we
> minimize the total number of pull requests in the repo.
>
> An alternative approach would be to open a new pull
> request for each new version of the series.  This would
> preserve the GitHub conversations for each version of
> the pull request.  All the earlier ones would be
> closed/abandoned, and only the final one would be
> closed/merged.
>
> Best regards,
>
> Mike
>
> From: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Sent: Monday, May 11, 2020 1:10 PM
> To: devel@edk2.groups.io; lersek@redhat.com; Kinney,
> Michael D <michael.d.kinney@intel.com>;
> rfc@edk2.groups.io
> Subject: RE: [EXTERNAL] Re: [edk2-devel] [edk2-rfc]
> GitHub Pull Request based Code Review Process
>
> As a counterpoint: if we force a new branch or force
> push on every tweak, we lose the “thread” of discussion
> on what caused the change, what changed as a result,
> and the easy hook for the original change requester to
> reply directly to the change as is.
>
> - Bret
>
> From: Laszlo Ersek via
> groups.io<mailto:lersek=redhat.com@groups.io>
> Sent: Monday, May 11, 2020 12:39 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>;
> Kinney, Michael D<mailto:michael.d.kinney@intel.com>;
> rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>
> Subject: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub
> Pull Request based Code Review Process
>
> On 05/09/20 04:59, Michael D Kinney wrote:
> > Hello,
> >
> > This is a proposal to change from the current email-
> based code review process to
> > a GitHub pull request-based code review process for
> all repositories maintained
> > in TianoCore.  The current email-based code review
> process and commit message
> > requirements are documented in Readme.md or
> Readme.rst at the root of
> > repositories along with a few Wiki pages:
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fblob%2Fmaste
> r%2FReadMe.rst&amp;data=02%7C01%7Cbret.barkelew%40micro
> soft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&am
> p;sdata=lVjWRLsBC3xJpyRFeDrGjFhMOzAgi2V3vsAPxj7lIDw%3D&
> amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlo%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611242055&amp;sdata=Gr4Mw1Yz36xH3riEM7yicL5DIKv1%2FVuXM%2FObxAIWbJo%3D&amp;reserved=0
> ok.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2
> %2Fblob%2Fmaster%2FReadMe.rst&data=02%7C01%7CBret.Barke
> lew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482659
> 51667090&sdata=S%2Fu9iHwOFHKtYL7jeqIVGZLwDRbG%2F8%2BUm6
> qQxtLpwH0%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FEDK-II-Development-
> Process&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata
> =sgAhQxCpyjmzC%2FW%2BFiLLwaF2M8wscBz3k93ne25qUXs%3D&amp
> ;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=B0BrCucxkG8t6JzA0b113MLW5PocHmW54lhtOAQdR%2F4%3D&amp;reserved=0.
> com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianoco
> re.github.io%2Fwiki%2FEDK-II-Development-
> Process&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951667090&sdata=l4uZzb0
> JliBkFXCQ7YfNBXs3Aoky0RYQn5gVT34AlH4%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FLaszlo%27s-unkempt-git-guide-for-edk2-
> contributors-and-
> maintainers&amp;data=02%7C01%7Cbret.barkelew%40microsof
> t.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f1
> 41af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;s
> data=eHP9fcPMw6yjqTU%2B%2BUZ3FZkq8jZeM1LTU6dGTzmFp4Q%3D
> &amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outl%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=%2FfG2%2BrXhM02OXgg%2Fa1G8tBYx0mHIH8lTm%2FF7bKSh10M%3D&amp;reserved=0
> ook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftia
> nocore.github.io%2Fwiki%2FLaszlo%27s-unkempt-git-guide-
> for-edk2-contributors-and-
> maintainers&data=02%7C01%7CBret.Barkelew%40microsoft.co
> m%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248265951677089&sdata=Wo2
> qJFt7cHi5zZS96kCml7MZI%2B32v%2FiRqPdICvpTw5c%3D&reserve
> d=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FCommit-Message-
> Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com
> %7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=
> uq8G6nGyLpa7m%2F0fD2pwrcM9uixbKs6SLTge8e77M%2FY%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.c%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=%2BfY0MvXLLyf%2Bt7w2VFkWgfnEGFjgqdJo7WsZmQN8yLE%3D&amp;reserved=0
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocor
> e.github.io%2Fwiki%2FCommit-Message-
> Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb
> 61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C637248265951677089&sdata=RjfxykjB
> SMEU%2BqsYkAmDPl%2FIgvBTPx%2BCvSIOPexpcc8%3D&reserved=0
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
> %2Fwiki%2FCommit-Signature-
> Format&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com
> %7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af9
> 1ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=
> Mz8dUn2L8dFwJdlo4LbaIKt2JrWE%2Fn4tBtVWenK%2F8Ck%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.c%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=%2BfY0MvXLLyf%2Bt7w2VFkWgfnEGFjgqdJo7WsZmQN8yLE%3D&amp;reserved=0
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocor
> e.github.io%2Fwiki%2FCommit-Signature-
> Format&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb
> 61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2
> d7cd011db47%7C1%7C0%7C637248265951687191&sdata=jPadwqi8
> wSOKaVOKU3o2JOegzvTqdz8o7bSLyT%2B0El8%3D&reserved=0>
> >
> > The goal is to post changes by opening a GitHub pull
> request and perform all
> > code review activity using the GitHub web interface.
> This proposal does not
> > change any licenses or commit message requirements.
> It does require all
> > developers, maintainers, and reviewers to have GitHub
> accounts.
> >
> > One requirement that was collected from previous
> discussions on this topic is
> > the need for an email archive of all patches and code
> review activities.  The
> > existing GitHub features to produce an email archive
> were deemed insufficient.
> > A proof of concept of a GitHub webhook has been
> implemented to provide the email
> > archive service.  This email archive is read-only.
> You will not be able to send
> > emails to this archive or reply to emails in the
> archive.
> >
> > The sections below provide more details on the
> proposed GitHub pull request
> > based code review process, details on the email
> archive service, and a set of
> > remaining tasks make the email archive service
> production quality.  It does not
> > make sense to support both the existing email-based
> code review and the GitHub
> > pull request-based code review at the same time.
> Instead, this proposal is to
> > switch to the GitHub pull request-based code review
> and retire the email based
> > code review process on the same date.
> >
> > The edk2 repository is using GitHub pull requests
> today to run automated
> > CI checks on the code changes and allows a maintainer
> to set the `push` label to
> > request the changes to be merged if all CI checks
> pass.  With this proposal,
> > once the code review is complete and the commit
> messages have been updated, the
> > same pull request can be used to perform a final set
> of CI checks and merge the
> > changes into the master branch.
> >
> > I would like to collect feedback on this proposal and
> the email archive service
> > over the next two weeks with close of comments on
> Friday May 22, 2020.  If all
> > issues and concerns can be addressed, then I would
> like to see the community
> > agree to make this change as soon as all remaining
> tasks are completed.
> >
> > # TianoCore Repositories to enable
> >
> > *
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951697080&sdata=YhFQ9Fxt8Y0kFdOTgY2v9vML9uCOpS9j5cKJ
> j%2FPCToc%3D&reserved=0>)
> > * [edk2-
> platforms](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.c%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=%2BfY0MvXLLyf%2Bt7w2VFkWgfnEGFjgqdJo7WsZmQN8yLE%3D&amp;reserved=0
> om/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> platforms&amp;data=02%7C01%7Cbret.barkelew%40microsoft.
> com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sda
> ta=g8mgGL6B%2FRsvm3935OpZMctOTKUoeHGi8jPuCVKQjbI%3D&amp
> ;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=B0BrCucxkG8t6JzA0b113MLW5PocHmW54lhtOAQdR%2F4%3D&amp;reserved=0.
> com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> platforms&data=02%7C01%7CBret.Barkelew%40microsoft.com%
> 7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91
> ab2d7cd011db47%7C1%7C0%7C637248265951697080&sdata=dG8Ns
> NTDjSX05wQWXargPnJydEGQyvZCia%2BSOIUyG6o%3D&reserved=0>
> )
> > * [edk2-non-
> osi](https://nam06.safelinks.protection.outlook.com/?ur
> l=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-
> osi&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=9lr
> EsZWOpc3wqylKs7yF%2FzxYwZsUUamP3oUrWDWcHCc%3D&amp;reser
> ved=0<https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-non-
> osi&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951707067&sdata=Mkz1CUW26UJ
> 2X6wIEgO0UGz0ZJqWhkXMFwc5v5420tU%3D&reserved=0>)
> > * [edk2-
> test](https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> test&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=8v
> 205MD3HTYg3yLmGJS3SIDA5um9sVJfOa5CXViZjyU%3D&amp;reserv
> ed=0<https://nam06.safelinks.protection.outlook.com/?ur
> l=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> test&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951707067&sdata=fzDqTRet6M
> gfwmVIq8mb5%2BgB3rAiVteCQklombfBMaU%3D&reserved=0>)
> > * [edk2-
> libc](https://nam06.safelinks.protection.outlook.com/?u
> rl=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> libc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata=Tz
> t293HJzFnGSkh1mUBew8dAsaZ4axWq2ml8UhQ%2FSTI%3D&amp;rese
> rved=0<https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> libc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951717077&sdata=wpaAwogDjz
> DwRzpoB5vInxeqtMkNoawyPFg1j3m9omw%3D&reserved=0>)
> > * [edk2-
> staging](https://nam06.safelinks.protection.outlook.com/
> /?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> staging&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586668645&amp;sdata
> =bcNbt7Y7KoBrcnW4fAc4jbGgJL%2B4lYUkVLhYNo37OiM%3D&amp;r
> eserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.co%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=bR1nvNPVi%2BQ1NrpSJQlwdLTZB3SigLFQ3KV63kUTUzk%3D&amp;reserved=0
> m/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2-
> staging&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951717077&sdata=5ehzG40
> XuHbF56QwzFHyY8krO1NwGDWnFVMZdZ9OUvM%3D&reserved=0>)
> >
> > # GitHub Pull Request Code Review Process
> >
> > **NOTE**: All steps below use
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9J
> zgwbKGP6o%3D&reserved=0>) as an
> > example.  Several repositories are supported.
> >
> > ## Author/Developer Steps
> >   * Create a personal fork of
> [edk2](https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2&amp;dat
> a=02%7C01%7Cbret.barkelew%40microsoft.com%7C1dcf1f8c03b
> 544e5095408d7f5e2fd56%7C72f988bf86f141af91ab2d7cd011db4
> 7%7C1%7C0%7C637248227586668645&amp;sdata=Jvbl8ypdXIi7U5
> Jnr3s0TOx6hD54N55mdsbXi9sCznM%3D&amp;reserved=0<https:/
> /nam06.safelinks.protection.outlook.com/?url=https%3A%2
> F%2Fgithub.com%2Ftianocore%2Fedk2&data=02%7C01%7CBret.B
> arkelew%40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebe
> b5b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637248
> 265951727054&sdata=IrlV0ptekGi4Sz%2FxBYhIa5LiRuI3jlKE9J
> zgwbKGP6o%3D&reserved=0>)
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetting-
> started-with-github%2Ffork-a-
> repo&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7
> C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=um
> I3eqOh03qmt9YlPo33ujypu90YwImAvuxh5SlrM%2Bw%3D&amp;rese
> rved=0<https://nam06.safelinks.protection.outlook.com/?
> url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fgetti
> ng-started-with-github%2Ffork-a-
> repo&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61
> ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7
> cd011db47%7C1%7C0%7C637248265951737053&sdata=pC%2F9W90Z
> NlmGQFNy97ee1xQMJ93XC%2Bdo5lmIdZhnRk0%3D&reserved=0>
> >
> >   * Create a new branch from edk2/master in personal
> fork of edk2 repository.
> >
> >   * Add set of commits for new feature or bug fix to
> new branch.  Make sure to
> >     follow the commit message format requirements.
> The only change with this
> >     RFC is that the Cc: lines to
> maintainers/reviewers should **not** be added.
> >     The Cc: lines are still supported, but they
> should only be used to add
> >     reviewers that do not have GitHub IDs or are not
> members of TianoCore.
> >
> >   * Push branch with new commits to personal fork
> >   * Create a pull request against TianoCore
> edk2/master
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaboratin
> g-with-issues-and-pull-requests%2Fcreating-a-pull-
> request&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata
> =2GVrQy0FGwd4CCeGveh99HL3zS1ekRfAAaKhhRiOMpU%3D&amp;res
> erved=0<https://nam06.safelinks.protection.outlook.com/
> ?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcoll
> aborating-with-issues-and-pull-requests%2Fcreating-a-
> pull-
> request&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951737053&sdata=sZjbyeR
> ye35une4C2y5RQ83ah%2B01o4XhFNXsvIXT1l0%3D&reserved=0>
> >
> >   * If pull request has more than 1 commit, then fill
> in the pull request title
> >     and decryption information for Patch #0.  Do not
> leave defaults.
>
> s/decryption/description/
>
> (Because I'm assuming this will turn into a wiki
> article at some point.)
>
> >
> >   * Do not assign reviewers.  The webhook assigns
> maintainers and reviewers to
> >     the pull request and each commit in the pull
> request.
> >
> >   * If maintainers/reviewers provide feedback that
> requires changes, then make
> >     add commits to the current branch with the
> requested changes.  Once all
>
> s/make add/add/
>
> >     changes are accepted on the current branch,
> reformulate the patch series and
> >     commit comments as needed for perform a forced
> push to the branch in the
> >     personal fork of the edk2 repository.  This step
> may be repeated if multiple
> >     versions of the patch series are required to
> address all code review
> >     feedback.
>
> Do I understand correctly that this recommends the
> contributor first
> push incremental patches on top of the series, then do
> a rebase
> (squashing updates as necessary) and finally do a
> force-push, for the
> next round of review?
>
> To me as a reviewer, that's extra work. I'm used to
> locally comparing
> the v(n) patch set to v(n+1) with git-range-diff,
> and/or with some
> personal scripts. I wouldn't encourage incremental
> patches appended --
> even temporarily -- to the branch, because (a) it's
> extra review work
> (it requires me to review something that has zero
> chance to get into the
> git history as-is), and (b) it superficially resembles
> the
> github.com-specific bad practice called "squash on
> merge", and (c) it
> runs the risk that the maintainer responsible for
> ultimately merging the
> series ends up actually merging the incremental (=
> "fixup") patches in
> isolation (without squashing them).
>
> >
> >   **OPEN**: How should minimum review period be set?
> Labels?
>
> Not sure about the best tooling. My recommendation
> would be to require
> reviewers to start providing their feedback within one
> week.
>
> One thing that I find important is that a maintainer
> can signal "I got
> your work in my queue, but I may need more time". And a
> special case of
> that are automated out-of-office responses. I think
> they are very
> helpful (when a contributor feels they are bottlenecked
> on review), but
> I'm not sure how one can configure that via github. I
> certainly would
> not share my out-of-office times with github. (I set
> the start/end dates
> in my email infrastructure, at the moment, but the out-
> of-office
> messages it sends do not contain the dates either, on
> purpose.)
>
> >
> > ## TianoCore GitHub Email Archive Webhook Service
> Steps
> >   * Receive an event that a new pull request was
> opened
> >   * Evaluate the files modified by the entire pull
> request and each commit in
> >     the pull request and cross references against
> `Maintainters.txt` in the root
>
> s/cross references/cross reference them/ ?
>
> >     of the repository to assign maintainers/reviewers
> to the pull request and
> >     each commit in the pull request. Individual
> commit assignments are performed
> >     by adding a commit comment of the following form:
> >
> >     [CodeReview] Review-request @mdkinney
> >
> >   * Generate and sends git patch review emails to the
> email archive.  Emails
>
> s/sends/send/
>
> >     are also sent to any Cc: tags in the commit
> messages.
> >
> >   * If the author/developer performs a forced push to
> the branch in their
> >     personal fork of the edk2 repository, then a new
> set of patch review emails
> >     with patch series Vx is sent to the email archive
> and any Cc: tags in commit
> >     messages.
> >
> >   * Receive events associated with all code review
> activities and generate
> >     and send emails to the email archive that shows
> all review comments and
> >     all responses closely matching the email contents
> seen in the current email
> >     based code review process.
> >
> >   * Generate and send email when pull request is
> merged or closed.
> >
> > ## Maintainer/Reviewer Steps
> >
> >   * Make sure GitHub configuration is setup to
> 'Watch' the repositories that
> >     you have maintainer ship or review
> responsibilities and that email
>
> s/maintainer ship/maintainership/
>
> >     notifications from GitHub are enabled.  This
> enables email notifications
> >     when a maintainer/reviewer is assigned to a pull
> request and individual
> >     commits.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fmanaging-
> subscriptions-and-notifications-on-
> github%2Fconfiguring-
> notifications&amp;data=02%7C01%7Cbret.barkelew%40micros
> oft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86
> f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp
> ;sdata=OlkiyymcQi39P8%2FOJZG4yjh4h%2FHerkHBe5bCSQQFLOU%
> 3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.ou%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=rMEZi8aziILY8pK5e%2BvnY3TgsvV4EqzQ2qXf8QvaqIc%3D&amp;reserved=0
> tlook.com/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgit
> hub%2Fmanaging-subscriptions-and-notifications-on-
> github%2Fconfiguring-
> notifications&data=02%7C01%7CBret.Barkelew%40microsoft.
> com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141
> af91ab2d7cd011db47%7C1%7C0%7C637248265951747044&sdata=F
> JwHjpFL7UFtZ9N12pSvQSXci3IqCJJVJXI9dDzlHHU%3D&reserved=
> 0>
> >
> >   * Subscribe to the email archive associated with
> the TianoCore GitHub Email
> >     Archive Webhook Service.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianoc
> ore-code-review-
> poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0I
> uvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserve
> d=0<https://nam06.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Fti
> anocore-code-review-
> poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951747044&sdata=zbDuJd9lW3K
> oiuzGo6%2BU14Mnude7lZnbCeXQHJnR%2BCg%3D&reserved=0>
>
> Important: as the name says ("-poc"), this is a Proof
> of Concept list,
> for now. Once we're ready to switch over, I'll file an
> internal ticket
> at RH to either rename the list, or (which is probably
> better) to create
> a new list (no "-poc" suffix).
>
> The second option seems more useful because then the
> webhook development
> / bugfixing (if any) could perhaps occur in parallel to
> the normal edk2
> workflow.
>
> >
> >   * Review pull requests and commits assigned by the
> TianoCore GitHub Email
> >     Archive Webhook Service and use the GitHub web UI
> to provide all review
> >     feedback.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fcollaboratin
> g-with-issues-and-pull-requests%2Freviewing-changes-in-
> pull-
> requests&amp;data=02%7C01%7Cbret.barkelew%40microsoft.c
> om%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141a
> f91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdat
> a=51Ljm3wUbBTWT8hcaBD1ZQznSROvAQqnoTzQmD6K%2FLY%3D&amp;
> reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.outlook.c%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=%2BfY0MvXLLyf%2Bt7w2VFkWgfnEGFjgqdJo7WsZmQN8yLE%3D&amp;reserved=0
> om/?url=https%3A%2F%2Fhelp.github.com%2Fen%2Fgithub%2Fc
> ollaborating-with-issues-and-pull-requests%2Freviewing-
> changes-in-pull-
> requests&data=02%7C01%7CBret.Barkelew%40microsoft.com%7
> Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=Iav2d2
> fP2vw9mS9uA76tN3uQcIbl52D6BT963cD7ZC4%3D&reserved=0>
> >
> >   * Wait for Author/Developer to respond to all
> feedback and add commits with
> >     code changes as needed to resolve all feedback.
> This step may be repeated
> >     if the developer/author need to produce multiple
> versions of the patch
> >     series to address all feedback.
>
> (same question about the incremental fixup patches as
> above)
>
> >
> >   * Once all feedback is addressed, add Reviewed-by,
> Acked-by, and Tested-by
> >     responses on individual commits.  Or add Series-
> reviewed-by, Series-acked-by,
> >     or Series-Tested-by responses to the entire pull
> request.
> >
> >   * Wait for Developer/Author to add tags to commit
> messages in the pull request.
> >
> >   * Perform final review of patches and commit
> message tags.  If there are not
> >     issues, set the `push` label to run final set of
> CI checks and auto merge
> >     the pull request into master.
> >
> > # Maintainers.txt Format Changes
> >
> > Add GitHub IDs of all maintainers and reviewers at
> the end of M: and R: lines
> > in [].  For example:
> >
> >     M: Michael D Kinney
> <michael.d.kinney@intel.com<mailto:michael.d.kinney@int
> el.com>> [mdkinney]
> >
> > # TianoCore GitHub Email Archive Webhook Service
> >
> > Assign reviewers to commits in a GitHub pull request
> based on assignments
> > documented in Maintainers.txt and generates an email
> archive of all pull request
> > and code review activities.
>
> s/generates/generate/
>
> (or s/Assign/Assigns/)
>
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-archive-
> webhook&amp;data=02%7C01%7Cbret.barkelew%40microsoft.co
> m%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af
> 91ab2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata
> =7CJNJMEXrxoynjavmEwjzUyRbfNUIZ3FEG4kDRXvhI4%3D&amp;res
> erved=0<https://nam06.safelinks.protection.outlook.com/
> ?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-email-
> archive-
> webhook&data=02%7C01%7CBret.Barkelew%40microsoft.com%7C
> b61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248265951757040&sdata=FihVwcw
> fqewohsmqUECWG36TR1iCQVhVHq02iMZkEL8%3D&reserved=0>
> >
> > # Email Archive Subscription Service
> >
> > The emails are being delivered to the following
> RedHat email subscription
> > service.  Please subscribe to receive the emails and
> to be able to view the
> > email archives.
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Ftianoc
> ore-code-review-
> poc&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C
> 1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C637248227586678640&amp;sdata=q0I
> uvS318pEkJU2td9xX87oIm0LbSlEvOvhpyOOFrE8%3D&amp;reserve
> d=0<https://nam06.safelinks.protection.outlook.com/?url
> =https%3A%2F%2Fwww.redhat.com%2Fmailman%2Flistinfo%2Fti
> anocore-code-review-
> poc&data=02%7C01%7CBret.Barkelew%40microsoft.com%7Cb61c
> e42027c0428ab31408d7f5ebeb5b%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637248265951767033&sdata=482jeizloSf
> EobAk0F8%2BvP9FZptdF6oC90A%2BYSQZcds%3D&reserved=0>
> >
> > The email archives are at this link:
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-
> poc%2Findex.html&amp;data=02%7C01%7Cbret.barkelew%40mic
> rosoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f988b
> f86f141af91ab2d7cd011db47%7C1%7C0%7C637248227586678640&
> amp;sdata=nedUfkmMmI5T6GtAxQCW4q6xt38%2FezeDYmfq6cpRD0M
> %3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection.o%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=ybz14MP2cfsQUGM2JxpsZNq2q%2BZU0e835ZNpKpV9M9Y%3D&amp;reserved=0
> utlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman%
> 2Fprivate%2Ftianocore-code-review-
> poc%2Findex.html&data=02%7C01%7CBret.Barkelew%40microso
> ft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf86f
> 141af91ab2d7cd011db47%7C1%7C0%7C637248265951767033&sdat
> a=78HRihY2z%2Fll0qoC05RMh8I%2Bwri9rwIwblxNBYFPecw%3D&re
> served=0>
> >
> > The following sections show some example pull
> requests and code reviews to
> > help review the generated emails, their contents, and
> threading.
> >
> > ## Email Achieve Thread View
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fthread.html%2300289&amp;data=02%7C01%7Cbret.barke
> lew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275
> 86678640&amp;sdata=GtrEudehfXiSU6ZwH2zKO35CPPPVk0ctZIzh
> kpI6DkE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.prote%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611252057&amp;sdata=ug4axzMxZLKuGz8Eps6xepyEjv1er8atkiOZDpFca2c%3D&amp;reserved=0
> ction.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fm
> ailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fthread.html%2300289&data=02%7C01%7CBret.Barkelew%
> 40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72
> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63724826595177
> 7024&sdata=%2BU8aFUi4kPBeJdlyW%2FERNtp5Fum5gZrWkgCTI8pw
> z14%3D&reserved=0>
> >
> > ## Example patch series with 1 patch
> >
> >
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fthread.html%2300340&amp;data=02%7C01%7Cbret.barke
> lew%40microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275
> 86678640&amp;sdata=ZGpI8%2BzIA9OMFm3pSCc2DQ4F5ZxtDSAXtj
> dFjD%2BY3NA%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.p%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=7pNJG%2BcGgplnp0gOQmPx8ly3IgxLk7DBPW0liL%2BQ97c%3D&amp;reserved=0
> rotection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com
> %2Fmailman%2Fprivate%2Ftianocore-code-review-
> poc%2F2020-
> May%2Fthread.html%2300340&data=02%7C01%7CBret.Barkelew%
> 40microsoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72
> f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63724826595177
> 7024&sdata=gXn%2B9z8MIdnJyUkk2xR1mK%2FL710jUDEeNhK74EUk
> b9A%3D&reserved=0>
> >
> > ## Example patch series with < 10 patches
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00289.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=JyaUyvYfZD7b%2F2wN%2BpS%2B68b%2BwyKoZ3Rba4
> ol%2FyahQVU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.p%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=7pNJG%2BcGgplnp0gOQmPx8ly3IgxLk7DBPW0liL%2BQ97c%3D&amp;reserved=0
> rotection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com
> %2Fmailman%2Fprivate%2Ftianocore-code-review-
> poc%2F2020-
> May%2Fmsg00289.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&s
> data=yvE8%2F%2FKOJUj%2BD4ZmtT3fZPAb2KBsGcAqxt%2FKwRKa6h
> A%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00030.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=bQHIJIQq4Pri8iK3vPxMDMWz%2BKtXcyuPdhr8y7gF
> pXA%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protectio%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=g37erq7Um78njCs8L%2BZ75MBqyGEoZrIEk1epGOJ5va0%3D&amp;reserved=0
> n.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailm
> an%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00030.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951787022&s
> data=zz5BgISidxyQVI%2BQihn%2Fq%2Fv9%2B5%2Fn2vpaeyvFbIiA
> igs%3D&reserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00018.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=uMIRGOq%2BVCOSwDzXkG4yueYS4ZJ7BWfsp3Z4%2B9
> lh6hE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protect%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=GcYd84S79b4L09WB1b40cfbx5y%2FREQnWZsljxNeAm2U%3D&amp;reserved=0
> ion.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmai
> lman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00018.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&s
> data=yN0WkrPdk1EEvyPwlqCG%2B6HTOpa3iT85w2QhXQJ2xYE%3D&r
> eserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00008.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=3CBkdqDxRt4IxtECpWQdKJL%2Bf4HFqqHCXo4loxNT
> zAE%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protectio%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=g37erq7Um78njCs8L%2BZ75MBqyGEoZrIEk1epGOJ5va0%3D&amp;reserved=0
> n.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailm
> an%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00008.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951797016&s
> data=qF2vw8XdQfY0ECclzhbpd60vIgTj2Bzl5RjC%2F4EdbA8%3D&r
> eserved=0>
> >
> > ## Example patch series with > 80 patches
> >
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00198.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=fDfQnifOMzyzLMdP4xH8koKCiSj7ZiuYyrrSZXTf3d
> 4%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=oLtR%2Br3OOn3hqOQ6glrp56aXidbNCpd%2F8w06mY8j7dA%3D&amp;reserved=0.
> outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmailman
> %2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00198.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&s
> data=uXAPx9iTDbQbiv40j3EiqIP51nlFLQ43ghvAsjt4dhs%3D&res
> erved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00116.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=lcxA3tTna%2BdmTpcNMmPlS%2B47llMAcIEjhCEqxV
> 7TDOc%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protect%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=GcYd84S79b4L09WB1b40cfbx5y%2FREQnWZsljxNeAm2U%3D&amp;reserved=0
> ion.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fmai
> lman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00116.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951807013&s
> data=4ANIbH2YDPjN7H4gHGtbgNkAbM9EeVFOf%2FoWqTiXtiA%3D&r
> eserved=0>
> > *
> https://nam06.safelinks.protection.outlook.com/?url=htt
> ps%3A%2F%2Fwww.redhat.com%2Fmailman%2Fprivate%2Ftianoco
> re-code-review-poc%2F2020-
> May%2Fmsg00035.html&amp;data=02%7C01%7Cbret.barkelew%40
> microsoft.com%7C1dcf1f8c03b544e5095408d7f5e2fd56%7C72f9
> 88bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372482275866786
> 40&amp;sdata=CgvZ8e%2B7L4nacvRE35KqEyC%2F1CjDYP6wI10qn%
> 2BoX39Y%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.prote%2F&amp;data=02%7C01%7Cbret.barkelew%40microsoft.com%7C4235599e115f484ba25c08d7f86dfde9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637251023611262052&amp;sdata=m34ZxHM9zr3VFz25TIiYpmxnqR0hPT96UIJppWgU4os%3D&amp;reserved=0
> ction.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fm
> ailman%2Fprivate%2Ftianocore-code-review-poc%2F2020-
> May%2Fmsg00035.html&data=02%7C01%7CBret.Barkelew%40micr
> osoft.com%7Cb61ce42027c0428ab31408d7f5ebeb5b%7C72f988bf
> 86f141af91ab2d7cd011db47%7C1%7C0%7C637248265951817004&s
> data=1jX%2FuPSMlRvzN9eZcv5U7k4JmwfJJwaVhDJ%2BrA6mPYc%3D
> &reserved=0>
> >
> > # Tasks to Complete
> >
> > * Create edk2-codereview repository for evaluation of
> new code review process.
> > * Add GitHub IDs to Maintainers.txt in edk2-
> codereview repository
> > * Update BaseTools/Scripts/GetMaintainer.py to be
> compatible with GitHub IDs at
> >   the end of M: and R: statements
> > * Update webhook to use Rabbit MQ to manage requests
> and emails
> > * Determine if webhook requests must be serialized?
> Current POC is serialized.
> > * Make sure webhook has error handling for all
> unexpected events/states.
> > * Add logging of all events and emails to webhook
>
> The logging sounds very useful, thank you.
>
> Whenever a log message relates to an email, please
> consider logging the
> message-id of that email, if possible.
>
> > * Add admin interface to webhook
> > * Deploy webhook on a production server with 24/7
> support
> >
> > # Ideas for Future Enhancements
> >
> > * Run PatchCheck.py before assigning
> maintainers/reviewers.
> > * Add a simple check that fails if a single patch
> spans more than one package.
>
> Hmmm, good idea in general, but there have been valid
> exceptions to this
> rule.
>
> > * Monitor comments for Reviewed-by, Acked-by, Tested-
> by, Series-Reviewed-by,
> >   Series-Acked-by, Series-Tested-by made by assigned
> maintainers/reviewers.
> >   Once all commits have required tags, auto update
> commit messages in the
> >   branch and wait for maintainer to set the `Push`
> label to run CI and auto
> >   merge if all CI checks pass.
>
> Thank you for writing this up (and for implementing the
> webhook)!
> Laszlo
>
>
>
>
>
>
>





[-- Attachment #2: Type: text/html, Size: 61319 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/14/20 23:26, Bret Barkelew via groups.io wrote:

> It’s code management for the Instagram generation

I find this an extremely good characterization!

And, I find the fact soul-destroying.

Laszlo

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Laszlo Ersek]

On 05/15/20 06:49, Bret Barkelew via groups.io wrote:

> I would far prefer the approach of individual PRs for commits to
> allow for the squash flexibility (and is the strategy I think I would
> pursue with my PRs). For example, the VarPol PR would be broken up
> into 9 PRs for each final commit, and we can get them in one by one.
> Ideally, each one would be a small back and forth and then in. If it
> had been done that way to begin with, it would be over in a week and
> a half or so, rather than the multiple months that we’re now verging
> on.

This differs extremely from how we've been working on edk2-devel (or
from how any git-based project works that I've ever been involved with).
And I think the above workflow is out of scope, for migrating the edk2
process to github.

Again, the structuring of a patch series is a primary trait. Iterating
only on individual patches does not allow for the reordering /
restructuring of the patch series (dropping patches, reordering patches,
inserting patches, moving hunks between patches).

It's common that the necessity to revise an earlier patch emerges while
reworking a later patch. For instance, the git-rebase(1) manual
dedicates a separate section to "splitting commits".

In the initial evaluation of "web forges", Phabricator was one of the
"contestants". Phabricator didn't support the "patch series" concept at
all, it only supported review requests for individual patches, and it
supported setting up dependencies between them. So, for example, a
27-patch series would require 27 submissions and 26 dependencies.

Lacking support for the patch series concept was an immediate deal
breaker with Phabricator.

The longest patch series I've ever submitted to edk2-devel had 58
patches. It was SMM enablement for OVMF. It went from v1 to v5 (v5 was
merged), and the patch count varied significantly:

v1: 58 patches (25 Jul 2015)
v2: 41 patches ( 9 Oct 2015)
v3: 52 patches (15 Oct 2015)
v4: 41 patches ( 3 Nov 2015)
v5: 33 patches (27 Nov 2015)

(The significant drop in the patch count was due to Mike Kinney open
sourcing and upstreaming the *real* PiSmmCpuDxeSmm driver (which was
huge work in its own right), allowing me to drop the Quark-originated
32-bit-only PiSmmCpuDxeSmm variant, from my series.)

The contribution process should make difficult things possible, even if
that complicates simple things somewhat. A process that makes simple
things simple and difficult things impossible is useless. This is what
the Instagram generation seems to be missing.


I don't know why the VariablePolicy work took months. I can see the
following threads on the list:

* [edk2-devel] [PATCH v1 0/9] Add the VariablePolicy feature
  Fri, 10 Apr 2020 11:36:01 -0700

* [edk2-devel] [PATCH v2 00/12] Add the VariablePolicy feature
  Mon, 11 May 2020 23:46:23 -0700

I have two sets of comments:

(1) It's difficult to tell in retrospect (because the series seem to
have been posted with somewhat problematic threading), but the delay
apparently came from multiple sources.

(1a) Review was slow and spotty.

The v1 blurb received some comments in the first week after it was
posted. But the rest of the v1 series (the actual patches) received
feedback like this:

- v1 1/9: no feedback
- v1 2/9: 12 days after posting
- v1 3/9: 16 days after posting
- v1 4/9: no feedback
- v1 5/9: no feedback
- v1 6/9: no feedback
- v1 7/9: no feedback
- v1 8/9: no feedback
- v1 9/9: no feedback

(1b) There was also quite some time between the last response in the v1
thread (Apr 26th, as far as I can see), and the posting of the v2 series
(May 11th).

(1c) The v2 blurb got almost immediate, and numerous feedback (on the
day of posting, and the day after). Regarding the individual patches,
they didn't fare too well:

- v2 01/12: superficial comment on the day of posting from me (not a
            designated MdeModulePkg review), on the day of posting; no
            other feedback thus far
- v2 02/12: ditto
- v2 03/12: no feedback
- v2 04/12: superficial (coding style) comments on the day of posting
- v2 05/12: no feedback
- v2 06/12: no feedback
- v2 07/12: no feedback
- v2 08/12: no feedback
- v2 09/12: no feedback
- v2 10/12: no feedback
- v2 11/12: reasonably in-depth review from responsible co-maintainer
            (yours truly), on the day of posting
- v2 12/12: no feedback

In total, I don't think the current process takes the blame for the
delay. If reviewers don't care (or have no time) now, that problem will
not change with the transition to github.com.


(2) The VariablePolicy series is actually a good example that patch
series restructuring is important.

(2a) The patch count went from 9 (in v1) to 12 (in v2).

(2b) And under v2, Liming still pointed out: "To keep each commit build
pass, the patch set should first add new library instance, then add the
library instance into each platform DSC, last update Variable driver to
consume new library instance."

Furthermore, I requested enabling the feature in ArmVirtPkg too, and
maybe (based on owner feedback) UefiPayloadPkg.

Thus, the v2->v3 update will most likely bring about both patch order
changes, and an increased patch count.

Thanks
Laszlo

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

“… boundless and bare, The lone and level sands stretch far away.”

- Bret

From: Laszlo Ersek<mailto:lersek@redhat.com>
Sent: Friday, May 15, 2020 12:34 AM
To: rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 05/14/20 23:26, Bret Barkelew via groups.io wrote:

> It’s code management for the Instagram generation

I find this an extremely good characterization!

And, I find the fact soul-destroying.

Laszlo


[-- Attachment #2: Type: text/html, Size: 2540 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Bret Barkelew]

[-- Attachment #1: Type: text/plain, Size: 6496 bytes --]

I agree with some of your points, but I don’t believe that this calls for dependencies at all.
If a PR can pass CI with the changes, it’s functionally unordered.
And if  a PR can’t, it has to wait until the PRs that can are in.

This also allows the group to focus on getting one thing done at a time.

I use rebase all the time and agree that it’s very good at precise history management. If a given PR requires that level of control, those tools will always be there.

But just as you say that the simple should not preclude the difficult, the difficult 5% should not needlessly complicated the simple 95%.

For what it’s worth, this is all posturing on my part. I intend – and, indeed, am eager to – follow the process that we’ve been helping Mike to set up.

- Bret

From: Laszlo Ersek<mailto:lersek@redhat.com>
Sent: Friday, May 15, 2020 2:08 AM
To: rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 05/15/20 06:49, Bret Barkelew via groups.io wrote:

> I would far prefer the approach of individual PRs for commits to
> allow for the squash flexibility (and is the strategy I think I would
> pursue with my PRs). For example, the VarPol PR would be broken up
> into 9 PRs for each final commit, and we can get them in one by one.
> Ideally, each one would be a small back and forth and then in. If it
> had been done that way to begin with, it would be over in a week and
> a half or so, rather than the multiple months that we’re now verging
> on.

This differs extremely from how we've been working on edk2-devel (or
from how any git-based project works that I've ever been involved with).
And I think the above workflow is out of scope, for migrating the edk2
process to github.

Again, the structuring of a patch series is a primary trait. Iterating
only on individual patches does not allow for the reordering /
restructuring of the patch series (dropping patches, reordering patches,
inserting patches, moving hunks between patches).

It's common that the necessity to revise an earlier patch emerges while
reworking a later patch. For instance, the git-rebase(1) manual
dedicates a separate section to "splitting commits".

In the initial evaluation of "web forges", Phabricator was one of the
"contestants". Phabricator didn't support the "patch series" concept at
all, it only supported review requests for individual patches, and it
supported setting up dependencies between them. So, for example, a
27-patch series would require 27 submissions and 26 dependencies.

Lacking support for the patch series concept was an immediate deal
breaker with Phabricator.

The longest patch series I've ever submitted to edk2-devel had 58
patches. It was SMM enablement for OVMF. It went from v1 to v5 (v5 was
merged), and the patch count varied significantly:

v1: 58 patches (25 Jul 2015)
v2: 41 patches ( 9 Oct 2015)
v3: 52 patches (15 Oct 2015)
v4: 41 patches ( 3 Nov 2015)
v5: 33 patches (27 Nov 2015)

(The significant drop in the patch count was due to Mike Kinney open
sourcing and upstreaming the *real* PiSmmCpuDxeSmm driver (which was
huge work in its own right), allowing me to drop the Quark-originated
32-bit-only PiSmmCpuDxeSmm variant, from my series.)

The contribution process should make difficult things possible, even if
that complicates simple things somewhat. A process that makes simple
things simple and difficult things impossible is useless. This is what
the Instagram generation seems to be missing.


I don't know why the VariablePolicy work took months. I can see the
following threads on the list:

* [edk2-devel] [PATCH v1 0/9] Add the VariablePolicy feature
  Fri, 10 Apr 2020 11:36:01 -0700

* [edk2-devel] [PATCH v2 00/12] Add the VariablePolicy feature
  Mon, 11 May 2020 23:46:23 -0700

I have two sets of comments:

(1) It's difficult to tell in retrospect (because the series seem to
have been posted with somewhat problematic threading), but the delay
apparently came from multiple sources.

(1a) Review was slow and spotty.

The v1 blurb received some comments in the first week after it was
posted. But the rest of the v1 series (the actual patches) received
feedback like this:

- v1 1/9: no feedback
- v1 2/9: 12 days after posting
- v1 3/9: 16 days after posting
- v1 4/9: no feedback
- v1 5/9: no feedback
- v1 6/9: no feedback
- v1 7/9: no feedback
- v1 8/9: no feedback
- v1 9/9: no feedback

(1b) There was also quite some time between the last response in the v1
thread (Apr 26th, as far as I can see), and the posting of the v2 series
(May 11th).

(1c) The v2 blurb got almost immediate, and numerous feedback (on the
day of posting, and the day after). Regarding the individual patches,
they didn't fare too well:

- v2 01/12: superficial comment on the day of posting from me (not a
            designated MdeModulePkg review), on the day of posting; no
            other feedback thus far
- v2 02/12: ditto
- v2 03/12: no feedback
- v2 04/12: superficial (coding style) comments on the day of posting
- v2 05/12: no feedback
- v2 06/12: no feedback
- v2 07/12: no feedback
- v2 08/12: no feedback
- v2 09/12: no feedback
- v2 10/12: no feedback
- v2 11/12: reasonably in-depth review from responsible co-maintainer
            (yours truly), on the day of posting
- v2 12/12: no feedback

In total, I don't think the current process takes the blame for the
delay. If reviewers don't care (or have no time) now, that problem will
not change with the transition to github.com.


(2) The VariablePolicy series is actually a good example that patch
series restructuring is important.

(2a) The patch count went from 9 (in v1) to 12 (in v2).

(2b) And under v2, Liming still pointed out: "To keep each commit build
pass, the patch set should first add new library instance, then add the
library instance into each platform DSC, last update Variable driver to
consume new library instance."

Furthermore, I requested enabling the feature in ArmVirtPkg too, and
maybe (based on owner feedback) UefiPayloadPkg.

Thus, the v2->v3 update will most likely bring about both patch order
changes, and an increased patch count.

Thanks
Laszlo


[-- Attachment #2: Type: text/html, Size: 9419 bytes --]

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Rebecca Cran]

On 5/15/20 1:34 AM, Laszlo Ersek wrote:

> On 05/14/20 23:26, Bret Barkelew via groups.io wrote:
>
>> It’s code management for the Instagram generation
> I find this an extremely good characterization!
>
> And, I find the fact soul-destroying.

I was working on a web project recently, and apparently people don't 
even check email any more! So someone had set up a Slack channel where 
Github pull requests were posted/linked, and we were supposed to react 
with thumbs-up, "OK" etc. emoji to indicate we'd seen/reviewed/accepted 
the request.


-- 

Rebecca Cran

Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Philippe Mathieu-Daudé]

Hi Bret,

On 5/15/20 5:43 PM, Bret Barkelew via groups.io wrote:
> I agree with some of your points, but I don’t believe that this calls for dependencies at all.

Which points are you disagreeing?

> If a PR can pass CI with the changes, it’s functionally unordered.
> And if  a PR can’t, it has to wait until the PRs that can are in.
> 
> This also allows the group to focus on getting one thing done at a time.
> 
> I use rebase all the time and agree that it’s very good at precise history management. If a given PR requires that level of control, those tools will always be there.
> 
> But just as you say that the simple should not preclude the difficult, the difficult 5% should not needlessly complicated the simple 95%.
> 
> For what it’s worth, this is all posturing on my part. I intend – and, indeed, am eager to – follow the process that we’ve been helping Mike to set up.
> 
> - Bret
> 
> From: Laszlo Ersek<mailto:lersek@redhat.com>
> Sent: Friday, May 15, 2020 2:08 AM
> To: rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D<mailto:michael.d.kinney@intel.com>
> Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process
> 
> On 05/15/20 06:49, Bret Barkelew via groups.io wrote:
> 
>> I would far prefer the approach of individual PRs for commits to
>> allow for the squash flexibility (and is the strategy I think I would
>> pursue with my PRs). For example, the VarPol PR would be broken up
>> into 9 PRs for each final commit, and we can get them in one by one.
>> Ideally, each one would be a small back and forth and then in. If it
>> had been done that way to begin with, it would be over in a week and
>> a half or so, rather than the multiple months that we’re now verging
>> on.
> 
> This differs extremely from how we've been working on edk2-devel (or
> from how any git-based project works that I've ever been involved with).
> And I think the above workflow is out of scope, for migrating the edk2
> process to github.
> 
> Again, the structuring of a patch series is a primary trait. Iterating
> only on individual patches does not allow for the reordering /
> restructuring of the patch series (dropping patches, reordering patches,
> inserting patches, moving hunks between patches).
> 
> It's common that the necessity to revise an earlier patch emerges while
> reworking a later patch. For instance, the git-rebase(1) manual
> dedicates a separate section to "splitting commits".
> 
> In the initial evaluation of "web forges", Phabricator was one of the
> "contestants". Phabricator didn't support the "patch series" concept at
> all, it only supported review requests for individual patches, and it
> supported setting up dependencies between them. So, for example, a
> 27-patch series would require 27 submissions and 26 dependencies.
> 
> Lacking support for the patch series concept was an immediate deal
> breaker with Phabricator.
> 
> The longest patch series I've ever submitted to edk2-devel had 58
> patches. It was SMM enablement for OVMF. It went from v1 to v5 (v5 was
> merged), and the patch count varied significantly:
> 
> v1: 58 patches (25 Jul 2015)
> v2: 41 patches ( 9 Oct 2015)
> v3: 52 patches (15 Oct 2015)
> v4: 41 patches ( 3 Nov 2015)
> v5: 33 patches (27 Nov 2015)
> 
> (The significant drop in the patch count was due to Mike Kinney open
> sourcing and upstreaming the *real* PiSmmCpuDxeSmm driver (which was
> huge work in its own right), allowing me to drop the Quark-originated
> 32-bit-only PiSmmCpuDxeSmm variant, from my series.)
> 
> The contribution process should make difficult things possible, even if
> that complicates simple things somewhat. A process that makes simple
> things simple and difficult things impossible is useless. This is what
> the Instagram generation seems to be missing.
> 
> 
> I don't know why the VariablePolicy work took months. I can see the
> following threads on the list:
> 
> * [edk2-devel] [PATCH v1 0/9] Add the VariablePolicy feature
>    Fri, 10 Apr 2020 11:36:01 -0700
> 
> * [edk2-devel] [PATCH v2 00/12] Add the VariablePolicy feature
>    Mon, 11 May 2020 23:46:23 -0700
> 
> I have two sets of comments:
> 
> (1) It's difficult to tell in retrospect (because the series seem to
> have been posted with somewhat problematic threading), but the delay
> apparently came from multiple sources.
> 
> (1a) Review was slow and spotty.

IIUC it is easier for the "Instagram generation" to write a GitHub
plugin which ping an unmerged pullrequest for them, rather than tracking
their WiP and send a "ping" via an email client.

That reminds me of the Prophet tool:

Prophet: The first generate-and-validate tool that uses machine learning
techniques to learn useful knowledge from past human patches to
recognize correct patches. It is evaluated on the same benchmark as
GenProg and generate correct patches (i.e., equivalent to human patches)
for 18 out of 69 cases.

https://groups.csail.mit.edu/pac/patchgen/
https://en.wikipedia.org/wiki/Automatic_bug_fixing#C [8]

Use it as source, combined with a fuzzer that open GH pull-requests, and 
see if a patch get merged... /s

> 
> The v1 blurb received some comments in the first week after it was
> posted. But the rest of the v1 series (the actual patches) received
> feedback like this:
> 
> - v1 1/9: no feedback
> - v1 2/9: 12 days after posting
> - v1 3/9: 16 days after posting
> - v1 4/9: no feedback
> - v1 5/9: no feedback
> - v1 6/9: no feedback
> - v1 7/9: no feedback
> - v1 8/9: no feedback
> - v1 9/9: no feedback
> 
> (1b) There was also quite some time between the last response in the v1
> thread (Apr 26th, as far as I can see), and the posting of the v2 series
> (May 11th).
> 
> (1c) The v2 blurb got almost immediate, and numerous feedback (on the
> day of posting, and the day after). Regarding the individual patches,
> they didn't fare too well:
> 
> - v2 01/12: superficial comment on the day of posting from me (not a
>              designated MdeModulePkg review), on the day of posting; no
>              other feedback thus far
> - v2 02/12: ditto
> - v2 03/12: no feedback
> - v2 04/12: superficial (coding style) comments on the day of posting
> - v2 05/12: no feedback
> - v2 06/12: no feedback
> - v2 07/12: no feedback
> - v2 08/12: no feedback
> - v2 09/12: no feedback
> - v2 10/12: no feedback
> - v2 11/12: reasonably in-depth review from responsible co-maintainer
>              (yours truly), on the day of posting
> - v2 12/12: no feedback
> 
> In total, I don't think the current process takes the blame for the
> delay. If reviewers don't care (or have no time) now, that problem will
> not change with the transition to github.com.
> 
> 
> (2) The VariablePolicy series is actually a good example that patch
> series restructuring is important.
> 
> (2a) The patch count went from 9 (in v1) to 12 (in v2).
> 
> (2b) And under v2, Liming still pointed out: "To keep each commit build
> pass, the patch set should first add new library instance, then add the
> library instance into each platform DSC, last update Variable driver to
> consume new library instance."
> 
> Furthermore, I requested enabling the feature in ArmVirtPkg too, and
> maybe (based on owner feedback) UefiPayloadPkg.
> 
> Thus, the v2->v3 update will most likely bring about both patch order
> changes, and an increased patch count.
> 
> Thanks
> Laszlo
> 
> 
> 
> 

Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

[Tomas Pilar (tpilar)]

I actually agree with you, when we migrated from reviewboard to github pull requests, I was sorely disappointed with the PR functionality and ergonomics.

Tomas Pilar

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Rebecca Cran via groups.io
Sent: 14 May 2020 22:47
To: rfc@edk2.groups.io; bret.barkelew@microsoft.com; Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io; lersek@redhat.com
Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process

On 5/14/20 3:26 PM, Bret Barkelew via groups.io wrote:

> I feel like this process is a good compromise. It�s not perfect (frankly, I�m a fan of enforced squash merges, which can maintain bisectability if managed well), but it allows for rapid iteration, ease of contribution, and approaches the workflow that many who have never used email to maintain a project would be familiar with.
>
> It�s code management for the Instagram generation, and I for one welcome our new insect overlords.

Or at least, that's what Microsoft is betting on! :D

Personally, I remain unconvinced about the usability of Github Pull Requests for a project the size of EDK2, but I hope to be proven wrong.


--
Rebecca Cran





IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.