public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
* [edk2-rfc] GitHub Pull Request based Code Review Process
@ 2020-05-09  2:59 Michael D Kinney
  2020-05-09  4:22 ` Ni, Ray
                   ` (3 more replies)
  0 siblings, 4 replies; 30+ messages in thread
From: Michael D Kinney @ 2020-05-09  2:59 UTC (permalink / raw)
  To: devel@edk2.groups.io, rfc@edk2.groups.io, Kinney, Michael D

Hello,

This is a proposal to change from the current email-based code review process to
a GitHub pull request-based code review process for all repositories maintained
in TianoCore.  The current email-based code review process and commit message
requirements are documented in Readme.md or Readme.rst at the root of
repositories along with a few Wiki pages:

* https://github.com/tianocore/edk2/blob/master/ReadMe.rst
* https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Development-Process
* https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers
* https://github.com/tianocore/tianocore.github.io/wiki/Commit-Message-Format
* https://github.com/tianocore/tianocore.github.io/wiki/Commit-Signature-Format

The goal is to post changes by opening a GitHub pull request and perform all
code review activity using the GitHub web interface.  This proposal does not
change any licenses or commit message requirements.  It does require all
developers, maintainers, and reviewers to have GitHub accounts.

One requirement that was collected from previous discussions on this topic is
the need for an email archive of all patches and code review activities.  The
existing GitHub features to produce an email archive were deemed insufficient.
A proof of concept of a GitHub webhook has been implemented to provide the email
archive service.  This email archive is read-only.  You will not be able to send
emails to this archive or reply to emails in the archive.

The sections below provide more details on the proposed GitHub pull request
based code review process, details on the email archive service, and a set of
remaining tasks make the email archive service production quality.  It does not
make sense to support both the existing email-based code review and the GitHub
pull request-based code review at the same time.  Instead, this proposal is to
switch to the GitHub pull request-based code review and retire the email based
code review process on the same date.

The edk2 repository is using GitHub pull requests today to run automated
CI checks on the code changes and allows a maintainer to set the `push` label to
request the changes to be merged if all CI checks pass.  With this proposal,
once the code review is complete and the commit messages have been updated, the
same pull request can be used to perform a final set of CI checks and merge the
changes into the master branch.

I would like to collect feedback on this proposal and the email archive service
over the next two weeks with close of comments on Friday May 22, 2020.  If all
issues and concerns can be addressed, then I would like to see the community
agree to make this change as soon as all remaining tasks are completed.

# TianoCore Repositories to enable

* [edk2](https://github.com/tianocore/edk2)
* [edk2-platforms](https://github.com/tianocore/edk2-platforms)
* [edk2-non-osi](https://github.com/tianocore/edk2-non-osi)
* [edk2-test](https://github.com/tianocore/edk2-test)
* [edk2-libc](https://github.com/tianocore/edk2-libc)
* [edk2-staging](https://github.com/tianocore/edk2-staging)

# GitHub Pull Request Code Review Process

**NOTE**: All steps below use [edk2](https://github.com/tianocore/edk2) as an
example.  Several repositories are supported.

## Author/Developer Steps
  * Create a personal fork of [edk2](https://github.com/tianocore/edk2)

    https://help.github.com/en/github/getting-started-with-github/fork-a-repo

  * Create a new branch from edk2/master in personal fork of edk2 repository.

  * Add set of commits for new feature or bug fix to new branch.  Make sure to
    follow the commit message format requirements.  The only change with this
    RFC is that the Cc: lines to maintainers/reviewers should **not** be added.
    The Cc: lines are still supported, but they should only be used to add
    reviewers that do not have GitHub IDs or are not members of TianoCore.

  * Push branch with new commits to personal fork
  * Create a pull request against TianoCore edk2/master

    https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request

  * If pull request has more than 1 commit, then fill in the pull request title
    and decryption information for Patch #0.  Do not leave defaults.

  * Do not assign reviewers.  The webhook assigns maintainers and reviewers to
    the pull request and each commit in the pull request.

  * If maintainers/reviewers provide feedback that requires changes, then make
    add commits to the current branch with the requested changes.  Once all
    changes are accepted on the current branch, reformulate the patch series and
    commit comments as needed for perform a forced push to the branch in the
    personal fork of the edk2 repository.  This step may be repeated if multiple
    versions of the patch series are required to address all code review
    feedback.

  **OPEN**: How should minimum review period be set?  Labels?

## TianoCore GitHub Email Archive Webhook Service Steps
  * Receive an event that a new pull request was opened
  * Evaluate the files modified by the entire pull request and each commit in
    the pull request and cross references against `Maintainters.txt` in the root
    of the repository to assign maintainers/reviewers to the pull request and
    each commit in the pull request. Individual commit assignments are performed
    by adding a commit comment of the following form:

    [CodeReview] Review-request @mdkinney

  * Generate and sends git patch review emails to the email archive.  Emails
    are also sent to any Cc: tags in the commit messages.

  * If the author/developer performs a forced push to the branch in their
    personal fork of the edk2 repository, then a new set of patch review emails
    with patch series Vx is sent to the email archive and any Cc: tags in commit
    messages.

  * Receive events associated with all code review activities and generate
    and send emails to the email archive that shows all review comments and
    all responses closely matching the email contents seen in the current email
    based code review process.

  * Generate and send email when pull request is merged or closed.

## Maintainer/Reviewer Steps

  * Make sure GitHub configuration is setup to 'Watch' the repositories that
    you have maintainer ship or review responsibilities and that email
    notifications from GitHub are enabled.  This enables email notifications
    when a maintainer/reviewer is assigned to a pull request and individual
    commits.

    https://help.github.com/en/github/managing-subscriptions-and-notifications-on-github/configuring-notifications

  * Subscribe to the email archive associated with the TianoCore GitHub Email
    Archive Webhook Service.

    https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc

  * Review pull requests and commits assigned by the TianoCore GitHub Email
    Archive Webhook Service and use the GitHub web UI to provide all review
    feedback.

    https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-changes-in-pull-requests

  * Wait for Author/Developer to respond to all feedback and add commits with
    code changes as needed to resolve all feedback.  This step may be repeated
    if the developer/author need to produce multiple versions of the patch
    series to address all feedback.

  * Once all feedback is addressed, add Reviewed-by, Acked-by, and Tested-by
    responses on individual commits.  Or add Series-reviewed-by, Series-acked-by,
    or Series-Tested-by responses to the entire pull request.

  * Wait for Developer/Author to add tags to commit messages in the pull request.

  * Perform final review of patches and commit message tags.  If there are not
    issues, set the `push` label to run final set of CI checks and auto merge
    the pull request into master.

# Maintainers.txt Format Changes

Add GitHub IDs of all maintainers and reviewers at the end of M: and R: lines
in [].  For example:

    M: Michael D Kinney <michael.d.kinney@intel.com> [mdkinney]

# TianoCore GitHub Email Archive Webhook Service

Assign reviewers to commits in a GitHub pull request based on assignments
documented in Maintainers.txt and generates an email archive of all pull request
and code review activities.

https://github.com/mdkinney/edk2-email-archive-webhook

# Email Archive Subscription Service

The emails are being delivered to the following RedHat email subscription
service.  Please subscribe to receive the emails and to be able to view the
email archives.

https://www.redhat.com/mailman/listinfo/tianocore-code-review-poc

The email archives are at this link:

https://www.redhat.com/mailman/private/tianocore-code-review-poc/index.html

The following sections show some example pull requests and code reviews to
help review the generated emails, their contents, and threading.

## Email Achieve Thread View

https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00289

## Example patch series with 1 patch

https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/thread.html#00340

## Example patch series with < 10 patches

* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00289.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00030.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00018.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00008.html

## Example patch series with > 80 patches

* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00198.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00116.html
* https://www.redhat.com/mailman/private/tianocore-code-review-poc/2020-May/msg00035.html

# Tasks to Complete

* Create edk2-codereview repository for evaluation of new code review process.
* Add GitHub IDs to Maintainers.txt in edk2-codereview repository
* Update BaseTools/Scripts/GetMaintainer.py to be compatible with GitHub IDs at
  the end of M: and R: statements
* Update webhook to use Rabbit MQ to manage requests and emails
* Determine if webhook requests must be serialized?  Current POC is serialized.
* Make sure webhook has error handling for all unexpected events/states.
* Add logging of all events and emails to webhook
* Add admin interface to webhook
* Deploy webhook on a production server with 24/7 support

# Ideas for Future Enhancements

* Run PatchCheck.py before assigning maintainers/reviewers.
* Add a simple check that fails if a single patch spans more than one package.
* Monitor comments for Reviewed-by, Acked-by, Tested-by, Series-Reviewed-by,
  Series-Acked-by, Series-Tested-by made by assigned maintainers/reviewers.
  Once all commits have required tags, auto update commit messages in the
  branch and wait for maintainer to set the `Push` label to run CI and auto
  merge if all CI checks pass.

Best regards,

Mike



^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process
@ 2020-05-19  7:21 Nate DeSimone
  2020-05-19 16:54 ` Sean
  0 siblings, 1 reply; 30+ messages in thread
From: Nate DeSimone @ 2020-05-19  7:21 UTC (permalink / raw)
  To: rfc@edk2.groups.io, lersek@redhat.com,
	bret.barkelew@microsoft.com, devel@edk2.groups.io,
	Kinney, Michael D


[-- Attachment #1.1: Type: text/plain, Size: 8034 bytes --]

Hi All,



I tend to agree with most of Laszlo's points. Specifically, that moving to pull requests will not fix the fact that maintainers are usually busy people and don't always give feedback in a punctual manner. Like Laszlo, I would also prefer that we do not squash patch series. My biggest reason for not squashing patch series is because when you put everything into a single commit, I have had to review commits with 500+ files changed. Opening git difftool on a commit like that is awful.



However, I would like to register my general endorsement for pull requests or some other web based system of code review… and I don’t have an Instagram account by the way :) Personally, I prefer Gerrit as I use it a lot with coreboot and other projects. But since we are using Github for hosting, pull requests are an easy switch and a logical choice. My main reason for being excited about pull requests mostly has to do with the amount of manual effort required to be a TianoCore maintainer right now. I have set up my email filter so that the mailing list is categorized like so:



[cid:image001.png@01D62D71.502B55E0]



Implementing the logic to parse the contents of emails to categorize them like this required me to define no less than 12 email filter rules in Microsoft Outlook, and I have to change my filtering logic every time I am added/removed from a Maintainers.txt file. I’m sure every other maintainer has spent a time separately implementing filtering logic like I have. This helps, but still for every thread, I have to go and check if one of the other maintainers has already reviewed/pushed that patch series yet, and if not review/push it. If I have ] feedback on a patch series, I have to categorize it as awaiting response from author and check up on it from time to time, sometimes I ping the author directly and remind them to send a new patch series. Implementing this state machine is a lot of manual work and it kind of feels like I’m a telephone operator in the 1950s. I greatly welcome automation here as I am sure it will increase the number of patch series I am able to review per hour.



Thanks,

Nate



-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Laszlo Ersek
Sent: Friday, May 15, 2020 2:08 AM
To: rfc@edk2.groups.io; bret.barkelew@microsoft.com; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] [edk2-rfc] GitHub Pull Request based Code Review Process



On 05/15/20 06:49, Bret Barkelew via groups.io wrote:



> I would far prefer the approach of individual PRs for commits to allow

> for the squash flexibility (and is the strategy I think I would pursue

> with my PRs). For example, the VarPol PR would be broken up into 9 PRs

> for each final commit, and we can get them in one by one.

> Ideally, each one would be a small back and forth and then in. If it

> had been done that way to begin with, it would be over in a week and a

> half or so, rather than the multiple months that we’re now verging

> on.



This differs extremely from how we've been working on edk2-devel (or from how any git-based project works that I've ever been involved with).

And I think the above workflow is out of scope, for migrating the edk2 process to github.



Again, the structuring of a patch series is a primary trait. Iterating only on individual patches does not allow for the reordering / restructuring of the patch series (dropping patches, reordering patches, inserting patches, moving hunks between patches).



It's common that the necessity to revise an earlier patch emerges while reworking a later patch. For instance, the git-rebase(1) manual dedicates a separate section to "splitting commits".



In the initial evaluation of "web forges", Phabricator was one of the "contestants". Phabricator didn't support the "patch series" concept at all, it only supported review requests for individual patches, and it supported setting up dependencies between them. So, for example, a 27-patch series would require 27 submissions and 26 dependencies.



Lacking support for the patch series concept was an immediate deal breaker with Phabricator.



The longest patch series I've ever submitted to edk2-devel had 58 patches. It was SMM enablement for OVMF. It went from v1 to v5 (v5 was merged), and the patch count varied significantly:



v1: 58 patches (25 Jul 2015)

v2: 41 patches ( 9 Oct 2015)

v3: 52 patches (15 Oct 2015)

v4: 41 patches ( 3 Nov 2015)

v5: 33 patches (27 Nov 2015)



(The significant drop in the patch count was due to Mike Kinney open sourcing and upstreaming the *real* PiSmmCpuDxeSmm driver (which was huge work in its own right), allowing me to drop the Quark-originated 32-bit-only PiSmmCpuDxeSmm variant, from my series.)



The contribution process should make difficult things possible, even if that complicates simple things somewhat. A process that makes simple things simple and difficult things impossible is useless. This is what the Instagram generation seems to be missing.





I don't know why the VariablePolicy work took months. I can see the following threads on the list:



* [edk2-devel] [PATCH v1 0/9] Add the VariablePolicy feature

  Fri, 10 Apr 2020 11:36:01 -0700



* [edk2-devel] [PATCH v2 00/12] Add the VariablePolicy feature

  Mon, 11 May 2020 23:46:23 -0700



I have two sets of comments:



(1) It's difficult to tell in retrospect (because the series seem to have been posted with somewhat problematic threading), but the delay apparently came from multiple sources.



(1a) Review was slow and spotty.



The v1 blurb received some comments in the first week after it was posted. But the rest of the v1 series (the actual patches) received feedback like this:



- v1 1/9: no feedback

- v1 2/9: 12 days after posting

- v1 3/9: 16 days after posting

- v1 4/9: no feedback

- v1 5/9: no feedback

- v1 6/9: no feedback

- v1 7/9: no feedback

- v1 8/9: no feedback

- v1 9/9: no feedback



(1b) There was also quite some time between the last response in the v1 thread (Apr 26th, as far as I can see), and the posting of the v2 series (May 11th).



(1c) The v2 blurb got almost immediate, and numerous feedback (on the day of posting, and the day after). Regarding the individual patches, they didn't fare too well:



- v2 01/12: superficial comment on the day of posting from me (not a

            designated MdeModulePkg review), on the day of posting; no

            other feedback thus far

- v2 02/12: ditto

- v2 03/12: no feedback

- v2 04/12: superficial (coding style) comments on the day of posting

- v2 05/12: no feedback

- v2 06/12: no feedback

- v2 07/12: no feedback

- v2 08/12: no feedback

- v2 09/12: no feedback

- v2 10/12: no feedback

- v2 11/12: reasonably in-depth review from responsible co-maintainer

            (yours truly), on the day of posting

- v2 12/12: no feedback



In total, I don't think the current process takes the blame for the delay. If reviewers don't care (or have no time) now, that problem will not change with the transition to github.com.





(2) The VariablePolicy series is actually a good example that patch series restructuring is important.



(2a) The patch count went from 9 (in v1) to 12 (in v2).



(2b) And under v2, Liming still pointed out: "To keep each commit build pass, the patch set should first add new library instance, then add the library instance into each platform DSC, last update Variable driver to consume new library instance."



Furthermore, I requested enabling the feature in ArmVirtPkg too, and maybe (based on owner feedback) UefiPayloadPkg.



Thus, the v2->v3 update will most likely bring about both patch order changes, and an increased patch count.



Thanks

Laszlo









[-- Attachment #1.2: Type: text/html, Size: 15040 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 2968 bytes --]

^ permalink raw reply	[flat|nested] 30+ messages in thread
end of thread, other threads:[~2020-05-26 10:08 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-05-09  2:59 [edk2-rfc] GitHub Pull Request based Code Review Process Michael D Kinney
2020-05-09  4:22 ` Ni, Ray
2020-05-11 17:30   ` Michael D Kinney
2020-05-11 19:47   ` [edk2-devel] " Laszlo Ersek
2020-05-09 18:24 ` Rebecca Cran
2020-05-10 21:29   ` Michael D Kinney
2020-05-10 21:43     ` Rebecca Cran
2020-05-11  1:37       ` Michael D Kinney
2020-05-11 20:05         ` Laszlo Ersek
2020-05-11 20:00       ` Laszlo Ersek
2020-05-11 19:50     ` Laszlo Ersek
2020-05-11 17:27 ` Michael D Kinney
2020-05-11 19:39 ` [edk2-devel] " Laszlo Ersek
2020-05-11 20:09   ` [EXTERNAL] " Bret Barkelew
2020-05-11 20:43     ` Michael D Kinney
2020-05-14 21:26       ` Bret Barkelew
2020-05-14 21:46         ` Rebecca Cran
2020-05-26 10:08           ` Tomas Pilar (tpilar)
2020-05-15  1:19         ` Michael D Kinney
2020-05-15  4:49           ` Bret Barkelew
2020-05-15  9:07             ` Laszlo Ersek
2020-05-15 15:43               ` Bret Barkelew
2020-05-18 11:48                 ` Philippe Mathieu-Daudé
2020-05-15  7:34         ` [EXTERNAL] " Laszlo Ersek
2020-05-15 15:36           ` Bret Barkelew
2020-05-18  2:29           ` Rebecca Cran
2020-05-11 22:07     ` Laszlo Ersek
  -- strict thread matches above, loose matches on Subject: below --
2020-05-19  7:21 Nate DeSimone
2020-05-19 16:54 ` Sean
2020-05-19 18:02   ` Nate DeSimone
2020-05-19 19:34     ` Bret Barkelew
2020-05-19 19:59       ` Nate DeSimone
2020-05-19 20:10         ` Bret Barkelew
2020-05-20 21:53           ` Laszlo Ersek
2020-05-22  5:31             ` [EXTERNAL] " Bret Barkelew
2020-05-19 20:41   ` Laszlo Ersek
2020-05-19 22:25     ` Sean
2020-05-21 13:30       ` Laszlo Ersek
2020-05-22  2:59         ` Andrew Fish
2020-05-22  5:48           ` [EXTERNAL] " Bret Barkelew
2020-05-22 17:20             ` Laszlo Ersek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox