[PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Michael Kubacki]

From: Michael Kubacki <michael.kubacki@microsoft.com>

The following 6 patches adds support for "Platform CI" for ArmVirtPkg,
OvmfPkg, and EmulatorPkg.

Each readme has live status and links to the builds as well as details
of how to build and run the same way the CI server will.

ArmVirtPkg:
https://github.com/spbrogan/edk2/blob/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v9/ArmVirtPkg/ReadMe.rst

EmulatorPkg:
https://github.com/spbrogan/edk2/blob/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v9/EmulatorPkg/README.rst

OvmfPkg:
https://github.com/spbrogan/edk2/blob/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v9/OvmfPkg/README.rst

The patch set also adds ArmVirtPkg, OvmfPkg, and EmulatorPkg to Core CI for
the code evaluation tests (not compiling). Details of those tests are here:
https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v9/.pytool

Changes for V2:
* PlatformBuild.py, iasl dependency, and azurepipeline files have been
  moved into a PlatformCI folder within the respective package. 
* PlatformBuild.py - RequiredSubmodules function configured to read from
  edk2 .gitmodules to lower required updates.
* ReadMe files have been switched to reStructuredText to make Build Status
  table less distracting when viewing plaintext.

Branch: https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v9

Please send feedback to the mailing list and do not leave feedback directly
on github.

On a separate note, shallow threading might not work on this patch series
due to changes made by the SMTP server. Please bear with me while I am
investigating if this can be changed.

Cc: Andrew Fish <afish@apple.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>

Sean Brogan (6):
  .azurepipelines: Add Platform CI template
  ArmVirtPkg: Add Platform CI and configuration for Core CI
  EmulatorPkg: Add Platform CI and configuration for Core CI
  OvmfPkg: Add Platform CI and configuration for Core CI
  .pytool: Update CI Settings to support Emulator, ArmVirt, and Ovmf
    packages
  .azurepipelines: Update Core CI build matrix to include platforms

 .azurepipelines/ReadMe.md                                 |  50 ++++
 .azurepipelines/templates/ReadMe.md                       |  59 +++++
 .azurepipelines/templates/platform-build-run-steps.yml    | 134 ++++++++++
 .azurepipelines/templates/pr-gate-build-job.yml           |   5 +
 .pytool/CISettings.py                                     |   7 +-
 .pytool/Plugin/SpellCheck/cspell.base.yaml                |  14 +
 .pytool/Readme.md                                         |  10 +-
 ArmVirtPkg/ArmVirtPkg.ci.yaml                             | 103 ++++++++
 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml     |  89 +++++++
 ArmVirtPkg/PlatformCI/PlatformBuild.py                    | 276 ++++++++++++++++++++
 ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml                   |  21 ++
 ArmVirtPkg/ReadMe.rst                                     | 139 ++++++++++
 EmulatorPkg/EmulatorPkg.ci.yaml                           |  85 ++++++
 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml    |  95 +++++++
 EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  85 ++++++
 EmulatorPkg/PlatformCI/PlatformBuild.py                   | 272 +++++++++++++++++++
 EmulatorPkg/README.rst                                    | 175 +++++++++++++
 OvmfPkg/OvmfPkg.ci.yaml                                   |  83 ++++++
 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml        | 133 ++++++++++
 OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml     | 138 ++++++++++
 OvmfPkg/PlatformCI/PlatformBuild.py                       | 254 ++++++++++++++++++
 OvmfPkg/PlatformCI/iasl_ext_dep.yaml                      |  21 ++
 OvmfPkg/README.rst                                        | 211 +++++++++++++++
 23 files changed, 2451 insertions(+), 8 deletions(-)
 create mode 100644 .azurepipelines/ReadMe.md
 create mode 100644 .azurepipelines/templates/ReadMe.md
 create mode 100644 .azurepipelines/templates/platform-build-run-steps.yml
 create mode 100644 ArmVirtPkg/ArmVirtPkg.ci.yaml
 create mode 100644 ArmVirtPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
 create mode 100644 ArmVirtPkg/PlatformCI/PlatformBuild.py
 create mode 100644 ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
 create mode 100644 ArmVirtPkg/ReadMe.rst
 create mode 100644 EmulatorPkg/EmulatorPkg.ci.yaml
 create mode 100644 EmulatorPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
 create mode 100644 EmulatorPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
 create mode 100644 EmulatorPkg/PlatformCI/PlatformBuild.py
 create mode 100644 EmulatorPkg/README.rst
 create mode 100644 OvmfPkg/OvmfPkg.ci.yaml
 create mode 100644 OvmfPkg/PlatformCI/.azurepipelines/Ubuntu-GCC5.yml
 create mode 100644 OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
 create mode 100644 OvmfPkg/PlatformCI/PlatformBuild.py
 create mode 100644 OvmfPkg/PlatformCI/iasl_ext_dep.yaml
 create mode 100644 OvmfPkg/README.rst

-- 
2.16.3.windows.1

Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Laszlo Ersek]

On 04/20/20 21:12, michael.kubacki@outlook.com wrote:

>  ArmVirtPkg/ReadMe.rst                                     | 139 ++++++++++
>  EmulatorPkg/README.rst                                    | 175 +++++++++++++
>  OvmfPkg/README.rst                                        | 211 +++++++++++++++

I made a mistake -- I apologize.

* In my email here (two links to the same message):

    http://mid.mail-archive.com/061ab841-33fb-b46f-a9e7-1b5ce0d2e2f4@redhat.com
    https://edk2.groups.io/g/devel/message/57468

  I gladly accepted the option offered by Sean that we put the
  CI-specific README files (for at least OvmfPkg and ArmVirtPkg) under
  the PlatformCI subdirectory. Quote:

On 04/16/20 16:51, Laszlo Ersek wrote:
> On 04/15/20 22:38, sean.brogan via [] wrote:
>> If that isn't desirable i would at least suggest we change the title
>> to just ReadMe.md so that GitHub shows it by default when the
>> PaltformCI folder is viewed form the web or in editor like vscode.
>
> This sounds 100% viable and great to me. I didn't expect this could
> work! (I'm generally unaware of the readme filename patterns, and
> locations, that github.com recognizes; sorry about that.) Having an
> "unadorned" ReadMe.md file under PlatformCI is just perfect.
>
> So if I understand correctly, we could choose:
>
>   ArmVirtPkg/ArmVirtPkg.ci.yaml
>   ArmVirtPkg/PlatformCI/PlatformBuild.py
>   ArmVirtPkg/PlatformCI/ReadMe.md
>   ArmVirtPkg/PlatformCI/Ubuntu-GCC5.yml
>   ArmVirtPkg/PlatformCI/iasl_ext_dep.yaml
>
> Do I understand right?
>
> Because, I'd find this great!

  Back then, the format for the readme was assumed MD, but that wasn't
  the main topic -- the main topic was the directory layout.

* Then, when the MD vs. RST format question came up, I checked the RST
  format (the *contents*), and I was happy with it. My email (two links
  again):

    http://mid.mail-archive.com/1be6a25d-06e2-4109-8acf-0fe82d150af4@redhat.com
    https://edk2.groups.io/g/devel/message/57611

  *However*, unfortunately, when looking at the RST format improvements,
  I missed that Sean implemented the RST files directly in the package
  root directories; and *not* under the PlatformCI directory. I'm very
  sorry for missing that -- I assumed that the text files were under
  PlatformCI like I had requested / agreed earlier, and I didn't realize
  from the URLs that Sean pasted that this wasn't the case.


So, I have two observations / requests:

(1) The three RST file names are capitalized differently (two use ALL
CAPS, one uses CamelCase), before the .rst suffix. Please stick with one
style.

(2) I'd really like these RST text files to be moved under the
respective PlatformCI directories. (My understanding is, from Sean's
message quoted above, that this is possible, and github can display it
well when the user browses the PlatformCI directory in a web browser.)

With those updates, for the OvmfPkg and ArmVirtPkg patches:

Acked-by: Laszlo Ersek <lersek@redhat.com>

Thanks!
Laszlo

Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Sean]

Thanks for providing feedback.  
My V10 branch is up which will be sent to the mailing list tomorrow as the V3 patch set unless there are any changes requested.

This now includes a 7th patch which converts the edk2 ReadMe.md to ReadMe.rst and shows the Platform CI status.  The other package ReadMe files that describe how to run locally have been updated to remove the status and some text changed.  Finally they were moved into the PlatformCI folder.  This should now align with Laszlo's requests.  

https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v10

Thanks
Sean



-----Original Message-----
From: Ard Biesheuvel <ard.biesheuvel@arm.com> 
Sent: Thursday, April 23, 2020 1:42 AM
To: Sean Brogan <sean.brogan@microsoft.com>; Laszlo Ersek <lersek@redhat.com>; michael.kubacki@outlook.com; devel@edk2.groups.io
Cc: Andrew Fish <afish@apple.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Jordan Justen <jordan.l.justen@intel.com>; Leif Lindholm <leif@nuviainc.com>; Liming Gao <liming.gao@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Ray Ni <ray.ni@intel.com>
Subject: Re: [EXTERNAL] Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

On 4/23/20 2:45 AM, Sean Brogan wrote:
> I was hoping that because it wasn't "HTML tag soup" that build status could be front and center in the package readme as I find that more in line with expectations on github based projects.   Nesting it deeper in the package just means less people find it when looking at your package.   But I could see a few ways to solve it so I am looking for feedback and alignment before doing anything else.
> 
> Option 1.
> 
> 1. Add the Platform CI build status for the three platforms to the Edk2 repo readme since these are platforms in edk2 this makes sense and brings the most visibility to their status and the existence of the builds.
> 2. Convert edk2 repo readme to RST so it avoids "HTML tag soup"
> 3. Move the remaining part of the OvmfPkg/Readme.rst file into OvmfPkg/PlatformCI/Readme.md  (I prefer MD when not doing large tables of links as it is easier to write and slightly easier to read as plain text).
> 4. Do step 3 for ArmVirtPkg and EmulatorPkg too
> 
> Option 2.
> 1. Leave the ReadMe.rst at the root of each package but only include 
> the build status table and add a link to the PlatformCi/ReadMe.md 2. 
> Move the Pytool and Platform CI focused readme contents to 
> PlatformCI/ReadMe.md
> 
> Option 3.
> 
> Something else.
> 

Given the intent to make it a requirement that changes don't regress on these platforms we've selected, I think having the CI build (and boot!) status of these platforms front and center makes sense.

I am absolutely fine with using either .md or .rst depending on the needs of the individual use cases.

So I suppose that means option #1, or some variation thereof.

> Finally does anyone have a strong preference for ReadMe vs README vs readme vs Readme vs ReAdMe?  I see a mix of these in the edk2 code tree.
> 

ReadMe is the most idiomatic given the EDK2 coding style, but I could live with README as well.

Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Laszlo Ersek]

On 04/24/20 07:55, Sean Brogan wrote:
> Thanks for providing feedback.  
> My V10 branch is up which will be sent to the mailing list tomorrow as the V3 patch set unless there are any changes requested.
> 
> This now includes a 7th patch which converts the edk2 ReadMe.md to ReadMe.rst and shows the Platform CI status.  The other package ReadMe files that describe how to run locally have been updated to remove the status and some text changed.  Finally they were moved into the PlatformCI folder.  This should now align with Laszlo's requests.  
> 
> https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v10

For your commit 3ce12a4eaa4d ("ArmVirtPkg: Add Platform CI and
configuration for Core CI", 2020-04-23):

Acked-by: Laszlo Ersek <lersek@redhat.com>

For your commit bdd9493c68a7 ("OvmfPkg: Add Platform CI and
configuration for Core CI", 2020-04-23):

Acked-by: Laszlo Ersek <lersek@redhat.com>

Thanks!
Laszlo

Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Laszlo Ersek]

On 04/24/20 14:12, Laszlo Ersek wrote:
> On 04/24/20 07:55, Sean Brogan wrote:
>> Thanks for providing feedback.  
>> My V10 branch is up which will be sent to the mailing list tomorrow as the V3 patch set unless there are any changes requested.
>>
>> This now includes a 7th patch which converts the edk2 ReadMe.md to ReadMe.rst and shows the Platform CI status.  The other package ReadMe files that describe how to run locally have been updated to remove the status and some text changed.  Finally they were moved into the PlatformCI folder.  This should now align with Laszlo's requests.  
>>
>> https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v10
> 
> For your commit 3ce12a4eaa4d ("ArmVirtPkg: Add Platform CI and
> configuration for Core CI", 2020-04-23):
> 
> Acked-by: Laszlo Ersek <lersek@redhat.com>
> 
> For your commit bdd9493c68a7 ("OvmfPkg: Add Platform CI and
> configuration for Core CI", 2020-04-23):
> 
> Acked-by: Laszlo Ersek <lersek@redhat.com>

For your commit d5e4e7a6d5e2 ("ReadMe: Convert to rst and add Platform
CI Status", 2020-04-23):

Reviewed-by: Laszlo Ersek <lersek@redhat.com>

as I've made a reasonable effort to proof-read it, in plaintext (not
rendered) format, comparing it to the format being replaced.

Thanks
Laszlo

Re: [PATCH v2 0/6] Edk2 Platform and Core CI for ArmVirtPkg, EmulatorPkg, and OvmfPkg

[Ard Biesheuvel]

On 4/24/20 2:12 PM, Laszlo Ersek wrote:
> On 04/24/20 07:55, Sean Brogan wrote:
>> Thanks for providing feedback.
>> My V10 branch is up which will be sent to the mailing list tomorrow as the V3 patch set unless there are any changes requested.
>>
>> This now includes a 7th patch which converts the edk2 ReadMe.md to ReadMe.rst and shows the Platform CI status.  The other package ReadMe files that describe how to run locally have been updated to remove the status and some text changed.  Finally they were moved into the PlatformCI folder.  This should now align with Laszlo's requests.
>>
>> https://github.com/spbrogan/edk2/tree/PlatformAndCoreCIForOvmfArmVirtEmulatorPackages_v10
> 
> For your commit 3ce12a4eaa4d ("ArmVirtPkg: Add Platform CI and
> configuration for Core CI", 2020-04-23):
> 
> Acked-by: Laszlo Ersek <lersek@redhat.com>
> 
> For your commit bdd9493c68a7 ("OvmfPkg: Add Platform CI and
> configuration for Core CI", 2020-04-23):
> 
> Acked-by: Laszlo Ersek <lersek@redhat.com>
> 

Likewise.

For the 7 patches on the branch above, at commit d5e4e7a6d5e2823fcda5, 
wherever it matters (or where it doesn't)

Acked-by: Ard Biesheuvel <ard.biesheuvel@arm.com>