Digging in even further, it looks like you’re asking about tests across test pipelines, including some of the platform pipelines. I don’t believe there’s a cohesive way to do that, as some of them require platform-specific parameters (such as the --FlashRom parameter for OvmfPkg).
Can you share the specific PR you’re failing so that we can talk in concrete terms?
My guess (from experience failing this myself) is that you are failing the spell check remotely and not running it locally (because Node, npm, or cspell are not installed). It’s possible that we should investigate documenting cross-pipeline and cross-package CI dependencies in a central place. Right now it’s distributed (e.g. did you follow the steps here: https://github.com/tianocore/edk2/tree/master/.pytool#spell-checking---cspell).
- Bret
From: Bret Barkelew via groups.io
Sent: Thursday, March 4, 2021 10:22 AM
To: devel@edk2.groups.io;
rebecca@nuviainc.com; lersek@redhat.com
Cc: Sean Brogan;
Kinney, Michael D
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs
Thinking further, it will skip some tests if you haven’t installed the prerequisites (e.g. spell check). We could potentially add a single script to install the prerequisites, but then there are the prereqs for the prereqs (e.g. Node).
To more clearly address your question, the “tests” that a GitHub CI is broken up into are arbitrary subdivisions of the tests that run locally (when you run “stuart_ci_build” without parameters) so that the workload can be divided amongst
the available worker agents. If you were to compare the individual test logs against your local, monolithic test log, you would find that all cases were covered in both (assuming the aforementioned prereqs are installed).
- Bret
From: Bret Barkelew via groups.io
Sent: Thursday, March 4, 2021 9:57 AM
To: devel@edk2.groups.io;
rebecca@nuviainc.com; lersek@redhat.com
Cc: Sean Brogan;
Kinney, Michael D
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs
*An* answer is:
- There’s a DevOps virtual environment that you can run that will pretend to be a build agent. I played with it years ago, but that was before the CI tooling (and it was years ago) so I don’t know how the capabilities compare.
My real answer is:
- If you run the three commands -- setup, update, and build -- without any parameters other than your TOOL_CHAIN_TAG, it should run all tests on all packages for that particular OS/toolchain.
- Bret
From: Rebecca Cran via groups.io
Sent: Thursday, March 4, 2021 9:45 AM
To: devel@edk2.groups.io;
lersek@redhat.com
Cc: Sean Brogan;
Bret Barkelew; Kinney, Michael D
Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs
On 3/4/21 10:04 AM, Laszlo Ersek wrote:
> (Sigh, my address book pulled one over me -- resending to the correct
> list address now. Please ignore the previous posting that you may have
> gotten "in private".)
>
> Hi All,
>
> while I can run specific CI tests locally, using the commands listed at:
>
>
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0
>
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh%2Bo%3D&reserved=0
>
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M%3D&reserved=0
>
> I don't know how I can determine and drive the whole "test plan" locally.
>
> When I submit a PR to github, "something" generates (say) 83 test cases.
> How can I run that "something" locally?
>
> Assume I have the following:
> - a Linux builder VM, with up-to-date tools,
> - a Windows builder VM, with up-to-date tools,
> - a topic branch, on top of "master".
>
> How do I generate and run those same 83 tests, locally?
>
> It has happened to me that all my local CI steps succeeded (meaning both
> the Windows and the Linux builder), but github still rejected the PR --
> the reason was that I failed to think of a particular test that github
> would run. Composing 83 "stuart" command lines isn't something I'd like
> to do manually.
FWIW it looks like people have been looking for similar functionality of
testing a pipeline locally, and not found it.
e.g.:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XMxZesyKxVN9Wg3SyRpUbnnmKzojevHCqQhzQ1Pc4%2Bg%3D&reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopercommunity.visualstudio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-commit%2F1302759&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gwGZAQJAIXIS0LODFpglrCf5Ua2JeMnNm3A3auCrhx8%3D&reserved=0
--
Rebecca Cran