From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (NAM02-DM3-obe.outbound.protection.outlook.com [40.107.95.124]) by mx.groups.io with SMTP id smtpd.web08.269.1614882771676187995 for ; Thu, 04 Mar 2021 10:32:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=d20s7Kl6; spf=pass (domain: microsoft.com, ip: 40.107.95.124, mailfrom: bret.barkelew@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oSUy9237aWKFCQOp2wKimht4u6+P7/DkEmv1TM7CgGOSRKQDq394WxIoWm282rfFjmb/34jYj/Fdt3/TbvCeu4Vjm77eJjWwEhGyGl3K69j8onIO9ZB2BHAKO/yyq9YKL08D+q91B3oJBGg2L2VEo/h2wsS1xLBgvQMgqkObCmqxj5+puVAeKZalNVEljKc44oKXgywbQdak3z+NjLeVoMoE06gt+lQkiBhk7m+KHa6XBE+ydH6BS8J4shtJkF6MCR9ZPmUqXZ2lztF+t25Gza6IUY+3wr3oiMQfq4hkRsZgatPGuo3/mUr+SoDFsCvf/WMpfe/iR1kGoD4byvJmkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sF9mjokDDlXCY2bEL5mkDY+xkE8IVj4YnRFyVsDrt28=; b=Esemf6nzughYg1JMlL48BS20M452Jdcuc2wcQPdiBmvP2ARgqBNaaQV6eDCOGKr1+ORVO3p9x8TGzOIP5GP4ciNBU1ujX4VuRe/btCsi/ImVe5t5V1iZn9g4QhWATM1JuKwBvuvtWWpMEs3rL2PIFyDQH8XxvOw14fFp4KZi05FDJwtoLhta2LprG9zQj8Ep5JFOOOZc9Cypa49UKkQoNgIJFbtVUZoW3oWMuL8cNAOkIBU5Dyc2Nvy82H1H8EpZ7lLxF1Sx9YQCWIEchUKqvJQZlH7fZC3Y30stBv6u6BzOyHcUlNSyxlakq6rxeY1f2ViIdePhCRJaW88meJibeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sF9mjokDDlXCY2bEL5mkDY+xkE8IVj4YnRFyVsDrt28=; b=d20s7Kl6mTRd55xDdB1IRx8eRDN7UPutCV4IqhEVQpcp50cZN+D5DRJcj+Cbk0E9phUqhKH/01keYjnXbFug/CvwPYmsbOmtT3pF9lDlOq7QX1sjq9pJgGKcI9Kc2eIbIMkrI61eGP7wOLSWgZzbvjITYwl6cOpeGPoqlt64HAQ= Received: from MW4PR21MB1907.namprd21.prod.outlook.com (2603:10b6:303:71::8) by MWHPR21MB0846.namprd21.prod.outlook.com (2603:10b6:300:77::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.12; Thu, 4 Mar 2021 18:32:47 +0000 Received: from MW4PR21MB1907.namprd21.prod.outlook.com ([fe80::8ddb:dbee:7757:2d6a]) by MW4PR21MB1907.namprd21.prod.outlook.com ([fe80::8ddb:dbee:7757:2d6a%7]) with mapi id 15.20.3933.018; Thu, 4 Mar 2021 18:32:47 +0000 From: "Bret Barkelew" To: "devel@edk2.groups.io" , "rebecca@nuviainc.com" , "lersek@redhat.com" CC: Sean Brogan , "Kinney, Michael D" Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs Thread-Topic: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs Thread-Index: AQHXER4pxmuNHM2d60+oVWyzGXc9hqp0HKQOgAAFP9WAAAOdUw== Date: Thu, 4 Mar 2021 18:32:47 +0000 Message-ID: References: <95b5738d-6af2-15ea-e69d-759b1599b00d@redhat.com>,,, In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-03-04T17:54:35.8737457Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Privileged authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=microsoft.com; x-originating-ip: [71.212.153.143] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 9323a591-827f-470d-dcf8-08d8df3be89a x-ms-traffictypediagnostic: MWHPR21MB0846: x-ms-exchange-transport-forked: True x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: WTZPQHKZdElfmCBNsyCmT7nCGub+9MHpOmdgkfXrlQsAWf3rmeegSBdd/W7pl/rIQWZEnXj9JDyv270mWA1+t5P6wfJFQxIp+7UyBmZfvibsC4Kc4OahyHWgIH7kdIWWZwjZTFeHEPk7DlW1kTaawp7SQu/UENAdovj6mNmgg7ENLuMJIumQN2D/lqWiHacI4KC7ggvLIJPkUPbsA4DaiL4wpLbbHqE4H/6yKDjCgcH5k9wGDXEHT1+K4bdYpmD9x5HdYTd5wE809Kxuw1BIEb2yAfXz+gNe/duy2s51WnPJFlSv82YY2deOuvqyyeu7zfL5hh4btc5OmdeprY9rb4OWGagXrcfoQMCqohnn97Bm+AV64ss02KzxB5uKjfDMgyoao93d5aiZYiJCkM7ueMoEedO/hYc4wkosOwWV4HjcLoBcTsnyG99ye23tjbO22sR0+EAgt2OvZxTVd9xd1OI89LjtuJYlevg6MBA/FWSqnRO/abflJue/kE0jAkxr7uO8u1fr0+HIKP/nz9TxFW/RyO5Y6iWKWoEZSYlG931A3XCwXM5pzrA5ZT66DnQE4SGNk7U6QA4L7EYyiPD4vQXu76gF3DRmZT85GGTzybII/VOwRX3zrvLOvApLYYvxCuuEcP6lEmhmPrxvoyukcw== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW4PR21MB1907.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(366004)(396003)(376002)(346002)(39860400002)(76116006)(8990500004)(66616009)(66946007)(66556008)(64756008)(66476007)(26005)(55016002)(186003)(2940100002)(166002)(71200400001)(5660300002)(66446008)(10290500003)(8676002)(478600001)(110136005)(316002)(54906003)(966005)(52536014)(33656002)(86362001)(8936002)(76236003)(82960400001)(99936003)(82950400001)(4326008)(7696005)(9686003)(6506007)(53546011)(2906002)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?Windows-1252?Q?fOGzrQhr5zMFLiDcKgBoDhudTmCynmOOE83DSfQviHJx5Cm6c+T+OBgK?= =?Windows-1252?Q?2mV4iwTGCufKaKmhpyqiJG/+KtRMCu/GQ3mCPvL1Fg6fM3jRp1ehkUMD?= =?Windows-1252?Q?4jSyEoim/+UiBV88QA3leoDE5N5y0lCc4DKdd7IYh9UbaNJjjNIoO2yk?= =?Windows-1252?Q?cAuUxaLxRdAeHKyfM1zYZ0y8Ql/zZne5BNxDfyhofmXKCUvuiNH1lKuE?= =?Windows-1252?Q?EF+Tw3C5kap8Kti9BJ8m2uZD3ukWHlAshA4V7bus6SGOMGHBiWPpocKz?= =?Windows-1252?Q?XmySZjTYpncuM+6iexmWnnwrT6qOWXgje8RT2N35H4tyoc3zG9ATJskT?= =?Windows-1252?Q?16rRJ4CBccIzVZSaXOKEAa0eVeEpvBrcsm/RZijG85vEPYDvuP1I3QAj?= =?Windows-1252?Q?faQDAzTSpwooOmsnQ6lQVt/tiOWGfHQ7PAIslq2/dbtr6KD8lVSW7LfR?= =?Windows-1252?Q?S77byje+8A0mVkAoHlbQQIXJpx5UYEvdfzBQiTQ6jLubL8bFdUTRGv2d?= =?Windows-1252?Q?fIzvICzDa83p2STByioRr4hnAgIccOLhtyy1HyPsKegFk1sWFgznEa0v?= =?Windows-1252?Q?Njrow9Y2IbYOoVO1dxci4G/RMg9A2dRCU858hKy4BLh/I0tmeXj6Bokq?= =?Windows-1252?Q?oYu8Nd94kwoxWiX/Rcsg9m34Cl2mz5gW0s2a2MnBuj65fiB36RGw4bPu?= =?Windows-1252?Q?o2nwEvSeBxiNc39OlNn3Py4WyTQ/lZRCyMdYAjOjF9WzPxKshjRTyYvL?= =?Windows-1252?Q?x7xWFtudgF2/kp1QKTVDZ47vQu56OiUfbZFkbHIQA26Q2b9ckW9ccXT+?= =?Windows-1252?Q?ZkdqFiNwBSzWRl2KEnPJzCBud/VMs/mG/1AaTPe3Y6yEaZFOK//Uh76q?= =?Windows-1252?Q?sCEMhRXCNRRjsrlJJ89ijez73HbpDChIvcAmMj5W+19anO7NREf5Br/q?= =?Windows-1252?Q?rNoC8nFBFUqBm3qjX68kOaQ9972SSHWtbf+jix25+WwpTTJvSiR+1ej1?= =?Windows-1252?Q?EJuZG2uh8idD1OdC3lzHVGcUUnfCpokgAGei0eZ4HsAIFQSd/hafquQq?= =?Windows-1252?Q?pvVydZuAmI3ub7s+5V2ZWZRvKjQtM1ANk0g4MIyQHEBEpEiHZrszWe8S?= =?Windows-1252?Q?2zi38YiJpVORTs/93P4h8jlkDEAZhFh5hKXPOP6ruNlwTyXVqSSyhfZ+?= =?Windows-1252?Q?LrXnCuFFRCmCMVYUlXkjNaNSU1JjsEr+KUCgAsi5UJo/xlI4DQlpxI1S?= =?Windows-1252?Q?sZ+o7ixXb69YghOhalTNjGEQ7vgtF6JSfryoJmRj/CF2YTN1UkmKn/2T?= =?Windows-1252?Q?ASfSkuia8Wq0ch+ExiMPijdmviaNSM53WYZ3saJGl2jj1I+mZV2KvHFF?= =?Windows-1252?Q?1scw8wsQloMcRVVEDuv5V6t/MvmLNnnAYQH2JIp39DY0NKcXxAdRkUGJ?= =?Windows-1252?Q?P4ZLqXYNYh7Ltjynu4bOyQ=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW4PR21MB1907.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9323a591-827f-470d-dcf8-08d8df3be89a X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Mar 2021 18:32:47.0427 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: NkwL7VlZ5GufW7ZJcW/cCwagW+hPsX/jbLofyyvmGdk3QD/F8zLTlB9Ae4XQt+8awQN9t9oawH7TVtIw9hvPOQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0846 X-Groupsio-MsgNum: 72440 Content-Language: en-US Content-Type: multipart/related; boundary="_004_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_"; type="multipart/alternative" --_004_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_ Content-Type: multipart/alternative; boundary="_000_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_" --_000_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Digging in even further, it looks like you=92re asking about tests across t= est pipelines, including some of the platform pipelines. I don=92t believe = there=92s a cohesive way to do that, as some of them require platform-speci= fic parameters (such as the --FlashRom parameter for OvmfPkg). Can you share the specific PR you=92re failing so that we can talk in conc= rete terms? My guess (from experience failing this myself) is that you are failing the= spell check remotely and not running it locally (because Node, npm, or csp= ell are not installed). It=92s possible that we should investigate document= ing cross-pipeline and cross-package CI dependencies in a central place. Ri= ght now it=92s distributed (e.g. did you follow the steps here: https://git= hub.com/tianocore/edk2/tree/master/.pytool#spell-checking---cspell). - Bret From: Bret Barkelew via groups.io Sent: Thursday, March 4, 2021 10:22 AM To: devel@edk2.groups.io; rebecca@nuviainc.co= m; lersek@redhat.com Cc: Sean Brogan; Kinney, Michael D Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for lo= cal CI runs Thinking further, it will skip some tests if you haven=92t installed the p= rerequisites (e.g. spell check). We could potentially add a single script t= o install the prerequisites, but then there are the prereqs for the prereqs= (e.g. Node). To more clearly address your question, the =93tests=94 that a GitHub CI is= broken up into are arbitrary subdivisions of the tests that run locally (w= hen you run =93stuart_ci_build=94 without parameters) so that the workload = can be divided amongst the available worker agents. If you were to compare = the individual test logs against your local, monolithic test log, you would= find that all cases were covered in both (assuming the aforementioned prer= eqs are installed). - Bret From: Bret Barkelew via groups.io Sent: Thursday, March 4, 2021 9:57 AM To: devel@edk2.groups.io; rebecca@nuviainc.co= m; lersek@redhat.com Cc: Sean Brogan; Kinney, Michael D Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for lo= cal CI runs *An* answer is: - There=92s a DevOps virtual environment that you can run that will preten= d to be a build agent. I played with it years ago, but that was before the = CI tooling (and it was years ago) so I don=92t know how the capabilities co= mpare. My real answer is: - If you run the three commands -- setup, update, and build -- without any= parameters other than your TOOL_CHAIN_TAG, it should run all tests on all = packages for that particular OS/toolchain. - Bret From: Rebecca Cran via groups.io Sent: Thursday, March 4, 2021 9:45 AM To: devel@edk2.groups.io; lersek@redhat.com Cc: Sean Brogan; Bret Barkelew; Kinney, Michael D Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local = CI runs On 3/4/21 10:04 AM, Laszlo Ersek wrote: > (Sigh, my address book pulled one over me -- resending to the correct > list address now. Please ignore the previous posting that you may have > gotten "in private".) > > Hi All, > > while I can run specific CI tests locally, using the commands listed at: > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith= ub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=3D04%7C01%7Cbr= et.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86= f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d= 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&= amp;sdata=3Dl8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserv= ed=3D0 > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith= ub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&dat= a=3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354= b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnkno= wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC= I6Mn0%3D%7C1000&sdata=3D8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh%2Bo%3= D&reserved=3D0 > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith= ub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data= =3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354= b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnkno= wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC= I6Mn0%3D%7C1000&sdata=3DNUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M%3D&= amp;reserved=3D0 > > I don't know how I can determine and drive the whole "test plan" locally= . > > When I submit a PR to github, "something" generates (say) 83 test cases. > How can I run that "something" locally? > > Assume I have the following: > - a Linux builder VM, with up-to-date tools, > - a Windows builder VM, with up-to-date tools, > - a topic branch, on top of "master". > > How do I generate and run those same 83 tests, locally? > > It has happened to me that all my local CI steps succeeded (meaning both > the Windows and the Linux builder), but github still rejected the PR -- > the reason was that I failed to think of a particular test that github > would run. Composing 83 "stuart" command lines isn't something I'd like > to do manually. FWIW it looks like people have been looking for similar functionality of testing a pipeline locally, and not found it. e.g.: https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764&data=3D04%7C01= %7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f98= 8bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbG= Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C= 1000&sdata=3DXMxZesyKxVN9Wg3SyRpUbnnmKzojevHCqQhzQ1Pc4%2Bg%3D&reser= ved=3D0 https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdevelo= percommunity.visualstudio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-co= mmit%2F1302759&data=3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493= 578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6375= 04767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi= LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DgwGZAQJAIXIS0LODFpglrCf5U= a2JeMnNm3A3auCrhx8%3D&reserved=3D0 -- Rebecca Cran --_000_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Digging in even further, it looks like you=92re ask= ing about tests across test pipelines, including some of the platform pipel= ines. I don=92t believe there=92s a cohesive way to do that, as some of the= m require platform-specific parameters (such as the --FlashRom parameter for OvmfPkg).

 

Can you share the specific PR you=92re failing so t= hat we can talk in concrete terms?

 

My guess (from experience failing this myself) is t= hat you are failing the spell check remotely and not running it locally (be= cause Node, npm, or cspell are not installed). It=92s possible that we shou= ld investigate documenting cross-pipeline and cross-package CI dependencies in a central place. Right now it=92s di= stributed (e.g. did you follow the steps here: https://github.com/tianocore= /edk2/tree/master/.pytool#spell-checking---cspell).

 

- Bret

 

From: Bret Barkelew via gr= oups.io
Sent: Thursday, March 4, 2021 10:22 AM
To: devel@edk2.groups.io; rebecca@nuviainc.com; lersek@redh= at.com
Cc: Sean Brogan; <= a href=3D"mailto:michael.d.kinney@intel.com"> Kinney, Michael D
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix= for local CI runs

 

Thinking further, it will skip some tests if you ha= ven=92t installed the prerequisites (e.g. spell check). We could potentiall= y add a single script to install the prerequisites, but then there are the = prereqs for the prereqs (e.g. Node).

 

To more clearly address your question, the =93tests= = =94 that a GitHub CI is broken up into are arbitrary subdivisions of the t= ests that run locally (when you run =93stuart_ci_build=94 without parameter= s) so that the workload can be divided amongst the available worker agents. If you were to compare the individual test l= ogs against your local, monolithic test log, you would find that all cases = were covered in both (assuming the aforementioned prereqs are installed).

 

- Bret

 

From: Bret Barkelew via groups.io
Sent: Thursday, March 4, 2021 9:57 AM
To: devel@edk2.groups.io; rebecca@nuviainc.com; lersek@redh= at.com
Cc: Sean Brogan; <= a href=3D"mailto:michael.d.kinney@intel.com"> Kinney, Michael D
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix= for local CI runs

 

*An* answer is:

- There=92s a DevOps virtual environment that you c= an run that will pretend to be a build agent. I played with it years ago, b= ut that was before the CI tooling (and it was years ago) so I don=92t know = how the capabilities compare.

 

My real answer is:

- If you run the three commands -- setup, update, a= nd build -- without any parameters other than your TOOL_CHAIN_TAG, it shoul= d run all tests on all packages for that particular OS/toolchain.

 

- Bret

 

From: Rebecca Cran via groups.io
Sent: Thursday, March 4, 2021 9:45 AM
To: devel@edk2.groups.io; lersek@redhat.com
Cc: Sean Brogan; <= a href=3D"mailto:Bret.Barkelew@microsoft.com"> Bret Barkelew; Kinney, M= ichael D
Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for= local CI runs

 

On 3/4/21 10:04 AM, = Laszlo Ersek wrote:
> (Sigh, my address book pulled one over me -- resending to the correct=
> list address now. Please ignore the previous posting that you may hav= e
> gotten "in private".)
>
> Hi All,
>
> while I can run specific CI tests locally, using the commands listed = at:
>
> https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&amp;data=3D04%7C01%7C= bret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf= 86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb= 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100= 0&amp;sdata=3Dl8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&= amp;reserved=3D0
> https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&amp;d= ata=3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df3= 54b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnk= nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX= VCI6Mn0%3D%7C1000&amp;sdata=3D8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh= %2Bo%3D&amp;reserved=3D0
> https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&amp;data= = =3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354= b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnkno= wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC= I6Mn0%3D%7C1000&amp;sdata=3DNUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M= %3D&amp;reserved=3D0
>
> I don't know how I can determine and drive the whole "test plan&= quot; locally.
>
> When I submit a PR to github, "something" generates (say) 8= 3 test cases.
> How can I run that "something" locally?
>
> Assume I have the following:
> - a Linux builder VM, with up-to-date tools,
> - a Windows builder VM, with up-to-date tools,
> - a topic branch, on top of "master".
>
> How do I generate and run those same 83 tests, locally?
>
> It has happened to me that all my local CI steps succeeded (meaning b= oth
> the Windows and the Linux builder), but github still rejected the PR = --
> the reason was that I failed to think of a particular test that githu= b
> would run. Composing 83 "stuart" command lines isn't someth= ing I'd like
> to do manually.

FWIW it looks like people have been looking for similar functionality of <= br> testing a pipeline locally, and not found it.

e.g.:
https://nam06.safelinks.protection.outlook.com/?url=3Dh= ttps%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764= &amp;data=3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b80= 4c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63750476727899= 5847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik= 1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3DXMxZesyKxVN9Wg3SyRpUbnnmKzojevH= CqQhzQ1Pc4%2Bg%3D&amp;reserved=3D0
https://nam06.safe= links.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdevelopercommunity.visual= studio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-commit%2F1302759&= amp;data=3D04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608= d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%= 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw= iLCJXVCI6Mn0%3D%7C1000&amp;sdata=3DgwGZAQJAIXIS0LODFpglrCf5Ua2JeMnNm3A3= auCrhx8%3D&amp;reserved=3D0

--
Rebecca Cran


 

 

 

--_000_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_-- --_004_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_ Content-Type: image/png; name="50944657DF1343288AECE7FF723557B6.png" Content-Description: 50944657DF1343288AECE7FF723557B6.png Content-Disposition: inline; filename="50944657DF1343288AECE7FF723557B6.png"; size=140; creation-date="Thu, 04 Mar 2021 18:32:46 GMT"; modification-date="Thu, 04 Mar 2021 18:32:46 GMT" Content-ID: Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAsQAAAABCAYAAADZ77itAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAhSURBVEhL7cMBDQAACAMg+5cygQkeRoMIG9WT VVXVn7MHYi5moJeByLMAAAAASUVORK5CYII= --_004_MW4PR21MB1907060C63A0C85D77BFD537EF979MW4PR21MB1907namp_--