From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (NAM10-DM6-obe.outbound.protection.outlook.com [40.107.93.114]) by mx.groups.io with SMTP id smtpd.web12.350.1587759737542886813 for ; Fri, 24 Apr 2020 13:22:17 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=IOYE0lkx; spf=pass (domain: microsoft.com, ip: 40.107.93.114, mailfrom: bret.barkelew@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JKZOayO/TUHqFZFAKObdqSkkB5pIN7U1uU45w6tE1cjV+0ZnLt6dQxOYa4i4yiXUmljgfZ4ad2HbpDNUoe4fMFr/lIHo3uhadEcvspbHh7xer0e3K+KVuZ+6EAG01A3znnoHMDmFAxS5GwJji4SS/wsnODNyM+QLqpxGzAAdH+9hW2DdPbIDTidjsbeESsOT6N5nJuG4Gs5tq5LVnp82u+3NT3MecKQp5QYnWPPaysPORgDlz/nvZFABzyJbWzKeLdNTUGVy/f1LMMYktGlqimpW/hYyvqzOwpYy74oV/DkL3g4ObxnXGKx1uXvpjpBQknY+dknbSX2sPhbZTuA5QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1R3kQ8FvVhAcgSRaWnK3rPU70sWYT9RgY+ebtEpuxR4=; b=IsAVGNSCsiA0ooeG4l7kHcfKMAapM/0OKjO/Y4eVlqj9J7LnXfD7wIyrVHAXBEc9ACMhMhSaoy2BvF+L7LmNaWVQsa1puGHqnP1/1dM7JqNXzcrPL+bEoYV8+A4M0RR0+6Tajrdma0E5q3uNWSLmM1x+2T9YTJbqlKnWgic6Yvl+ETs4mv+lXOnynfYlEJyKNIlSoTjMlQE5QJiL53qDYmFW4GSH3Bg2j0wh4AkMxmTQ7bMV4icpHDeIi09LU52owlMyXQO2C0X8U8EoR4EXgntCQ8BL0s+rvuCGSRkTRL/uD+SqpDCLSOeWY/4voYu+e0bpikx2/ZuF2GzUnNq+cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1R3kQ8FvVhAcgSRaWnK3rPU70sWYT9RgY+ebtEpuxR4=; b=IOYE0lkxTQMz7cXgwhBz0PVUMU5MLBWL9tKIpPzXnla9E7ge99QJPJqBWVXiuIi2YFIyskPrDeqvl1+RjxiOiDyRSei/krQKMJ3TyaEqyHm7TTFrgWbnLdlXw7EPmNkScJCfWyOZ5Kn1OUG3QdFfPRb96se2f3x2qwidX7Z6ME8= Received: from CY4PR21MB0743.namprd21.prod.outlook.com (2603:10b6:903:b2::9) by CY4PR21MB0744.namprd21.prod.outlook.com (2603:10b6:903:b2::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.3; Fri, 24 Apr 2020 20:22:15 +0000 Received: from CY4PR21MB0743.namprd21.prod.outlook.com ([fe80::90d:10d9:c5bc:5318]) by CY4PR21MB0743.namprd21.prod.outlook.com ([fe80::90d:10d9:c5bc:5318%12]) with mapi id 15.20.2958.010; Fri, 24 Apr 2020 20:22:15 +0000 From: "Bret Barkelew" To: "devel@edk2.groups.io" , "michael.kubacki@outlook.com" CC: Sean Brogan , "Kinney, Michael D" , Liming Gao Subject: Re: [EXTERNAL] [edk2-devel] [PATCH v2 1/6] .azurepipelines: Add Platform CI template Thread-Topic: [EXTERNAL] [edk2-devel] [PATCH v2 1/6] .azurepipelines: Add Platform CI template Thread-Index: AQHWF0e00tRIa7OkCkKY2vgl5r5u1aiIvYJf Date: Fri, 24 Apr 2020 20:22:15 +0000 Message-ID: References: <20200420191216.24572-1-michael.kubacki@outlook.com>, In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-04-24T20:22:10.9775531Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Privileged authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bret.Barkelew@microsoft.com; x-originating-ip: [174.21.83.205] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 4a028d92-bc50-48d2-d7ba-08d7e88d2e0a x-ms-traffictypediagnostic: CY4PR21MB0744:|CY4PR21MB0744:|CY4PR21MB0744: x-ms-exchange-transport-forked: True x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8273; x-forefront-prvs: 03838E948C x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR21MB0743.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(366004)(346002)(376002)(396003)(39860400002)(136003)(186003)(66946007)(76116006)(2906002)(110136005)(66446008)(64756008)(66556008)(66476007)(8936002)(5660300002)(53546011)(26005)(52536014)(82950400001)(82960400001)(54906003)(55016002)(33656002)(8990500004)(7696005)(9686003)(6506007)(30864003)(86362001)(4326008)(478600001)(71200400001)(8676002)(81156014)(316002)(10290500003)(966005);DIR:OUT;SFP:1102; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Wwsajh69KwHrmlTvQjEIjSu1VZro/GKIpTFogP51ypJgMXvnTQatgGMRC7iU7MHXxbu7FMRRoAxnba+watuJ4FImnZpbCD0fd14owPrDpFZZxgbHV5l9hWMBUXe1n2CCHpAmUuAzzRsXRaLIc2iig9f6qxxUtoPchbjEqdQKY6g2gGbmopYUHO2CAZ9IRr4ZiVkjd2P8EJmSJUUx66WyO1LrhodkP9QxgtaaXNuS+jC19veRDe8rWB+RjQJbiKgauek+hpXCEyYrcuOVr+cxhOCXU3Iuwha+nsZFw8/DtXCgtll5gl7HDZ682rqAStQlygqAg/RpX48hm9FJspeQ6MEhAJhM8716ixMUj5mlLv8Ngi90D666/cO2PJnpJa3w9k8r91BIubNFWho7qL/9yjL7pHW6hI4eOQvM6YVkOXNiwcaPr3O2MhrQP3XQwpxfR0p2lCfRQDOpny99dMKhVziUki4pbzvZKBpQvrwlpZ8rw/nxg6Pn1pzgsuzbpA4ovTThy+RDXNkmROodob6cXg== x-ms-exchange-antispam-messagedata: kZQNZWZeVCIZ212TrqgoIrAGufLgSe08QNk9nilQOXxaiYiTreu7vmkzjMbPyjBL5icgPXgQp/rlQFQrhLT/1XHD/7XHcyNI4KjUExHl5YBaPmM8D12bKQ/ecoXbZ1V2mmhe4aA9a0ZgaP9Z7jY85qAIzHuQ16EJe8HmAixXx9sbP4kxhu842AYDMyQvHS3B7E1iQUSvCjCy7jTP/M9TCTrifzk7MvLtH72f6Knz3xXpMnIhFlMo4xiV0TUUMuIM2MjITBwDHYKM0wVCEcx3KJa4mQpyAFqLEndHwf4fUN2077o99XK7iXTP+3O43X4U/zFyaBEiRhDuO09k2S4zRFirihwOnQHR/9D3x3nKJLm6RKFMegx0kuP9t3xQ58QNdZWnU6y/HUMnaD9giuUcWTZ86+ub4eCEzAfDWZDFNQLtV4AMFGF5v7TgjmUjars/ieXcJiB0JK17RU/3T4sBxS+XuaN2BA1vz3IIExxamRRsHNXqm64sL6ZcdDNrT71KLkZnKaYKLPeRcAvU3SC23YvdIN0sG1fYNZzG7yPW1Z+q5pxe3+G1jTJk2CX2I+jouVqcnc6XpJ+nFWQz60xLDmIFhgCiFq8hjA0KMY7s4Pc2Cz0VwuGNQ+DW9pvB594KZeLT6UdEpuVSWsW4xbLtU0RzYMDWDJ+IQHAxidZ6LHH6id6x9hPCaLRbRTeLFj05fF3jlqc7zI/BY3V3RKdubOt3th4gZ6V7lLEqoTsxBds165eTxE+As0LNwqs8u064oiD9EDWxx2m3d48IT2/rNkS88DBPV3s+JhmOYKU8iOQ= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4a028d92-bc50-48d2-d7ba-08d7e88d2e0a X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Apr 2020 20:22:15.6060 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mwjAWq+Gg+YZH+zilWRfFK5BbTRnuKaOB2PdVgoDsaFUemYe1zHHbTN3ImmY6R2DWvsr2rXKP4vjRScJ9L4D7Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR21MB0744 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_CY4PR21MB0743572522F70724917CBDC5EFD00CY4PR21MB0743namp_" --_000_CY4PR21MB0743572522F70724917CBDC5EFD00CY4PR21MB0743namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Bret Barkelew - Bret ________________________________ From: devel@edk2.groups.io on behalf of Michael Kub= acki via groups.io Sent: Monday, April 20, 2020 12:12:11 PM To: devel@edk2.groups.io Cc: Sean Brogan ; Bret Barkelew ; Kinney, Michael D ; Liming Gao = Subject: [EXTERNAL] [edk2-devel] [PATCH v2 1/6] .azurepipelines: Add Platf= orm CI template From: Sean Brogan REF:https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fbu= gzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2570&data=3D02%7C01%7Cbret.b= arkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141= af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&sdata=3Demu31XA%2FKpr%2= FSD4gy4DbcjYVj1PYaDMaaWk8GGtJwUU%3D&reserved=3D0 Add template for Platform CI steps for a Pytools based build. Add README to describe the template and how to use it. Add helpful information for working with azurepipelines, templates, and lessons learned. Cc: Sean Brogan Cc: Bret Barkelew Cc: Michael D Kinney Cc: Liming Gao Signed-off-by: Sean Brogan Signed-off-by: Michael Kubacki --- .azurepipelines/ReadMe.md | 50 ++++++++ .azurepipelines/templates/ReadMe.md | 59 +++++++++ .azurepipelines/templates/platform-build-run-steps.yml | 134 ++++++++++++= ++++++++ 3 files changed, 243 insertions(+) diff --git a/.azurepipelines/ReadMe.md b/.azurepipelines/ReadMe.md new file mode 100644 index 000000000000..cf57d282c197 --- /dev/null +++ b/.azurepipelines/ReadMe.md @@ -0,0 +1,50 @@ +# Azure DevOps Pipelines + +These yml files are used to provide CI builds using the Azure DevOps Pipe= line Service. +Most of the CI leverages edk2-pytools to support cross platform building = and execution. + +## Core CI + +Focused on building and testing all packages in Edk2 without an actual ta= rget platform. + +See `.pytools/ReadMe.py` for more details + +## Platform CI + +Focused on building a single target platform and confirming functionality= on that platform. + +## Conventions + +* Files extension should be *.yml. *.yaml is also supported but in Edk2 = we use those for our package configuration. +* Platform CI files should be in the `/.azurepipelines` fold= er. +* Core CI files are in the root folder. +* Shared templates are in the `templates` folder. +* Top level CI files should be named `-.yml` + +## Links + +* Basic Azure Landing Site - https://nam06.safelinks.protection.outlook.c= om/?url=3Dhttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipel= ines%2F%3Fview%3Dazure-devops&data=3D02%7C01%7Cbret.barkelew%40microsof= t.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af91ab2d7cd011db47= %7C1%7C0%7C637230067792915138&sdata=3DXPGqra1YU6OrOTx6cWKyKfETYLlvTrNm%= 2BoLmzv66fQ8%3D&reserved=3D0 +* Pipeline jobs - https://nam06.safelinks.protection.outlook.com/?url=3Dh= ttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%2Fproc= ess%2Fphases%3Fview%3Dazure-devops%26tabs%3Dyaml&data=3D02%7C01%7Cbret.= barkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f14= 1af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&sdata=3DFtXq1ps15kp5Vw= jl95BksV0XOFV6X4oA2nxw9w983u0%3D&reserved=3D0 +* Pipeline yml scheme - https://nam06.safelinks.protection.outlook.com/?u= rl=3Dhttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%= 2Fyaml-schema%3Fview%3Dazure-devops%26tabs%3Dschema%252Cparameter-schema&am= p;data=3D02%7C01%7Cbret.barkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7= e55ed649%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&am= p;sdata=3DT%2FqPGOWIgIN4TS%2BK116FWgdN3irKc4%2BlVqxG5LCpKwA%3D&reserved= = =3D0 +* Pipeline expression - https://nam06.safelinks.protection.outlook.com/?u= rl=3Dhttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%= 2Fprocess%2Fexpressions%3Fview%3Dazure-devops&data=3D02%7C01%7Cbret.bar= kelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af= 91ab2d7cd011db47%7C1%7C0%7C637230067792915138&sdata=3Dx5V2xgJQRV31fZ7pJ= biuHXjAuOxAHTdHgYvtNvB%2F6Tc%3D&reserved=3D0 +* PyTools - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3= A%2F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-extensions&data=3D02%7C01%7= Cbret.barkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988b= f86f141af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&sdata=3Duqzx7GHM= 7ahuQKGgm%2FWt2C5%2B9nQyQOCQLFpQkOiBt7M%3D&reserved=3D0 and https://nam= 06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Ftiano= core%2Fedk2-pytool-library&data=3D02%7C01%7Cbret.barkelew%40microsoft.c= om%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af91ab2d7cd011db47%7C= 1%7C0%7C637230067792915138&sdata=3DbL9qDvlEe4kAgM08xI%2FncpOiMBtyn8wA35= M5j57tDgI%3D&reserved=3D0 + +## Lessons Learned + +### Templates and parameters + +They are great but evil. If they are used as part of determining the ste= ps of a build they must resolve before the build starts. They can not use = variables set in a yml or determined as part of a matrix. If they are used= in a step then they can be bound late. + +### File matching patterns + +On Linux this can hang if there are too many files in the search list. + +### Templates and file splitting + +Suggestion is to do one big yaml file that does what you want for one of = your targets. Then do the second one and find the deltas. From that you c= an start to figure out the right split of files, steps, jobs. + +### Conditional steps + +If you want the step to show up in the log but not run, use a step condit= ional. This is great when a platform doesn't currently support a feature bu= t you want the builders to know that the features exists and maybe someday = it will. + +If you want the step to not show up use a template step conditional wrapp= er. Beware this will be evaluated early (at build start). This can hide t= hings not needed on a given OS for example. diff --git a/.azurepipelines/templates/ReadMe.md b/.azurepipelines/templat= es/ReadMe.md new file mode 100644 index 000000000000..fa433e3ef597 --- /dev/null +++ b/.azurepipelines/templates/ReadMe.md @@ -0,0 +1,59 @@ +# CI Templates + +This folder contains azure pipeline yml templates for "Core" and "Platfor= m" Continuous Integration and PR validation. + +## Common CI templates + +### basetools-build-steps.yml + +This template compiles the Edk2 basetools from source. The steps in this= template are +conditional and will only run if variable `pkg_count` is greater than 0. + +It also has two conditional steps only used when the toolchain contains G= CC. These two steps +use `apt` to update the system packages and add those necessary for Edk2 = builds. + +## Core CI templates + +### pr-gate-build-job.yml + +This templates contains the jobs and most importantly the matrix of which= packages and +targets to run for Core CI. + +### pr-gate-steps.yml + +This template is the main Core CI template. It controls all the steps ru= n and is responsible for most functionality of the Core CI process. This t= emplate sets +the `pkg_count` variable using the `stuart_pr_eval` tool when the +build type is "pull request" + +### spell-check-prereq-steps.yml + +This template installs the node based tools used by the spell checker plu= gin. The steps +in this template are conditional and will only run if variable `pkg_count= ` is greater than 0. + +## Platform CI templates + +### platform-build-run-steps.yml + +This template makes heavy use of pytools to build and run a platform in t= he Edk2 repo + +Also uses basetools-build-steps.yml to compile basetools + +#### Special Notes + +* For a build type of pull request it will conditionally build if the pat= ches change files that impact the platform. + * uses `stuart_pr_eval` to determine impact +* For manual builds or CI builds it will always build the platform +* It compiles basetools from source +* Will use `stuart_build --FlashOnly` to attempt to run the built image i= f the `Run` parameter is set. +* See the parameters block for expected configuration options +* Parameter `extra_install_step` allows the caller to insert extra steps.= This is useful if additional dependencies, tools, or other things need to= be installed. Here is an example of installing qemu on Windows. + + ``` yaml + steps: + - template: ../../.azurepipelines/templates/build-run-steps.yml + parameters: + extra_install_step: + - powershell: choco install qemu; Write-Host "##vso[task.prependp= ath]c:\Program Files\qemu" + displayName: Install QEMU and Set QEMU on path # friendly name = displayed in the UI + condition: and(gt(variables.pkg_count, 0), succeeded()) + ``` diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azu= repipelines/templates/platform-build-run-steps.yml new file mode 100644 index 000000000000..ebf674bfc7d8 --- /dev/null +++ b/.azurepipelines/templates/platform-build-run-steps.yml @@ -0,0 +1,134 @@ + +## @file +# File steps.yml +# +# template file containing the steps to build +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: +- name: tool_chain_tag + type: string + default: '' +- name: build_pkg + type: string + default: '' +- name: build_target + type: string + default: '' +- name: build_arch + type: string + default: '' +- name: build_file + type: string + default: '' +- name: build_flags + type: string + default: '' +- name: run_flags + type: string + default: '' + +- name: extra_install_step + type: stepList + default: [] + +steps: +- checkout: self + clean: true + fetchDepth: 1 + +- task: UsePythonVersion@0 + inputs: + versionSpec: "3.8.x" + architecture: "x64" + +- script: pip install -r pip-requirements.txt --upgrade + displayName: 'Install/Upgrade pip modules' + +# Set default +- bash: echo "##vso[task.setvariable variable=3Dpkg_count]${{ 1 }}" + +# trim the package list if this is a PR +- task: CmdLine@1 + displayName: Check if ${{ parameters.build_pkg }} need testing + inputs: + filename: stuart_pr_eval + arguments: -c ${{ parameters.build_file }} -t ${{ parameters.build_ta= rget}} -a ${{ parameters.build_arch}} --pr-target origin/$(System.PullReque= st.targetBranch) --output-count-format-string "##vso[task.setvariable varia= ble=3Dpkg_count;isOutpout=3Dtrue]{pkgcount}" + condition: eq(variables['Build.Reason'], 'PullRequest') + + # Setup repo +- task: CmdLine@1 + displayName: Setup + inputs: + filename: stuart_setup + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=3D${{ param= eters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.b= uild_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# Stuart Update +- task: CmdLine@1 + displayName: Update + inputs: + filename: stuart_update + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=3D${{ param= eters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.b= uild_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# build basetools +# do this after setup and update so that code base dependencies +# are all resolved. +- template: basetools-build-steps.yml + parameters: + tool_chain_tag: ${{ parameters.tool_chain_tag }} + +# Potential Extra steps +- ${{ parameters.extra_install_step }} + +# Build +- task: CmdLine@1 + displayName: Build + inputs: + filename: stuart_build + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=3D${{ param= eters.tool_chain_tag}} TARGET=3D${{ parameters.build_target}} -a ${{ parame= ters.build_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# Run +- task: CmdLine@1 + displayName: Run to shell + inputs: + filename: stuart_build + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=3D${{ param= eters.tool_chain_tag}} TARGET=3D${{ parameters.build_target}} -a ${{ parame= ters.build_arch}} ${{ parameters.build_flags}} ${{ parameters.run_flags }} = --FlashOnly + condition: and(and(gt(variables.pkg_count, 0), succeeded()), eq(variabl= es['Run'], true)) + timeoutInMinutes: 1 + +# Copy the build logs to the artifact staging directory +- task: CopyFiles@2 + displayName: "Copy build logs" + inputs: + targetFolder: "$(Build.ArtifactStagingDirectory)" + SourceFolder: "Build" + contents: | + BUILDLOG_*.txt + BUILDLOG_*.md + CI_*.txt + CI_*.md + CISETUP.txt + SETUPLOG.txt + UPDATE_LOG.txt + PREVALLOG.txt + TestSuites.xml + **/BUILD_TOOLS_REPORT.html + **/OVERRIDELOG.TXT + BASETOOLS_BUILD*.* + flattenFolders: true + condition: succeededOrFailed() + +# Publish build artifacts to Azure Artifacts/TFS or a file share +- task: PublishBuildArtifacts@1 + continueOnError: true + displayName: "Publish build logs" + inputs: + pathtoPublish: "$(Build.ArtifactStagingDirectory)" + artifactName: "Build Logs $(System.JobName)" + condition: succeededOrFailed() -- 2.16.3.windows.1 --_000_CY4PR21MB0743572522F70724917CBDC5EFD00CY4PR21MB0743namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Reviewed-by: Bret Barkelew <bret.barkelew@micros= oft.com>

 

- Bret

 

From: devel@edk2.groups.io= <devel@edk2.groups.io> on behalf of Michael Kubacki via groups.io &l= t;michael.kubacki=3Doutlook.com@groups.io>
Sent: Monday, April 20, 2020 12:12:11 PM
To: devel@edk2.groups.io <devel@edk2.groups.io>
Cc: Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew &l= t;Bret.Barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@i= ntel.com>; Liming Gao <liming.gao@intel.com>
Subject: [EXTERNAL] [edk2-devel] [PATCH v2 1/6] .azurepipelines: Ad= d Platform CI template
 
From: Sean Brogan <sean.brogan@microsoft.com&g= t;

REF:https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fbu= gzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2570&amp;data=3D02%7C01%7Cbr= et.barkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86= f141af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&amp;sdata=3Demu31XA= %2FKpr%2FSD4gy4DbcjYVj1PYaDMaaWk8GGtJwUU%3D&amp;reserved=3D0

Add template for Platform CI steps for a Pytools based build.
Add README to describe the template and how to use it.
Add helpful information for working with azurepipelines, templates, and lessons learned.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 .azurepipelines/ReadMe.md       &= nbsp;           &nbs= p;          |  50 +&#= 43;++++++
 .azurepipelines/templates/ReadMe.md     &nb= sp;            =   |  59 +++++++++
 .azurepipelines/templates/platform-build-run-steps.yml | 134 +&#= 43;++++++++++++++&#= 43;+++
 3 files changed, 243 insertions(+)

diff --git a/.azurepipelines/ReadMe.md b/.azurepipelines/ReadMe.md
new file mode 100644
index 000000000000..cf57d282c197
--- /dev/null
+++ b/.azurepipelines/ReadMe.md
@@ -0,0 +1,50 @@
+# Azure DevOps Pipelines
+
+These yml files are used to provide CI builds using the Azure DevOps = Pipeline Service.
+Most of the CI leverages edk2-pytools to support cross platform build= ing and execution.
+
+## Core CI
+
+Focused on building and testing all packages in Edk2 without an actua= l target platform.
+
+See `.pytools/ReadMe.py` for more details
+
+## Platform CI
+
+Focused on building a single target platform and confirming functiona= lity on that platform.
+
+## Conventions
+
+* Files extension should be *.yml.  *.yaml is also supported but= in Edk2 we use those for our package configuration.
+* Platform CI files should be in the `<PlatformPkg>/.azurepipel= ines` folder.
+* Core CI files are in the root folder.
+* Shared templates are in the `templates` folder.
+* Top level CI files should be named `<host os>-<tool_chain_= tag>.yml`
+
+## Links
+
+* Basic Azure Landing Site - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdocs.m= icrosoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%2F%3Fview%3Dazure-devops&= amp;amp;data=3D02%7C01%7Cbret.barkelew%40microsoft.com%7Cbd31ac9439e54096f3= 1808d7e55ed649%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637230067792915= 138&amp;sdata=3DXPGqra1YU6OrOTx6cWKyKfETYLlvTrNm%2BoLmzv66fQ8%3D&am= p;reserved=3D0
+* Pipeline jobs - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdocs.m= icrosoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%2Fprocess%2Fphases%3Fview= %3Dazure-devops%26tabs%3Dyaml&amp;data=3D02%7C01%7Cbret.barkelew%40micr= osoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af91ab2d7cd011= db47%7C1%7C0%7C637230067792915138&amp;sdata=3DFtXq1ps15kp5Vwjl95BksV0XO= FV6X4oA2nxw9w983u0%3D&amp;reserved=3D0
+* Pipeline yml scheme - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdocs.m= icrosoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%2Fyaml-schema%3Fview%3Daz= ure-devops%26tabs%3Dschema%252Cparameter-schema&amp;data=3D02%7C01%7Cbr= et.barkelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86= f141af91ab2d7cd011db47%7C1%7C0%7C637230067792915138&amp;sdata=3DT%2FqPG= OWIgIN4TS%2BK116FWgdN3irKc4%2BlVqxG5LCpKwA%3D&amp;reserved=3D0
+* Pipeline expression - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdocs.m= icrosoft.com%2Fen-us%2Fazure%2Fdevops%2Fpipelines%2Fprocess%2Fexpressions%3= Fview%3Dazure-devops&amp;data=3D02%7C01%7Cbret.barkelew%40microsoft.com= %7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af91ab2d7cd011db47%7C1%= 7C0%7C637230067792915138&amp;sdata=3Dx5V2xgJQRV31fZ7pJbiuHXjAuOxAHTdHgY= vtNvB%2F6Tc%3D&amp;reserved=3D0
+* PyTools - https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Ftianocore%2Fedk2-pytool-extensions&amp;data=3D02%7C01%7Cbret.bar= kelew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af= 91ab2d7cd011db47%7C1%7C0%7C637230067792915138&amp;sdata=3Duqzx7GHM7ahuQ= KGgm%2FWt2C5%2B9nQyQOCQLFpQkOiBt7M%3D&amp;reserved=3D0 and https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Ftianocore%2Fedk2-pytool-library&amp;data=3D02%7C01%7Cbret.barkel= ew%40microsoft.com%7Cbd31ac9439e54096f31808d7e55ed649%7C72f988bf86f141af91a= b2d7cd011db47%7C1%7C0%7C637230067792915138&amp;sdata=3DbL9qDvlEe4kAgM08= xI%2FncpOiMBtyn8wA35M5j57tDgI%3D&amp;reserved=3D0
+
+## Lessons Learned
+
+### Templates and parameters
+
+They are great but evil.  If they are used as part of determinin= g the steps of a build they must resolve before the build starts.  The= y can not use variables set in a yml or determined as part of a matrix.&nbs= p; If they are used in a step then they can be bound late.
+
+### File matching patterns
+
+On Linux this can hang if there are too many files in the search list= .
+
+### Templates and file splitting
+
+Suggestion is to do one big yaml file that does what you want for one= of your targets.  Then do the second one and find the deltas.  F= rom that you can start to figure out the right split of files, steps, jobs.=
+
+### Conditional steps
+
+If you want the step to show up in the log but not run, use a step co= nditional. This is great when a platform doesn't currently support a featur= e but you want the builders to know that the features exists and maybe some= day it will.
+
+If you want the step to not show up use a template step conditional w= rapper.  Beware this will be evaluated early (at build start).  T= his can hide things not needed on a given OS for example.
diff --git a/.azurepipelines/templates/ReadMe.md b/.azurepipelines/templat= es/ReadMe.md
new file mode 100644
index 000000000000..fa433e3ef597
--- /dev/null
+++ b/.azurepipelines/templates/ReadMe.md
@@ -0,0 +1,59 @@
+# CI Templates
+
+This folder contains azure pipeline yml templates for "Core"= ; and "Platform" Continuous Integration and PR validation.
+
+## Common CI templates
+
+### basetools-build-steps.yml
+
+This template compiles the Edk2 basetools from source.  The step= s in this template are
+conditional and will only run if variable `pkg_count` is greater than= 0.
+
+It also has two conditional steps only used when the toolchain contai= ns GCC. These two steps
+use `apt` to update the system packages and add those necessary for E= dk2 builds.
+
+## Core CI templates
+
+### pr-gate-build-job.yml
+
+This templates contains the jobs and most importantly the matrix of w= hich packages and
+targets to run for Core CI.
+
+### pr-gate-steps.yml
+
+This template is the main Core CI template.  It controls all the= steps run and is responsible for most functionality of the Core CI process= .  This template sets
+the `pkg_count` variable using the `stuart_pr_eval` tool when the
+build type is "pull request"
+
+### spell-check-prereq-steps.yml
+
+This template installs the node based tools used by the spell checker= plugin. The steps
+in this template are conditional and will only run if variable `pkg_c= ount` is greater than 0.
+
+## Platform CI templates
+
+### platform-build-run-steps.yml
+
+This template makes heavy use of pytools to build and run a platform = in the Edk2 repo
+
+Also uses basetools-build-steps.yml to compile basetools
+
+#### Special Notes
+
+* For a build type of pull request it will conditionally build if the= patches change files that impact the platform.
+  * uses `stuart_pr_eval` to determine impact
+* For manual builds or CI builds it will always build the platform +* It compiles basetools from source
+* Will use `stuart_build --FlashOnly` to attempt to run the built ima= ge if the `Run` parameter is set.
+* See the parameters block for expected configuration options
+* Parameter `extra_install_step` allows the caller to insert extra st= eps.  This is useful if additional dependencies, tools, or other thing= s need to be installed.  Here is an example of installing qemu on Wind= ows.
+
+    ``` yaml
+    steps:
+    - template: ../../.azurepipelines/templates/build-= run-steps.yml
+      parameters:
+        extra_install_step:
+        - powershell: choco instal= l qemu; Write-Host "##vso[task.prependpath]c:\Program Files\qemu"=
+          displayName: I= nstall QEMU and Set QEMU on path # friendly name displayed in the UI
+          condition: and= (gt(variables.pkg_count, 0), succeeded())
+    ```
diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azu= repipelines/templates/platform-build-run-steps.yml
new file mode 100644
index 000000000000..ebf674bfc7d8
--- /dev/null
+++ b/.azurepipelines/templates/platform-build-run-steps.yml @@ -0,0 +1,134 @@
+
+## @file
+# File steps.yml
+#
+# template file containing the steps to build
+#
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+- name: tool_chain_tag
+  type: string
+  default: ''
+- name: build_pkg
+  type: string
+  default: ''
+- name: build_target
+  type: string
+  default: ''
+- name: build_arch
+  type: string
+  default: ''
+- name: build_file
+  type: string
+  default: ''
+- name: build_flags
+  type: string
+  default: ''
+- name: run_flags
+  type: string
+  default: ''
+
+- name: extra_install_step
+  type: stepList
+  default: []
+
+steps:
+- checkout: self
+  clean: true
+  fetchDepth: 1
+
+- task: UsePythonVersion@0
+  inputs:
+    versionSpec: "3.8.x"
+    architecture: "x64"
+
+- script: pip install -r pip-requirements.txt --upgrade
+  displayName: 'Install/Upgrade pip modules'
+
+# Set default
+- bash: echo "##vso[task.setvariable variable=3Dpkg_count]${{ 1 = }}"
+
+# trim the package list if this is a PR
+- task: CmdLine@1
+  displayName: Check if ${{ parameters.build_pkg }} need testing=
+  inputs:
+    filename: stuart_pr_eval
+    arguments: -c ${{ parameters.build_file }} -t ${{ = parameters.build_target}} -a ${{ parameters.build_arch}} --pr-target origin= /$(System.PullRequest.targetBranch) --output-count-format-string "##vs= o[task.setvariable variable=3Dpkg_count;isOutpout=3Dtrue]{pkgcount}" +  condition: eq(variables['Build.Reason'], 'PullRequest')
+
+ # Setup repo
+- task: CmdLine@1
+  displayName: Setup
+  inputs:
+    filename: stuart_setup
+    arguments: -c ${{ parameters.build_file }} TOOL_CH= AIN_TAG=3D${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} = -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+  condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# Stuart Update
+- task: CmdLine@1
+  displayName: Update
+  inputs:
+    filename: stuart_update
+    arguments: -c ${{ parameters.build_file }} TOOL_CH= AIN_TAG=3D${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} = -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+  condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# build basetools
+#   do this after setup and update so that code base depend= encies
+#   are all resolved.
+- template: basetools-build-steps.yml
+  parameters:
+    tool_chain_tag: ${{ parameters.tool_chain_tag }} +
+# Potential Extra steps
+- ${{ parameters.extra_install_step }}
+
+# Build
+- task: CmdLine@1
+  displayName: Build
+  inputs:
+    filename: stuart_build
+    arguments: -c ${{ parameters.build_file }} TOOL_CH= AIN_TAG=3D${{ parameters.tool_chain_tag}} TARGET=3D${{ parameters.build_tar= get}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}}
+  condition: and(gt(variables.pkg_count, 0), succeeded())
+
+# Run
+- task: CmdLine@1
+  displayName: Run to shell
+  inputs:
+    filename: stuart_build
+    arguments: -c ${{ parameters.build_file }} TOOL_CH= AIN_TAG=3D${{ parameters.tool_chain_tag}} TARGET=3D${{ parameters.build_tar= get}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} ${{ param= eters.run_flags }} --FlashOnly
+  condition: and(and(gt(variables.pkg_count, 0), succeeded()), e= q(variables['Run'], true))
+  timeoutInMinutes: 1
+
+# Copy the build logs to the artifact staging directory
+- task: CopyFiles@2
+  displayName: "Copy build logs"
+  inputs:
+    targetFolder: "$(Build.ArtifactStagingDirecto= ry)"
+    SourceFolder: "Build"
+    contents: |
+      BUILDLOG_*.txt
+      BUILDLOG_*.md
+      CI_*.txt
+      CI_*.md
+      CISETUP.txt
+      SETUPLOG.txt
+      UPDATE_LOG.txt
+      PREVALLOG.txt
+      TestSuites.xml
+      **/BUILD_TOOLS_REPORT.html
+      **/OVERRIDELOG.TXT
+      BASETOOLS_BUILD*.*
+    flattenFolders: true
+  condition: succeededOrFailed()
+
+# Publish build artifacts to Azure Artifacts/TFS or a file share
+- task: PublishBuildArtifacts@1
+  continueOnError: true
+  displayName: "Publish build logs"
+  inputs:
+    pathtoPublish: "$(Build.ArtifactStagingDirect= ory)"
+    artifactName: "Build Logs $(System.JobName)&q= uot;
+  condition: succeededOrFailed()
--
2.16.3.windows.1




--_000_CY4PR21MB0743572522F70724917CBDC5EFD00CY4PR21MB0743namp_--