From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga09.intel.com (mga09.intel.com []) by mx.groups.io with SMTP id smtpd.web11.1463.1573089233798338813 for ; Wed, 06 Nov 2019 17:13:57 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=fail (domain: intel.com, ip: , mailfrom: michael.d.kinney@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Nov 2019 17:13:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,276,1569308400"; d="scan'208";a="205520636" Received: from unknown (HELO mdkinney-MOBL2.amr.corp.intel.com) ([10.241.98.74]) by orsmga003.jf.intel.com with ESMTP; 06 Nov 2019 17:13:57 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao Subject: [Patch v4 20/22] .azurepipelines: Add Azure Pipelines YML configuration files Date: Wed, 6 Nov 2019 17:13:47 -0800 Message-Id: <20191107011349.16524-21-michael.d.kinney@intel.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191107011349.16524-1-michael.d.kinney@intel.com> References: <20191107011349.16524-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Sean Brogan Add YML configuration files used to run the EDK II Continuous Integration (CI) checks on Azure Pipelines agents. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Signed-off-by: Michael D Kinney --- .azurepipelines/Ubuntu-GCC5.yml | 18 +++ .azurepipelines/Ubuntu-PatchCheck.yml | 32 +++++ .azurepipelines/Windows-VS2019.yml | 18 +++ .../templates/basetools-build-steps.yml | 36 +++++ .../templates/pr-gate-build-job.yml | 79 +++++++++++ .azurepipelines/templates/pr-gate-steps.yml | 129 ++++++++++++++++++ .../templates/spell-check-prereq-steps.yml | 21 +++ 7 files changed, 333 insertions(+) create mode 100644 .azurepipelines/Ubuntu-GCC5.yml create mode 100644 .azurepipelines/Ubuntu-PatchCheck.yml create mode 100644 .azurepipelines/Windows-VS2019.yml create mode 100644 .azurepipelines/templates/basetools-build-steps.yml create mode 100644 .azurepipelines/templates/pr-gate-build-job.yml create mode 100644 .azurepipelines/templates/pr-gate-steps.yml create mode 100644 .azurepipelines/templates/spell-check-prereq-steps.yml diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml new file mode 100644 index 0000000000..191bda8a91 --- /dev/null +++ b/.azurepipelines/Ubuntu-GCC5.yml @@ -0,0 +1,18 @@ +## +# Azure Pipeline build file for a build using ubuntu and GCC5 +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +trigger: +- master +pr: +- master + +jobs: +- template: templates/pr-gate-build-job.yml + parameters: + tool_chain_tag: 'GCC5' + vm_image: 'ubuntu-latest' + arch_list: "IA32,X64,ARM,AARCH64" + diff --git a/.azurepipelines/Ubuntu-PatchCheck.yml b/.azurepipelines/Ubuntu-PatchCheck.yml new file mode 100644 index 0000000000..0d91d78fb2 --- /dev/null +++ b/.azurepipelines/Ubuntu-PatchCheck.yml @@ -0,0 +1,32 @@ +## @file +# Azure Pipielines YML file that evalues the patch series in a PR using the +# python script BaseTools/Scripts/PatchCheck.py. +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +# https://github.com/tianocore +# +## + +trigger: none + +pr: +- master + +pool: + vmImage: 'ubuntu-latest' + +steps: +- checkout: self + clean: true + +- task: UsePythonVersion@0 + inputs: + versionSpec: '3.7.x' + architecture: 'x64' + +- script: | + git fetch origin $(System.PullRequest.TargetBranch):$(System.PullRequest.TargetBranch) + python BaseTools/Scripts/PatchCheck.py $(System.PullRequest.TargetBranch)..$(System.PullRequest.SourceCommitId) + displayName: 'Use PatchCheck.py to verify patch series in pull request' diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml new file mode 100644 index 0000000000..2143162e02 --- /dev/null +++ b/.azurepipelines/Windows-VS2019.yml @@ -0,0 +1,18 @@ +## +# Azure Pipeline build file for a build using Windows and VS2019 +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +trigger: +- master + +pr: +- master + +jobs: +- template: templates/pr-gate-build-job.yml + parameters: + tool_chain_tag: 'VS2019' + vm_image: 'windows-latest' + arch_list: "IA32,X64" diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml new file mode 100644 index 0000000000..dad59e2a1f --- /dev/null +++ b/.azurepipelines/templates/basetools-build-steps.yml @@ -0,0 +1,36 @@ +# File templates/basetools-build-job.yml +# +# template file to build basetools +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +parameters: + tool_chain_tag: '' + +steps: +- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}: + - bash: sudo apt-get update + displayName: Update apt + condition: and(gt(variables.pkg_count, 0), succeeded()) + + - bash: sudo apt-get install gcc g++ make uuid-dev + displayName: Install required tools + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CmdLine@1 + displayName: Build Base Tools from source + inputs: + filename: python + arguments: BaseTools/Edk2ToolsBuild.py -t ${{ parameters.tool_chain_tag }} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CopyFiles@2 + displayName: "Copy base tools build log" + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + SourceFolder: 'BaseTools/BaseToolsBuild' + contents: | + BASETOOLS_BUILD*.* + flattenFolders: true + condition: and(gt(variables.pkg_count, 0), succeededOrFailed()) diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml new file mode 100644 index 0000000000..523ce77544 --- /dev/null +++ b/.azurepipelines/templates/pr-gate-build-job.yml @@ -0,0 +1,79 @@ +# File templates/pr-gate-build-job.yml +# +# template file used to build supported packages. +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + tool_chain_tag: '' + vm_image: '' + arch_list: '' + +# Build step +jobs: + +- job: Build_${{ parameters.tool_chain_tag }} + + #Use matrix to speed up the build process + strategy: + matrix: + TARGET_MDE_CPU: + Build.Pkgs: 'MdePkg,UefiCpuPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + TARGET_MDEMODULE_DEBUG: + Build.Pkgs: 'MdeModulePkg' + Build.Targets: 'DEBUG' + TARGET_MDEMODULE_RELEASE: + Build.Pkgs: 'MdeModulePkg' + Build.Targets: 'RELEASE,NO-TARGET' + TARGET_NETWORK: + Build.Pkgs: 'NetworkPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + TARGET_OTHER: + Build.Pkgs: 'PcAtChipsetPkg,ShellPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + TARGET_FMP: + Build.Pkgs: 'FmpDevicePkg,FatPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + TARGET_CRYPTO: + Build.Pkgs: 'CryptoPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + TARGET_SECURITY: + Build.Pkgs: 'SecurityPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET' + + workspace: + clean: all + + pool: + vmImage: ${{ parameters.vm_image }} + + steps: + - template: pr-gate-steps.yml + parameters: + tool_chain_tag: ${{ parameters.tool_chain_tag }} + build_pkgs: $(Build.Pkgs) + build_targets: $(Build.Targets) + build_archs: ${{ parameters.arch_list }} + +- job: FINISHED + dependsOn: Build_${{ parameters.tool_chain_tag }} + condition: succeeded() + steps: + - checkout: none + - script: | + echo FINISHED + sleep 10 + displayName: FINISHED + +- job: FAILED + dependsOn: Build_${{ parameters.tool_chain_tag }} + condition: failed() + steps: + - checkout: none + - script: | + echo FAILED + sleep 10 + displayName: FAILED diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml new file mode 100644 index 0000000000..d3c7eec4ad --- /dev/null +++ b/.azurepipelines/templates/pr-gate-steps.yml @@ -0,0 +1,129 @@ +# File templates/pr-gate-steps.yml +# +# template file containing the steps to build +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + tool_chain_tag: '' + build_pkgs: '' + build_targets: '' + build_archs: '' + +steps: +- checkout: self + clean: true + fetchDepth: 1 + +- task: UsePythonVersion@0 + inputs: + versionSpec: '3.7.x' + architecture: 'x64' + +- script: pip install -r requirements.txt --upgrade + displayName: 'Install/Upgrade pip modules' + +# Set default +- bash: | + echo "##vso[task.setvariable variable=pkgs_to_build]${{ parameters.build_pkgs }}" + echo "##vso[task.setvariable variable=pkg_count]${{ 1 }}" + +# trim the package list if this is a PR +- task: CmdLine@1 + displayName: Check if ${{ parameters.build_pkgs }} need testing + inputs: + filename: stuart_pr_eval + arguments: -c .pytool/CISettings.py -p ${{ parameters.build_pkgs }} --pr-target origin/$(System.PullRequest.targetBranch) --output-csv-format-string "##vso[task.setvariable variable=pkgs_to_build;isOutpout=true]{pkgcsv}" --output-count-format-string "##vso[task.setvariable variable=pkg_count;isOutpout=true]{pkgcount}" + condition: eq(variables['Build.Reason'], 'PullRequest') + +# build basetools +- template: basetools-build-steps.yml + parameters: + tool_chain_tag: ${{ parameters.tool_chain_tag }} + +# install spell check prereqs +- template: spell-check-prereq-steps.yml + +# Build repo +- task: CmdLine@1 + displayName: Setup ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_setup + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CmdLine@1 + displayName: Update ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_update + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CmdLine@1 + displayName: Build and Test ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_ci_build + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# Publish Test Results to Azure Pipelines/TFS +- task: PublishTestResults@2 + displayName: 'Publish junit test results' + continueOnError: true + condition: and( succeededOrFailed(),gt(variables.pkg_count, 0)) + inputs: + testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit + testResultsFiles: 'Build/TestSuites.xml' + #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional + mergeTestResults: true # Optional + testRunTitle: $(System.JobName) # Optional + #buildPlatform: # Optional + #buildConfiguration: # Optional + publishRunAttachments: true # Optional + +# Publish Test Results to Azure Pipelines/TFS +- task: PublishTestResults@2 + displayName: 'Publish host based test results for $(System.JobName)' + continueOnError: true + condition: and( succeededOrFailed(), gt(variables.pkg_count, 0)) + inputs: + testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit + testResultsFiles: 'Build/**/*.result.xml' + #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional + mergeTestResults: false # Optional + testRunTitle: ${{ parameters.build_pkgs }} # Optional + #buildPlatform: # Optional + #buildConfiguration: # Optional + publishRunAttachments: true # Optional + +# Copy the build logs to the artifact staging directory +- task: CopyFiles@2 + displayName: "Copy build logs" + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + SourceFolder: 'Build' + contents: | + BUILDLOG_*.txt + BUILDLOG_*.md + CI_*.txt + CI_*.md + CISETUP.txt + SETUPLOG.txt + UPDATE_LOG.txt + PREVALLOG.txt + TestSuites.xml + **/BUILD_TOOLS_REPORT.html + **/OVERRIDELOG.TXT + flattenFolders: true + condition: succeededOrFailed() + +# Publish build artifacts to Azure Artifacts/TFS or a file share +- task: PublishBuildArtifacts@1 + continueOnError: true + displayName: "Publish build logs" + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: 'Build Logs $(System.JobName)' + condition: succeededOrFailed() diff --git a/.azurepipelines/templates/spell-check-prereq-steps.yml b/.azurepipelines/templates/spell-check-prereq-steps.yml new file mode 100644 index 0000000000..7d7797ffed --- /dev/null +++ b/.azurepipelines/templates/spell-check-prereq-steps.yml @@ -0,0 +1,21 @@ +# File templates/spell-check-prereq-steps.yml +# +# template file used to install spell checking prerequisits +# +# Copyright (c) 2019, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + none: '' + +steps: +- task: NodeTool@0 + inputs: + versionSpec: '10.x' + #checkLatest: false # Optional + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- script: npm install -g cspell + displayName: 'Install cspell npm' + condition: and(gt(variables.pkg_count, 0), succeeded()) -- 2.21.0.windows.1