* [PATCH v6 0/3] Add code coverage support for GCC/MSVC
@ 2022-12-14 3:19 Guo, Gua
2022-12-14 3:19 ` [PATCH v6 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC Guo, Gua
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-14 3:19 UTC (permalink / raw)
To: devel; +Cc: Gua Guo
From: Gua Guo <gua.guo@intel.com>
V1: Add coverage option for GCC
V2: Add ReadMe.md for how to generate coverage report
V3: Add VS2019 and GCC code coverage support
V4: Add VS2019 and GCC Azure CI/CD support
V5: Fix some typo and some flow issue
V6: Remove html coverage information
- Due to python 3.11 install lxml will be failure,
pycobertura need it to convert cobertura format to
html file.
- Add section for developer how to use OpenCppCoverage
on IDE Visual Studio
Gua Guo (3):
UnitTestFrameworkPkg: Add code coverage support for GCC
BaseTools/Plugin: Add coverage support for Unit Test
.azurepipelines: Install code coverage tool
.azurepipelines/Ubuntu-GCC5.yml | 5 +-
.azurepipelines/Windows-VS2019.yml | 5 +
.../templates/pr-gate-build-job.yml | 2 +
.azurepipelines/templates/pr-gate-steps.yml | 4 +
.../HostBasedUnitTestRunner.py | 100 ++++++++++++++++++
UnitTestFrameworkPkg/ReadMe.md | 45 +++++++-
.../UnitTestFrameworkPkg.ci.yaml | 1 +
.../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
pip-requirements.txt | 2 +
9 files changed, 163 insertions(+), 4 deletions(-)
--
2.31.1.windows.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v6 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-12-14 3:19 [PATCH v6 0/3] Add code coverage support for GCC/MSVC Guo, Gua
@ 2022-12-14 3:19 ` Guo, Gua
2022-12-14 3:19 ` [PATCH v6 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
2022-12-14 3:19 ` [PATCH v6 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2 siblings, 0 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-14 3:19 UTC (permalink / raw)
To: devel
Cc: Gua Guo, Michael D Kinney, Sean Brogan, Bret Barkelew,
Michael Kubacki
From: Gua Guo <gua.guo@intel.com>
In order to collect code coverage after running executable
file, generate *.gcda and *.gcno file that require by lcov tool
to generate code coverage report.
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael Kubacki <mikuback@linux.microsoft.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
UnitTestFrameworkPkg/ReadMe.md | 45 ++++++++++++++++++-
.../UnitTestFrameworkPkg.ci.yaml | 1 +
.../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
3 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md
index 9ce04b7f3e..5987385c06 100644
--- a/UnitTestFrameworkPkg/ReadMe.md
+++ b/UnitTestFrameworkPkg/ReadMe.md
@@ -115,7 +115,7 @@ you should be good to go.
See this example in 'SampleUnitTestUefiShell.inf'...
-```
+```inf
[Packages]
MdePkg/MdePkg.dec
@@ -130,7 +130,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
Also, if you want you test to automatically be picked up by the Test Runner plugin, you will need
to make sure that the module `BASE_NAME` contains the word `Test`...
-```
+```inf
[Defines]
BASE_NAME = SampleUnitTestUefiShell
```
@@ -583,6 +583,47 @@ GTEST_OUTPUT=xml:<absolute or relative path to output file>
This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
+### Code Coverage
+
+Host based Unit Tests will automatically enable coverage data.
+
+For Windows, This is primarily leverage for pipeline builds, but this can be leveraged locally using the
+OpenCppCoverage windows tool to parse coverage data to cobertura xml format.
+
+- Windows Prerequisite
+ ```bash
+ Download and install https://github.com/OpenCppCoverage/OpenCppCoverage/releases
+ python -m pip install --upgrade -r ./pip-requirements.txt
+ stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
+ Open Build/coverage.xml
+ ```
+
+ - How to see code coverage data on IDE Visual Studio
+ ```
+ Open Visual Studio VS2019 or above version
+ Click "Tools" -> "OpenCppCoverage Settings"
+ Fill your execute file into "Program to run:"
+ Click "Tools" -> "Run OpenCppCoverage"
+ ```
+
+
+For Linux, This is primarily leveraged for pipeline builds, but this can be leveraged locally using the
+lcov linux tool, and parsed using the lcov_cobertura python tool to parse it to cobertura xml format.
+
+- Linux Prerequisite
+ ```bash
+ sudo apt-get install -y lcov
+ python -m pip install --upgrade -r ./pip-requirements.txt
+ stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
+ Open Build/coverage.xml
+ ```
+ - How to see code coverage data on IDE Visual Studio Code
+ ```
+ Download plugin "Coverage Gutters"
+ Press Hot Key "Ctrl + Shift + P" and click option "Coverage Gutters: Display Coverage"
+ ```
+
+
### Important Note
This works on both Windows and Linux but is currently limited to x64 architectures. Working on getting others, but we
diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
index 072df6208c..d8f8e024c4 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
@@ -91,6 +91,7 @@
"pytools",
"NOFAILURE",
"cmockery",
+ "cobertura",
"DHAVE", # build flag for cmocka in the INF
"gtest", # file name in GoogleTestLib.inf
"corthon", # Contact GitHub account in Readme
diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
index 8009337552..7f5dfa30ed 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
@@ -27,7 +27,8 @@
GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
!endif
-
+ GCC:*_GCC5_*_CC_FLAGS = --coverage
+ GCC:*_GCC5_*_DLINK_FLAGS = --coverage
[BuildOptions.common.EDKII.HOST_APPLICATION]
#
# MSFT
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v6 2/3] BaseTools/Plugin: Add coverage support for Unit Test
2022-12-14 3:19 [PATCH v6 0/3] Add code coverage support for GCC/MSVC Guo, Gua
2022-12-14 3:19 ` [PATCH v6 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC Guo, Gua
@ 2022-12-14 3:19 ` Guo, Gua
2022-12-14 3:19 ` [PATCH v6 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2 siblings, 0 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-14 3:19 UTC (permalink / raw)
To: devel
Cc: Gua Guo, Bob Feng, Bret Barkelew, Liming Gao, Michael D Kinney,
Sean Brogan
From: Gua Guo <gua.guo@intel.com>
For GCC, use lcov to generate Unit Test code coverage
report
For VS2019, use OpenCppCoverage to generate code
coverage report
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <gaoliming@bysoft.com.cn>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
.../HostBasedUnitTestRunner.py | 100 ++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index a8220aacd3..ad63923b69 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -115,4 +115,104 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
" %s - %s" % (case.attrib['name'], result.text))
failure_count += 1
+ if thebuilder.env.GetValue("CODE_COVERAGE") != "FALSE":
+ if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "GCC5":
+ self.gen_code_coverage_gcc(thebuilder)
+ elif thebuilder.env.GetValue("TOOL_CHAIN_TAG").startswith ("VS"):
+ self.gen_code_coverage_msvc(thebuilder)
+ else:
+ logging.info("Skipping code coverage. Currently, support GCC and MSVC compiler.")
+
return failure_count
+
+ def gen_code_coverage_gcc(self, thebuilder):
+ logging.info("Generating UnitTest code coverage")
+
+ buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
+ workspace = thebuilder.env.GetValue("WORKSPACE")
+
+ # Generate base code coverage for all source files
+ ret = RunCmd("lcov", f"--no-external --capture --initial --directory {buildOutputBase} --output-file {buildOutputBase}/cov-base.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to build initial coverage data.")
+ return 1
+
+ # Coverage data for tested files only
+ ret = RunCmd("lcov", f"--capture --directory {buildOutputBase}/ --output-file {buildOutputBase}/coverage-test.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to build coverage data for tested files.")
+ return 1
+
+ # Aggregate all coverage data
+ ret = RunCmd("lcov", f"--add-tracefile {buildOutputBase}/cov-base.info --add-tracefile {buildOutputBase}/coverage-test.info --output-file {buildOutputBase}/total-coverage.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to aggregate coverage data.")
+ return 1
+
+ # Generate coverage XML
+ ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info -o {buildOutputBase}/compare.xml")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate coverage XML.")
+ return 1
+
+ # Filter out auto-generated and test code
+ ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {buildOutputBase}/coverage.xml")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed generate filtered coverage XML.")
+ return 1
+
+ # Generate all coverage file
+ testCoverageList = glob.glob (f"{workspace}/Build/**/total-coverage.info", recursive=True)
+
+ coverageFile = ""
+ for testCoverage in testCoverageList:
+ coverageFile += " --add-tracefile " + testCoverage
+ ret = RunCmd("lcov", f"{coverageFile} --output-file {workspace}/Build/all-coverage.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed generate all coverage file.")
+ return 1
+
+ # Generate and XML file if requested.for all package
+ if os.path.isfile(f"{workspace}/Build/coverage.xml"):
+ os.remove(f"{workspace}/Build/coverage.xml")
+ ret = RunCmd("lcov_cobertura",f"{workspace}/Build/all-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {workspace}/Build/coverage.xml")
+
+ return 0
+
+
+ def gen_code_coverage_msvc(self, thebuilder):
+ logging.info("Generating UnitTest code coverage")
+
+
+ buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
+ testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
+ workspace = thebuilder.env.GetValue("WORKSPACE")
+ workspace = (workspace + os.sep) if workspace[-1] != os.sep else workspace
+ # Generate coverage file
+ coverageFile = ""
+ for testFile in testList:
+ ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
+ coverageFile += " --input_coverage=" + testFile + ".cov"
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to collect coverage data.")
+ return 1
+
+ DiskName = workspace[:workspace.find (":\\") + 2]
+ # Generate and XML file if requested.by each package
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} --working_dir={workspace}Build {coverageFile}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
+ return 1
+
+ # Generate total report XML file for all package
+ testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
+ coverageFile = ""
+ for testCoverage in testCoverageList:
+ coverageFile += " --input_coverage=" + testCoverage
+
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
+ return 1
+
+ return 0
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-14 3:19 [PATCH v6 0/3] Add code coverage support for GCC/MSVC Guo, Gua
2022-12-14 3:19 ` [PATCH v6 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC Guo, Gua
2022-12-14 3:19 ` [PATCH v6 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
@ 2022-12-14 3:19 ` Guo, Gua
2022-12-20 21:23 ` Michael D Kinney
2 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-12-14 3:19 UTC (permalink / raw)
To: devel; +Cc: Gua Guo, Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
From: Gua Guo <gua.guo@intel.com>
For Windows add below tool for code coverage
1. OpenCppCoverage: parsing pdb file to generate coverage
data
2. pycobertura: show up html format data for coverage data
For Linux add below tool for code coverage
1. lcov: parsing gcda gcno file to generate coverage data
2. lcov-cobertura: convert coverage data to cobertura format
3. pycobertura: show up html format data for coverage data
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
.azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
.azurepipelines/Windows-VS2019.yml | 5 +++++
.azurepipelines/templates/pr-gate-build-job.yml | 2 ++
.azurepipelines/templates/pr-gate-steps.yml | 4 ++++
pip-requirements.txt | 2 ++
5 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 1acd8d2a46..f83951eeaf 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -19,4 +19,7 @@ jobs:
tool_chain_tag: 'GCC5'
vm_image: 'ubuntu-latest'
arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
-
+ extra_install_step:
+ - bash: sudo apt-get install -y lcov
+ displayName: Install Code Coverage Tools
+ condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
index e4bd4b1d22..c07e5bb434 100644
--- a/.azurepipelines/Windows-VS2019.yml
+++ b/.azurepipelines/Windows-VS2019.yml
@@ -18,3 +18,8 @@ jobs:
tool_chain_tag: 'VS2019'
vm_image: 'windows-2019'
arch_list: "IA32,X64"
+ extra_install_step:
+ - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
+ displayName: Install Code Coverage Tool
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index 7f88b41dc8..0ff1ce2952 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
tool_chain_tag: ''
vm_image: ''
arch_list: ''
+ extra_install_step: []
# Build step
jobs:
@@ -77,3 +78,4 @@ jobs:
build_pkgs: $(Build.Pkgs)
build_targets: $(Build.Targets)
build_archs: ${{ parameters.arch_list }}
+ extra_install_step: ${{ parameters.extra_install_step }}
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index cb431e53fc..080f60aea6 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -12,6 +12,7 @@ parameters:
build_pkgs: ''
build_targets: ''
build_archs: ''
+ extra_install_step: []
steps:
- checkout: self
@@ -37,6 +38,8 @@ steps:
displayName: fetch target branch
condition: eq(variables['Build.Reason'], 'PullRequest')
+- ${{ parameters.extra_install_step }}
+
# trim the package list if this is a PR
- task: CmdLine@1
displayName: Check if ${{ parameters.build_pkgs }} need testing
@@ -125,6 +128,7 @@ steps:
TestSuites.xml
**/BUILD_TOOLS_REPORT.html
**/OVERRIDELOG.TXT
+ coverage.xml
flattenFolders: true
condition: succeededOrFailed()
diff --git a/pip-requirements.txt b/pip-requirements.txt
index be8c7a1c37..4ffcadddd8 100644
--- a/pip-requirements.txt
+++ b/pip-requirements.txt
@@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
edk2-pytool-extensions~=0.20.0
edk2-basetools==0.1.39
antlr4-python3-runtime==4.7.1
+lcov-cobertura==2.0.2
+
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-14 3:19 ` [PATCH v6 3/3] .azurepipelines: Install code coverage tool Guo, Gua
@ 2022-12-20 21:23 ` Michael D Kinney
2022-12-21 1:32 ` Guo, Gua
0 siblings, 1 reply; 11+ messages in thread
From: Michael D Kinney @ 2022-12-20 21:23 UTC (permalink / raw)
To: Guo, Gua, devel@edk2.groups.io, Kinney, Michael D, Sean Brogan,
Kubacki, Michael
Cc: Sean Brogan, Barkelew, Bret, Gao, Liming, Kinney, Michael D
Hi Gua,
I created a PR with this series.
https://github.com/tianocore/edk2/pull/3795
When I follow link to Azure Pipelines for the GCC builds, I can see the coverage.xml file in the ZIP file of artifacts.
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&view=results
But I do not see an information from Azure Pielines view on code coverage or a way to view the lines covered from the Azure Pipelines web pages.
Are there additional items to enable?
Mike
> -----Original Message-----
> From: Guo, Gua <gua.guo@intel.com>
> Sent: Tuesday, December 13, 2022 7:20 PM
> To: devel@edk2.groups.io
> Cc: Guo, Gua <gua.guo@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>;
> Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>
> Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> From: Gua Guo <gua.guo@intel.com>
>
> For Windows add below tool for code coverage
> 1. OpenCppCoverage: parsing pdb file to generate coverage
> data
> 2. pycobertura: show up html format data for coverage data
>
> For Linux add below tool for code coverage
> 1. lcov: parsing gcda gcno file to generate coverage data
> 2. lcov-cobertura: convert coverage data to cobertura format
> 3. pycobertura: show up html format data for coverage data
>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
> .azurepipelines/Windows-VS2019.yml | 5 +++++
> .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
> pip-requirements.txt | 2 ++
> 5 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
> index 1acd8d2a46..f83951eeaf 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -19,4 +19,7 @@ jobs:
> tool_chain_tag: 'GCC5'
>
> vm_image: 'ubuntu-latest'
>
> arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> -
>
> + extra_install_step:
>
> + - bash: sudo apt-get install -y lcov
>
> + displayName: Install Code Coverage Tools
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
> index e4bd4b1d22..c07e5bb434 100644
> --- a/.azurepipelines/Windows-VS2019.yml
> +++ b/.azurepipelines/Windows-VS2019.yml
> @@ -18,3 +18,8 @@ jobs:
> tool_chain_tag: 'VS2019'
>
> vm_image: 'windows-2019'
>
> arch_list: "IA32,X64"
>
> + extra_install_step:
>
> + - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> + displayName: Install Code Coverage Tool
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> +
>
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
> index 7f88b41dc8..0ff1ce2952 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
> tool_chain_tag: ''
>
> vm_image: ''
>
> arch_list: ''
>
> + extra_install_step: []
>
>
>
> # Build step
>
> jobs:
>
> @@ -77,3 +78,4 @@ jobs:
> build_pkgs: $(Build.Pkgs)
>
> build_targets: $(Build.Targets)
>
> build_archs: ${{ parameters.arch_list }}
>
> + extra_install_step: ${{ parameters.extra_install_step }}
>
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fc..080f60aea6 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
> build_pkgs: ''
>
> build_targets: ''
>
> build_archs: ''
>
> + extra_install_step: []
>
>
>
> steps:
>
> - checkout: self
>
> @@ -37,6 +38,8 @@ steps:
> displayName: fetch target branch
>
> condition: eq(variables['Build.Reason'], 'PullRequest')
>
>
>
> +- ${{ parameters.extra_install_step }}
>
> +
>
> # trim the package list if this is a PR
>
> - task: CmdLine@1
>
> displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> @@ -125,6 +128,7 @@ steps:
> TestSuites.xml
>
> **/BUILD_TOOLS_REPORT.html
>
> **/OVERRIDELOG.TXT
>
> + coverage.xml
>
> flattenFolders: true
>
> condition: succeededOrFailed()
>
>
>
> diff --git a/pip-requirements.txt b/pip-requirements.txt
> index be8c7a1c37..4ffcadddd8 100644
> --- a/pip-requirements.txt
> +++ b/pip-requirements.txt
> @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
> edk2-pytool-extensions~=0.20.0
>
> edk2-basetools==0.1.39
>
> antlr4-python3-runtime==4.7.1
>
> +lcov-cobertura==2.0.2
>
> +
>
> --
> 2.31.1.windows.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-20 21:23 ` Michael D Kinney
@ 2022-12-21 1:32 ` Guo, Gua
2022-12-21 3:11 ` [edk2-devel] " Michael Kubacki
2022-12-21 3:17 ` Michael D Kinney
0 siblings, 2 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 1:32 UTC (permalink / raw)
To: Kinney, Michael D, devel@edk2.groups.io, Sean Brogan,
Kubacki, Michael
Cc: Sean Brogan, Barkelew, Bret, Gao, Liming
[-- Attachment #1.1: Type: text/plain, Size: 8147 bytes --]
@Kinney, Michael D<mailto:michael.d.kinney@intel.com>
I assume you want to see pipeline "Code Coverage" like below “3. Azure Pipeline Change” diagram.
But it will have a limitation. Current Edk2 CI/CD design is use “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask for each package. And each package will have itself Coverage Data, but the design can not be covered by Azure API “PublishCodeCoverageResult”.
That’s why I don’t integrated it into the change.
[cid:image006.png@01D9151F.141EE9C0]
1. Azure API doc: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines
1. Code Change:
[cid:image001.png@01D9151D.697795E0]
1. Azure Pipeline Change:
VS2019
GCC5
https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220422452
[cid:image002.png@01D9151E.01661610]
https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220615640
[cid:image003.png@01D9151E.01661610]
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
[cid:image004.png@01D9151E.01661610]
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
[cid:image005.png@01D9151E.4BF4D900]
Thanks,
Gua
-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Wednesday, December 21, 2022 5:23 AM
To: Guo, Gua <gua.guo@intel.com>; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Kubacki, Michael <michael.kubacki@microsoft.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming <gaoliming@byosoft.com.cn>; Kinney, Michael D <michael.d.kinney@intel.com>
Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
Hi Gua,
I created a PR with this series.
https://github.com/tianocore/edk2/pull/3795
When I follow link to Azure Pipelines for the GCC builds, I can see the coverage.xml file in the ZIP file of artifacts.
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&view=results
But I do not see an information from Azure Pielines view on code coverage or a way to view the lines covered from the Azure Pipelines web pages.
Are there additional items to enable?
Mike
> -----Original Message-----
> From: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>
> Sent: Tuesday, December 13, 2022 7:20 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
> <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>; Kinney, Michael D
> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> From: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
>
> For Windows add below tool for code coverage 1. OpenCppCoverage:
> parsing pdb file to generate coverage data 2. pycobertura: show up
> html format data for coverage data
>
> For Linux add below tool for code coverage 1. lcov: parsing gcda gcno
> file to generate coverage data 2. lcov-cobertura: convert coverage
> data to cobertura format 3. pycobertura: show up html format data for
> coverage data
>
> Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
> Cc: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
> Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Signed-off-by: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
> ---
> .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
> .azurepipelines/Windows-VS2019.yml | 5 +++++
> .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
> pip-requirements.txt | 2 ++
> 5 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml
> b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -19,4 +19,7 @@ jobs:
> tool_chain_tag: 'GCC5'
>
> vm_image: 'ubuntu-latest'
>
> arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> -
>
> + extra_install_step:
>
> + - bash: sudo apt-get install -y lcov
>
> + displayName: Install Code Coverage Tools
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> diff --git a/.azurepipelines/Windows-VS2019.yml
> b/.azurepipelines/Windows-VS2019.yml
> index e4bd4b1d22..c07e5bb434 100644
> --- a/.azurepipelines/Windows-VS2019.yml
> +++ b/.azurepipelines/Windows-VS2019.yml
> @@ -18,3 +18,8 @@ jobs:
> tool_chain_tag: 'VS2019'
>
> vm_image: 'windows-2019'
>
> arch_list: "IA32,X64"
>
> + extra_install_step:
>
> + - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> + displayName: Install Code Coverage Tool
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> +
>
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> b/.azurepipelines/templates/pr-gate-build-job.yml
> index 7f88b41dc8..0ff1ce2952 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
> tool_chain_tag: ''
>
> vm_image: ''
>
> arch_list: ''
>
> + extra_install_step: []
>
>
>
> # Build step
>
> jobs:
>
> @@ -77,3 +78,4 @@ jobs:
> build_pkgs: $(Build.Pkgs)
>
> build_targets: $(Build.Targets)
>
> build_archs: ${{ parameters.arch_list }}
>
> + extra_install_step: ${{ parameters.extra_install_step }}
>
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fc..080f60aea6 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
> build_pkgs: ''
>
> build_targets: ''
>
> build_archs: ''
>
> + extra_install_step: []
>
>
>
> steps:
>
> - checkout: self
>
> @@ -37,6 +38,8 @@ steps:
> displayName: fetch target branch
>
> condition: eq(variables['Build.Reason'], 'PullRequest')
>
>
>
> +- ${{ parameters.extra_install_step }}
>
> +
>
> # trim the package list if this is a PR
>
> - task: CmdLine@1
>
> displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> @@ -125,6 +128,7 @@ steps:
> TestSuites.xml
>
> **/BUILD_TOOLS_REPORT.html
>
> **/OVERRIDELOG.TXT
>
> + coverage.xml
>
> flattenFolders: true
>
> condition: succeededOrFailed()
>
>
>
> diff --git a/pip-requirements.txt b/pip-requirements.txt index
> be8c7a1c37..4ffcadddd8 100644
> --- a/pip-requirements.txt
> +++ b/pip-requirements.txt
> @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
> edk2-pytool-extensions~=0.20.0
>
> edk2-basetools==0.1.39
>
> antlr4-python3-runtime==4.7.1
>
> +lcov-cobertura==2.0.2
>
> +
>
> --
> 2.31.1.windows.1
[-- Attachment #1.2: Type: text/html, Size: 26464 bytes --]
[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 97870 bytes --]
[-- Attachment #3: image002.png --]
[-- Type: image/png, Size: 28311 bytes --]
[-- Attachment #4: image003.png --]
[-- Type: image/png, Size: 30419 bytes --]
[-- Attachment #5: image004.png --]
[-- Type: image/png, Size: 49974 bytes --]
[-- Attachment #6: image005.png --]
[-- Type: image/png, Size: 52236 bytes --]
[-- Attachment #7: image006.png --]
[-- Type: image/png, Size: 48816 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-21 1:32 ` Guo, Gua
@ 2022-12-21 3:11 ` Michael Kubacki
2022-12-21 4:20 ` Guo, Gua
2022-12-21 3:17 ` Michael D Kinney
1 sibling, 1 reply; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 3:11 UTC (permalink / raw)
To: devel, gua.guo, Kinney, Michael D, Sean Brogan, Kubacki, Michael
Cc: Barkelew, Bret, Gao, Liming
I might be misunderstanding the problem, but can you merge the XML file
from each package into one and then pass that to the
"PublishCodeCoverageResult" task?
Perhaps with something like this
https://github.com/danielpalme/ReportGenerator
Thanks,
Michael
On 12/20/2022 8:32 PM, Guo, Gua wrote:
> @Kinney, Michael D <mailto:michael.d.kinney@intel.com>
>
> I assume you want to see pipeline "Code Coverage" like below “3. Azure
> Pipeline Change” diagram.
>
> But it will have a limitation. Current Edk2 CI/CD design is use “Ubuntu
> GCC PR5” and “Windows VS2019 PR” that include each subtask for each
> package. And each package will have itself Coverage Data, but the design
> can not be covered by Azure API “PublishCodeCoverageResult”.
>
> That’s why I don’t integrated it into the change.
>
> 1. Azure API doc:
> https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines>
>
> 2. Code Change:
>
> 3. Azure Pipeline Change:
>
> VS2019
>
>
>
> GCC5
>
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220422452
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220422452>
>
>
>
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220615640
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220615640>
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab>
>
>
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab>
>
> Thanks,
>
> Gua
>
> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Wednesday, December 21, 2022 5:23 AM
> To: Guo, Gua <gua.guo@intel.com>; devel@edk2.groups.io; Kinney, Michael
> D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>;
> Kubacki, Michael <michael.kubacki@microsoft.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret
> <bret.barkelew@microsoft.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> Hi Gua,
>
> I created a PR with this series.
>
> https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>
>
> When I follow link to Azure Pipelines for the GCC builds, I can see the
> coverage.xml file in the ZIP file of artifacts.
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&view=results
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&view=results>
>
> But I do not see an information from Azure Pielines view on code
> coverage or a way to view the lines covered from the Azure Pipelines web
> pages.
>
> Are there additional items to enable?
>
> Mike
>
> > -----Original Message-----
>
> > From: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com>>
>
> > Sent: Tuesday, December 13, 2022 7:20 PM
>
> > To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
>
> > Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com>>; Sean Brogan
>
> > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com>>;
> Barkelew, Bret
>
> > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com>>;
> Kinney, Michael D
>
> > <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>;
> Gao, Liming <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>
>
> > Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> >
>
> > From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com>>
>
> >
>
> > For Windows add below tool for code coverage 1. OpenCppCoverage:
>
> > parsing pdb file to generate coverage data 2. pycobertura: show up
>
> > html format data for coverage data
>
> >
>
> > For Linux add below tool for code coverage 1. lcov: parsing gcda gcno
>
> > file to generate coverage data 2. lcov-cobertura: convert coverage
>
> > data to cobertura format 3. pycobertura: show up html format data for
>
> > coverage data
>
> >
>
> > Cc: Sean Brogan <sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com>>
>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com
> <mailto:Bret.Barkelew@microsoft.com>>
>
> > Cc: Michael D Kinney <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Cc: Liming Gao <gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>
>
> > Signed-off-by: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com>>
>
> > ---
>
> > .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
>
> > .azurepipelines/Windows-VS2019.yml | 5 +++++
>
> > .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
>
> > .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
>
> > pip-requirements.txt | 2 ++
>
> > 5 files changed, 17 insertions(+), 1 deletion(-)
>
> >
>
> > diff --git a/.azurepipelines/Ubuntu-GCC5.yml
>
> > b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf 100644
>
> > --- a/.azurepipelines/Ubuntu-GCC5.yml
>
> > +++ b/.azurepipelines/Ubuntu-GCC5.yml
>
> > @@ -19,4 +19,7 @@ jobs:
>
> > tool_chain_tag: 'GCC5'
>
> >
>
> > vm_image: 'ubuntu-latest'
>
> >
>
> > arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> >
>
> > -
>
> >
>
> > + extra_install_step:
>
> >
>
> > + - bash: sudo apt-get install -y lcov
>
> >
>
> > + displayName: Install Code Coverage Tools
>
> >
>
> > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > diff --git a/.azurepipelines/Windows-VS2019.yml
>
> > b/.azurepipelines/Windows-VS2019.yml
>
> > index e4bd4b1d22..c07e5bb434 100644
>
> > --- a/.azurepipelines/Windows-VS2019.yml
>
> > +++ b/.azurepipelines/Windows-VS2019.yml
>
> > @@ -18,3 +18,8 @@ jobs:
>
> > tool_chain_tag: 'VS2019'
>
> >
>
> > vm_image: 'windows-2019'
>
> >
>
> > arch_list: "IA32,X64"
>
> >
>
> > + extra_install_step:
>
> >
>
> > + - powershell: choco install opencppcoverage; Write-Host
> "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> >
>
> > + displayName: Install Code Coverage Tool
>
> >
>
> > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > +
>
> >
>
> > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
>
> > b/.azurepipelines/templates/pr-gate-build-job.yml
>
> > index 7f88b41dc8..0ff1ce2952 100644
>
> > --- a/.azurepipelines/templates/pr-gate-build-job.yml
>
> > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
>
> > @@ -12,6 +12,7 @@ parameters:
>
> > tool_chain_tag: ''
>
> >
>
> > vm_image: ''
>
> >
>
> > arch_list: ''
>
> >
>
> > + extra_install_step: []
>
> >
>
> >
>
> >
>
> > # Build step
>
> >
>
> > jobs:
>
> >
>
> > @@ -77,3 +78,4 @@ jobs:
>
> > build_pkgs: $(Build.Pkgs)
>
> >
>
> > build_targets: $(Build.Targets)
>
> >
>
> > build_archs: ${{ parameters.arch_list }}
>
> >
>
> > + extra_install_step: ${{ parameters.extra_install_step }}
>
> >
>
> > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
>
> > b/.azurepipelines/templates/pr-gate-steps.yml
>
> > index cb431e53fc..080f60aea6 100644
>
> > --- a/.azurepipelines/templates/pr-gate-steps.yml
>
> > +++ b/.azurepipelines/templates/pr-gate-steps.yml
>
> > @@ -12,6 +12,7 @@ parameters:
>
> > build_pkgs: ''
>
> >
>
> > build_targets: ''
>
> >
>
> > build_archs: ''
>
> >
>
> > + extra_install_step: []
>
> >
>
> >
>
> >
>
> > steps:
>
> >
>
> > - checkout: self
>
> >
>
> > @@ -37,6 +38,8 @@ steps:
>
> > displayName: fetch target branch
>
> >
>
> > condition: eq(variables['Build.Reason'], 'PullRequest')
>
> >
>
> >
>
> >
>
> > +- ${{ parameters.extra_install_step }}
>
> >
>
> > +
>
> >
>
> > # trim the package list if this is a PR
>
> >
>
> > - task: CmdLine@1
>
> >
>
> > displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> >
>
> > @@ -125,6 +128,7 @@ steps:
>
> > TestSuites.xml
>
> >
>
> > **/BUILD_TOOLS_REPORT.html
>
> >
>
> > **/OVERRIDELOG.TXT
>
> >
>
> > + coverage.xml
>
> >
>
> > flattenFolders: true
>
> >
>
> > condition: succeededOrFailed()
>
> >
>
> >
>
> >
>
> > diff --git a/pip-requirements.txt b/pip-requirements.txt index
>
> > be8c7a1c37..4ffcadddd8 100644
>
> > --- a/pip-requirements.txt
>
> > +++ b/pip-requirements.txt
>
> > @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
>
> > edk2-pytool-extensions~=0.20.0
>
> >
>
> > edk2-basetools==0.1.39
>
> >
>
> > antlr4-python3-runtime==4.7.1
>
> >
>
> > +lcov-cobertura==2.0.2
>
> >
>
> > +
>
> >
>
> > --
>
> > 2.31.1.windows.1
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-21 1:32 ` Guo, Gua
2022-12-21 3:11 ` [edk2-devel] " Michael Kubacki
@ 2022-12-21 3:17 ` Michael D Kinney
1 sibling, 0 replies; 11+ messages in thread
From: Michael D Kinney @ 2022-12-21 3:17 UTC (permalink / raw)
To: Guo, Gua, devel@edk2.groups.io, Sean Brogan, Kubacki, Michael,
Kinney, Michael D
Cc: Sean Brogan, Barkelew, Bret, Gao, Liming, Kinney, Michael D
[-- Attachment #1.1: Type: text/plain, Size: 9503 bytes --]
Hi Gua,
I see. The lack of merge capability means the results and view of source files covered inside Azure Pipelines are not meaningful.
I am ok with this initial version that really only provide code coverage information for local builds or local evaluation of XML files generated by CI into artifacts.
Reviewed-by: Michael D Kinney michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>
Please bring the current status of this feature to a future TianoCore Tools/CI meeting for discussion on how we can improve this feature.
Thanks,
Mike
From: Guo, Gua <gua.guo@intel.com>
Sent: Tuesday, December 20, 2022 5:32 PM
To: Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io; Sean Brogan <sean.brogan@microsoft.com>; Kubacki, Michael <michael.kubacki@microsoft.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming <gaoliming@byosoft.com.cn>
Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
@Kinney, Michael D<mailto:michael.d.kinney@intel.com>
I assume you want to see pipeline "Code Coverage" like below “3. Azure Pipeline Change” diagram.
But it will have a limitation. Current Edk2 CI/CD design is use “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask for each package. And each package will have itself Coverage Data, but the design can not be covered by Azure API “PublishCodeCoverageResult”.
That’s why I don’t integrated it into the change.
[cid:image006.png@01D914A7.B016F9D0]
1. Azure API doc: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines
1. Code Change:
[cid:image007.png@01D914A7.B016F9D0]
1. Azure Pipeline Change:
VS2019
GCC5
https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220422452
[cid:image008.png@01D914A7.B016F9D0]
https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220615640
[cid:image009.png@01D914A7.B016F9D0]
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
[cid:image010.png@01D914A7.B016F9D0]
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codecoverage-tab
[cid:image011.png@01D914A7.B016F9D0]
Thanks,
Gua
-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Wednesday, December 21, 2022 5:23 AM
To: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Kubacki, Michael <michael.kubacki@microsoft.com<mailto:michael.kubacki@microsoft.com>>
Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Barkelew, Bret <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
Hi Gua,
I created a PR with this series.
https://github.com/tianocore/edk2/pull/3795
When I follow link to Azure Pipelines for the GCC builds, I can see the coverage.xml file in the ZIP file of artifacts.
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&view=results
But I do not see an information from Azure Pielines view on code coverage or a way to view the lines covered from the Azure Pipelines web pages.
Are there additional items to enable?
Mike
> -----Original Message-----
> From: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>
> Sent: Tuesday, December 13, 2022 7:20 PM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
> <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>; Kinney, Michael D
> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> From: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
>
> For Windows add below tool for code coverage 1. OpenCppCoverage:
> parsing pdb file to generate coverage data 2. pycobertura: show up
> html format data for coverage data
>
> For Linux add below tool for code coverage 1. lcov: parsing gcda gcno
> file to generate coverage data 2. lcov-cobertura: convert coverage
> data to cobertura format 3. pycobertura: show up html format data for
> coverage data
>
> Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
> Cc: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
> Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Signed-off-by: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
> ---
> .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
> .azurepipelines/Windows-VS2019.yml | 5 +++++
> .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
> pip-requirements.txt | 2 ++
> 5 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml
> b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -19,4 +19,7 @@ jobs:
> tool_chain_tag: 'GCC5'
>
> vm_image: 'ubuntu-latest'
>
> arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> -
>
> + extra_install_step:
>
> + - bash: sudo apt-get install -y lcov
>
> + displayName: Install Code Coverage Tools
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> diff --git a/.azurepipelines/Windows-VS2019.yml
> b/.azurepipelines/Windows-VS2019.yml
> index e4bd4b1d22..c07e5bb434 100644
> --- a/.azurepipelines/Windows-VS2019.yml
> +++ b/.azurepipelines/Windows-VS2019.yml
> @@ -18,3 +18,8 @@ jobs:
> tool_chain_tag: 'VS2019'
>
> vm_image: 'windows-2019'
>
> arch_list: "IA32,X64"
>
> + extra_install_step:
>
> + - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> + displayName: Install Code Coverage Tool
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> +
>
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> b/.azurepipelines/templates/pr-gate-build-job.yml
> index 7f88b41dc8..0ff1ce2952 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
> tool_chain_tag: ''
>
> vm_image: ''
>
> arch_list: ''
>
> + extra_install_step: []
>
>
>
> # Build step
>
> jobs:
>
> @@ -77,3 +78,4 @@ jobs:
> build_pkgs: $(Build.Pkgs)
>
> build_targets: $(Build.Targets)
>
> build_archs: ${{ parameters.arch_list }}
>
> + extra_install_step: ${{ parameters.extra_install_step }}
>
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fc..080f60aea6 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
> build_pkgs: ''
>
> build_targets: ''
>
> build_archs: ''
>
> + extra_install_step: []
>
>
>
> steps:
>
> - checkout: self
>
> @@ -37,6 +38,8 @@ steps:
> displayName: fetch target branch
>
> condition: eq(variables['Build.Reason'], 'PullRequest')
>
>
>
> +- ${{ parameters.extra_install_step }}
>
> +
>
> # trim the package list if this is a PR
>
> - task: CmdLine@1
>
> displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> @@ -125,6 +128,7 @@ steps:
> TestSuites.xml
>
> **/BUILD_TOOLS_REPORT.html
>
> **/OVERRIDELOG.TXT
>
> + coverage.xml
>
> flattenFolders: true
>
> condition: succeededOrFailed()
>
>
>
> diff --git a/pip-requirements.txt b/pip-requirements.txt index
> be8c7a1c37..4ffcadddd8 100644
> --- a/pip-requirements.txt
> +++ b/pip-requirements.txt
> @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
> edk2-pytool-extensions~=0.20.0
>
> edk2-basetools==0.1.39
>
> antlr4-python3-runtime==4.7.1
>
> +lcov-cobertura==2.0.2
>
> +
>
> --
> 2.31.1.windows.1
[-- Attachment #1.2: Type: text/html, Size: 68542 bytes --]
[-- Attachment #2: image006.png --]
[-- Type: image/png, Size: 48816 bytes --]
[-- Attachment #3: image007.png --]
[-- Type: image/png, Size: 97870 bytes --]
[-- Attachment #4: image008.png --]
[-- Type: image/png, Size: 28311 bytes --]
[-- Attachment #5: image009.png --]
[-- Type: image/png, Size: 30419 bytes --]
[-- Attachment #6: image010.png --]
[-- Type: image/png, Size: 49974 bytes --]
[-- Attachment #7: image011.png --]
[-- Type: image/png, Size: 52236 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-21 3:11 ` [edk2-devel] " Michael Kubacki
@ 2022-12-21 4:20 ` Guo, Gua
2022-12-21 4:29 ` Michael Kubacki
0 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 4:20 UTC (permalink / raw)
To: Michael Kubacki, devel@edk2.groups.io, Kinney, Michael D,
Sean Brogan, Kubacki, Michael
Cc: Barkelew, Bret, Gao, Liming
[-- Attachment #1.1: Type: text/plain, Size: 12425 bytes --]
@Kubacki, Michael<mailto:michael.kubacki@microsoft.com>
Try to see the block diagram "Current behavior" vs "Expected behavior". We find a way to transfer “Current behavior” to “Expected behavior” if we want to support pipeline tab “Code Coverage”. Hope it’s helpful for you to realize the limitation.
[cid:image001.png@01D91536.A4A7ADD0]
Thanks,
Gua
-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 21, 2022 11:11 AM
To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Kubacki, Michael <michael.kubacki@microsoft.com>
Cc: Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming <gaoliming@byosoft.com.cn>
Subject: Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
I might be misunderstanding the problem, but can you merge the XML file from each package into one and then pass that to the "PublishCodeCoverageResult" task?
Perhaps with something like this
https://github.com/danielpalme/ReportGenerator
Thanks,
Michael
On 12/20/2022 8:32 PM, Guo, Gua wrote:
> @Kinney, Michael D <mailto:michael.d.kinney@intel.com>
>
> I assume you want to see pipeline "Code Coverage" like below “3. Azure
> Pipeline Change” diagram.
>
> But it will have a limitation. Current Edk2 CI/CD design is use
> “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask for
> each package. And each package will have itself Coverage Data, but the
> design can not be covered by Azure API “PublishCodeCoverageResult”.
>
> That’s why I don’t integrated it into the change.
>
> 1. Azure API doc:
> https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines
>
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/refere
> nce/publish-code-coverage-results-v1?view=azure-pipelines>
>
> 2. Code Change:
>
> 3. Azure Pipeline Change:
>
> VS2019
>
>
>
> GCC5
>
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204
> 22452
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 422452>
>
>
>
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206
> 15640
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 615640>
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_
> a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
> overage-tab
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&
> _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
> coverage-tab>
>
>
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_
> a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
> overage-tab
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&
> _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
> coverage-tab>
>
> Thanks,
>
> Gua
>
> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
> Sent: Wednesday, December 21, 2022 5:23 AM
> To: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney,
> Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Kubacki, Michael
> <michael.kubacki@microsoft.com<mailto:michael.kubacki@microsoft.com>>
> Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
> <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>; Gao, Liming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>;
> Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
> Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage
> tool
>
> Hi Gua,
>
> I created a PR with this series.
>
> https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>
>
> When I follow link to Azure Pipelines for the GCC builds, I can see
> the coverage.xml file in the ZIP file of artifacts.
>
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v
> iew=results
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&
> view=results>
>
> But I do not see an information from Azure Pielines view on code
> coverage or a way to view the lines covered from the Azure Pipelines
> web pages.
>
> Are there additional items to enable?
>
> Mike
>
> > -----Original Message-----
>
> > From: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com<mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> > Sent: Tuesday, December 13, 2022 7:20 PM
>
> > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <mailto:devel@edk2.groups.io>
>
> > Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com<mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>; Sean
> Brogan
>
> > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com%20%3cmailto:sean.brogan@microsoft.com>>>;
> Barkelew, Bret
>
> > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com%20%3cmailto:bret.barkelew@microsoft.com>>>;
> Kinney, Michael D
>
> > <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com%20%3cmailto:michael.d.kinney@intel.com>>>;
> Gao, Liming <gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>
>
> > Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> >
>
> > From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com<mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> >
>
> > For Windows add below tool for code coverage 1. OpenCppCoverage:
>
> > parsing pdb file to generate coverage data 2. pycobertura: show up
>
> > html format data for coverage data
>
> >
>
> > For Linux add below tool for code coverage 1. lcov: parsing gcda
> gcno
>
> > file to generate coverage data 2. lcov-cobertura: convert coverage
>
> > data to cobertura format 3. pycobertura: show up html format data
> for
>
> > coverage data
>
> >
>
> > Cc: Sean Brogan <sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com>>
>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com
> <mailto:Bret.Barkelew@microsoft.com>>
>
> > Cc: Michael D Kinney <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Cc: Liming Gao <gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>
>
> > Signed-off-by: Gua Guo <gua.guo@intel.com
> <mailto:gua.guo@intel.com>>
>
> > ---
>
> > .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
>
> > .azurepipelines/Windows-VS2019.yml | 5 +++++
>
> > .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
>
> > .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
>
> > pip-requirements.txt | 2 ++
>
> > 5 files changed, 17 insertions(+), 1 deletion(-)
>
> >
>
> > diff --git a/.azurepipelines/Ubuntu-GCC5.yml
>
> > b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf
> 100644
>
> > --- a/.azurepipelines/Ubuntu-GCC5.yml
>
> > +++ b/.azurepipelines/Ubuntu-GCC5.yml
>
> > @@ -19,4 +19,7 @@ jobs:
>
> > tool_chain_tag: 'GCC5'
>
> >
>
> > vm_image: 'ubuntu-latest'
>
> >
>
> > arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> >
>
> > -
>
> >
>
> > + extra_install_step:
>
> >
>
> > + - bash: sudo apt-get install -y lcov
>
> >
>
> > + displayName: Install Code Coverage Tools
>
> >
>
> > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > diff --git a/.azurepipelines/Windows-VS2019.yml
>
> > b/.azurepipelines/Windows-VS2019.yml
>
> > index e4bd4b1d22..c07e5bb434 100644
>
> > --- a/.azurepipelines/Windows-VS2019.yml
>
> > +++ b/.azurepipelines/Windows-VS2019.yml
>
> > @@ -18,3 +18,8 @@ jobs:
>
> > tool_chain_tag: 'VS2019'
>
> >
>
> > vm_image: 'windows-2019'
>
> >
>
> > arch_list: "IA32,X64"
>
> >
>
> > + extra_install_step:
>
> >
>
> > + - powershell: choco install opencppcoverage; Write-Host
> "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> >
>
> > + displayName: Install Code Coverage Tool
>
> >
>
> > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > +
>
> >
>
> > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
>
> > b/.azurepipelines/templates/pr-gate-build-job.yml
>
> > index 7f88b41dc8..0ff1ce2952 100644
>
> > --- a/.azurepipelines/templates/pr-gate-build-job.yml
>
> > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
>
> > @@ -12,6 +12,7 @@ parameters:
>
> > tool_chain_tag: ''
>
> >
>
> > vm_image: ''
>
> >
>
> > arch_list: ''
>
> >
>
> > + extra_install_step: []
>
> >
>
> >
>
> >
>
> > # Build step
>
> >
>
> > jobs:
>
> >
>
> > @@ -77,3 +78,4 @@ jobs:
>
> > build_pkgs: $(Build.Pkgs)
>
> >
>
> > build_targets: $(Build.Targets)
>
> >
>
> > build_archs: ${{ parameters.arch_list }}
>
> >
>
> > + extra_install_step: ${{ parameters.extra_install_step }}
>
> >
>
> > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
>
> > b/.azurepipelines/templates/pr-gate-steps.yml
>
> > index cb431e53fc..080f60aea6 100644
>
> > --- a/.azurepipelines/templates/pr-gate-steps.yml
>
> > +++ b/.azurepipelines/templates/pr-gate-steps.yml
>
> > @@ -12,6 +12,7 @@ parameters:
>
> > build_pkgs: ''
>
> >
>
> > build_targets: ''
>
> >
>
> > build_archs: ''
>
> >
>
> > + extra_install_step: []
>
> >
>
> >
>
> >
>
> > steps:
>
> >
>
> > - checkout: self
>
> >
>
> > @@ -37,6 +38,8 @@ steps:
>
> > displayName: fetch target branch
>
> >
>
> > condition: eq(variables['Build.Reason'], 'PullRequest')
>
> >
>
> >
>
> >
>
> > +- ${{ parameters.extra_install_step }}
>
> >
>
> > +
>
> >
>
> > # trim the package list if this is a PR
>
> >
>
> > - task: CmdLine@1
>
> >
>
> > displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> >
>
> > @@ -125,6 +128,7 @@ steps:
>
> > TestSuites.xml
>
> >
>
> > **/BUILD_TOOLS_REPORT.html
>
> >
>
> > **/OVERRIDELOG.TXT
>
> >
>
> > + coverage.xml
>
> >
>
> > flattenFolders: true
>
> >
>
> > condition: succeededOrFailed()
>
> >
>
> >
>
> >
>
> > diff --git a/pip-requirements.txt b/pip-requirements.txt index
>
> > be8c7a1c37..4ffcadddd8 100644
>
> > --- a/pip-requirements.txt
>
> > +++ b/pip-requirements.txt
>
> > @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
>
> > edk2-pytool-extensions~=0.20.0
>
> >
>
> > edk2-basetools==0.1.39
>
> >
>
> > antlr4-python3-runtime==4.7.1
>
> >
>
> > +lcov-cobertura==2.0.2
>
> >
>
> > +
>
> >
>
> > --
>
> > 2.31.1.windows.1
>
>
[-- Attachment #1.2: Type: text/html, Size: 40574 bytes --]
[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 66947 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-21 4:20 ` Guo, Gua
@ 2022-12-21 4:29 ` Michael Kubacki
2022-12-21 6:19 ` Guo, Gua
0 siblings, 1 reply; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 4:29 UTC (permalink / raw)
To: devel, gua.guo, Kinney, Michael D, Sean Brogan, Kubacki, Michael
Cc: Barkelew, Bret, Gao, Liming
I understand the problem. Have you looked into tools like the one I
linked? It also has a DevOps extension -
https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator.
I have not looked at it closely, but at a high-level, it appears to at
least attempt to address this problem.
Thanks,
Michael
On 12/20/2022 11:20 PM, Guo, Gua wrote:
> @Kubacki, Michael <mailto:michael.kubacki@microsoft.com>
>
> Try to see the block diagram "Current behavior" vs "Expected behavior".
> We find a way to transfer “Current behavior” to “Expected behavior” if
> we want to support pipeline tab “Code Coverage”. Hope it’s helpful for
> you to realize the limitation.
>
> Thanks,
>
> Gua
>
> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com>
> Sent: Wednesday, December 21, 2022 11:11 AM
> To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, Michael
> D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>;
> Kubacki, Michael <michael.kubacki@microsoft.com>
> Cc: Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming
> <gaoliming@byosoft.com.cn>
> Subject: Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code
> coverage tool
>
> I might be misunderstanding the problem, but can you merge the XML file
> from each package into one and then pass that to the
> "PublishCodeCoverageResult" task?
>
> Perhaps with something like this
>
> https://github.com/danielpalme/ReportGenerator
> <https://github.com/danielpalme/ReportGenerator>
>
> Thanks,
>
> Michael
>
> On 12/20/2022 8:32 PM, Guo, Gua wrote:
>
> > @Kinney, Michael D <mailto:michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> >
>
> > I assume you want to see pipeline "Code Coverage" like below “3. Azure
>
> > Pipeline Change” diagram.
>
> >
>
> > But it will have a limitation. Current Edk2 CI/CD design is use
>
> > “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask for
>
> > each package. And each package will have itself Coverage Data, but the
>
> > design can not be covered by Azure API “PublishCodeCoverageResult”.
>
> >
>
> > That’s why I don’t integrated it into the change.
>
> >
>
> > 1. Azure API doc:
>
> >
> https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines>
>
> >
>
> > <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/refere
>
> > nce/publish-code-coverage-results-v1?view=azure-pipelines>
>
> >
>
> > 2. Code Change:
>
> >
>
> > 3. Azure Pipeline Change:
>
> >
>
> > VS2019
>
> >
>
> >
>
> >
>
> > GCC5
>
> >
>
> >
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204>
>
> > 22452
>
> > <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
>
> > 422452>
>
> >
>
> >
>
> >
>
> >
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206>
>
> > 15640
>
> > <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
>
> > 615640>
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_>
>
> > a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
>
> > overage-tab
>
> > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&
>
> > _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
>
> > coverage-tab>
>
> >
>
> >
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_>
>
> > a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
>
> > overage-tab
>
> > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&
>
> > _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
>
> > coverage-tab>
>
> >
>
> > Thanks,
>
> >
>
> > Gua
>
> >
>
> > -----Original Message-----
>
> > From: Kinney, Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Sent: Wednesday, December 21, 2022 5:23 AM
>
> > To: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com>>;
> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney,
>
> > Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>; Sean Brogan
>
> > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com>>;
> Kubacki, Michael
>
> > <michael.kubacki@microsoft.com <mailto:michael.kubacki@microsoft.com>>
>
> > Cc: Sean Brogan <sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
>
> > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com>>;
> Gao, Liming <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>;
>
> > Kinney, Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage
>
> > tool
>
> >
>
> > Hi Gua,
>
> >
>
> > I created a PR with this series.
>
> >
>
> > https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>
>
> > <https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>>
>
> >
>
> > When I follow link to Azure Pipelines for the GCC builds, I can see
>
> > the coverage.xml file in the ZIP file of artifacts.
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v>
>
> > iew=results
>
> > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&
>
> > view=results>
>
> >
>
> > But I do not see an information from Azure Pielines view on code
>
> > coverage or a way to view the lines covered from the Azure Pipelines
>
> > web pages.
>
> >
>
> > Are there additional items to enable?
>
> >
>
> > Mike
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> >
>
> > > Sent: Tuesday, December 13, 2022 7:20 PM
>
> >
>
> > > To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
>
> >
>
> > > Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>; Sean
>
> > Brogan
>
> >
>
> > > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com%20%3cmailto:sean.brogan@microsoft.com>>>;
>
> > Barkelew, Bret
>
> >
>
> > > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com
> <mailto:bret.barkelew@microsoft.com%20%3cmailto:bret.barkelew@microsoft.com>>>;
>
> > Kinney, Michael D
>
> >
>
> > > <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com%20%3cmailto:michael.d.kinney@intel.com>>>;
>
> > Gao, Liming <gaoliming@byosoft.com.cn
>
> > <mailto:gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>>
>
> >
>
> > > Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
>
> >
>
> > >
>
> >
>
> > > From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> >
>
> > >
>
> >
>
> > > For Windows add below tool for code coverage 1. OpenCppCoverage:
>
> >
>
> > > parsing pdb file to generate coverage data 2. pycobertura: show up
>
> >
>
> > > html format data for coverage data
>
> >
>
> > >
>
> >
>
> > > For Linux add below tool for code coverage 1. lcov: parsing gcda
>
> > gcno
>
> >
>
> > > file to generate coverage data 2. lcov-cobertura: convert coverage
>
> >
>
> > > data to cobertura format 3. pycobertura: show up html format data
>
> > for
>
> >
>
> > > coverage data
>
> >
>
> > >
>
> >
>
> > > Cc: Sean Brogan <sean.brogan@microsoft.com
>
> > <mailto:sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com>>>
>
> >
>
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com
>
> > <mailto:Bret.Barkelew@microsoft.com
> <mailto:Bret.Barkelew@microsoft.com>>>
>
> >
>
> > > Cc: Michael D Kinney <michael.d.kinney@intel.com
>
> > <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>>
>
> >
>
> > > Cc: Liming Gao <gaoliming@byosoft.com.cn
>
> > <mailto:gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>>
>
> >
>
> > > Signed-off-by: Gua Guo <gua.guo@intel.com
>
> > <mailto:gua.guo@intel.com <mailto:gua.guo@intel.com>>>
>
> >
>
> > > ---
>
> >
>
> > > .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
>
> >
>
> > > .azurepipelines/Windows-VS2019.yml | 5 +++++
>
> >
>
> > > .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
>
> >
>
> > > .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
>
> >
>
> > > pip-requirements.txt | 2 ++
>
> >
>
> > > 5 files changed, 17 insertions(+), 1 deletion(-)
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf
>
> > 100644
>
> >
>
> > > --- a/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > +++ b/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > @@ -19,4 +19,7 @@ jobs:
>
> >
>
> > > tool_chain_tag: 'GCC5'
>
> >
>
> > >
>
> >
>
> > > vm_image: 'ubuntu-latest'
>
> >
>
> > >
>
> >
>
> > > arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> >
>
> > >
>
> >
>
> > > -
>
> >
>
> > >
>
> >
>
> > > + extra_install_step:
>
> >
>
> > >
>
> >
>
> > > + - bash: sudo apt-get install -y lcov
>
> >
>
> > >
>
> >
>
> > > + displayName: Install Code Coverage Tools
>
> >
>
> > >
>
> >
>
> > > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > b/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > index e4bd4b1d22..c07e5bb434 100644
>
> >
>
> > > --- a/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > +++ b/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > @@ -18,3 +18,8 @@ jobs:
>
> >
>
> > > tool_chain_tag: 'VS2019'
>
> >
>
> > >
>
> >
>
> > > vm_image: 'windows-2019'
>
> >
>
> > >
>
> >
>
> > > arch_list: "IA32,X64"
>
> >
>
> > >
>
> >
>
> > > + extra_install_step:
>
> >
>
> > >
>
> >
>
> > > + - powershell: choco install opencppcoverage; Write-Host
>
> > "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> >
>
> > >
>
> >
>
> > > + displayName: Install Code Coverage Tool
>
> >
>
> > >
>
> >
>
> > > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > b/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > index 7f88b41dc8..0ff1ce2952 100644
>
> >
>
> > > --- a/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > @@ -12,6 +12,7 @@ parameters:
>
> >
>
> > > tool_chain_tag: ''
>
> >
>
> > >
>
> >
>
> > > vm_image: ''
>
> >
>
> > >
>
> >
>
> > > arch_list: ''
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: []
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > # Build step
>
> >
>
> > >
>
> >
>
> > > jobs:
>
> >
>
> > >
>
> >
>
> > > @@ -77,3 +78,4 @@ jobs:
>
> >
>
> > > build_pkgs: $(Build.Pkgs)
>
> >
>
> > >
>
> >
>
> > > build_targets: $(Build.Targets)
>
> >
>
> > >
>
> >
>
> > > build_archs: ${{ parameters.arch_list }}
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: ${{ parameters.extra_install_step }}
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > b/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > index cb431e53fc..080f60aea6 100644
>
> >
>
> > > --- a/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > +++ b/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > @@ -12,6 +12,7 @@ parameters:
>
> >
>
> > > build_pkgs: ''
>
> >
>
> > >
>
> >
>
> > > build_targets: ''
>
> >
>
> > >
>
> >
>
> > > build_archs: ''
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: []
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > steps:
>
> >
>
> > >
>
> >
>
> > > - checkout: self
>
> >
>
> > >
>
> >
>
> > > @@ -37,6 +38,8 @@ steps:
>
> >
>
> > > displayName: fetch target branch
>
> >
>
> > >
>
> >
>
> > > condition: eq(variables['Build.Reason'], 'PullRequest')
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > +- ${{ parameters.extra_install_step }}
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > # trim the package list if this is a PR
>
> >
>
> > >
>
> >
>
> > > - task: CmdLine@1
>
> >
>
> > >
>
> >
>
> > > displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> >
>
> > >
>
> >
>
> > > @@ -125,6 +128,7 @@ steps:
>
> >
>
> > > TestSuites.xml
>
> >
>
> > >
>
> >
>
> > > **/BUILD_TOOLS_REPORT.html
>
> >
>
> > >
>
> >
>
> > > **/OVERRIDELOG.TXT
>
> >
>
> > >
>
> >
>
> > > + coverage.xml
>
> >
>
> > >
>
> >
>
> > > flattenFolders: true
>
> >
>
> > >
>
> >
>
> > > condition: succeededOrFailed()
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > diff --git a/pip-requirements.txt b/pip-requirements.txt index
>
> >
>
> > > be8c7a1c37..4ffcadddd8 100644
>
> >
>
> > > --- a/pip-requirements.txt
>
> >
>
> > > +++ b/pip-requirements.txt
>
> >
>
> > > @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
>
> >
>
> > > edk2-pytool-extensions~=0.20.0
>
> >
>
> > >
>
> >
>
> > > edk2-basetools==0.1.39
>
> >
>
> > >
>
> >
>
> > > antlr4-python3-runtime==4.7.1
>
> >
>
> > >
>
> >
>
> > > +lcov-cobertura==2.0.2
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > --
>
> >
>
> > > 2.31.1.windows.1
>
> >
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
2022-12-21 4:29 ` Michael Kubacki
@ 2022-12-21 6:19 ` Guo, Gua
0 siblings, 0 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 6:19 UTC (permalink / raw)
To: Michael Kubacki, devel@edk2.groups.io, Kinney, Michael D,
Sean Brogan, Kubacki, Michael
Cc: Barkelew, Bret, Gao, Liming
[-- Attachment #1.1: Type: text/plain, Size: 19256 bytes --]
I think I may need to have the access right to install pipeline by the step. I guess I need to have enough access right to control https://dev.azure.com/tianocore/edk2-ci to install DevOps Server Extensions.
[cid:image001.png@01D91547.2CAA9B60]
Thanks,
Gua
-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 21, 2022 12:30 PM
To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Kubacki, Michael <michael.kubacki@microsoft.com>
Cc: Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming <gaoliming@byosoft.com.cn>
Subject: Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool
I understand the problem. Have you looked into tools like the one I linked? It also has a DevOps extension - https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator.
I have not looked at it closely, but at a high-level, it appears to at least attempt to address this problem.
Thanks,
Michael
On 12/20/2022 11:20 PM, Guo, Gua wrote:
> @Kubacki, Michael <mailto:michael.kubacki@microsoft.com>
>
> Try to see the block diagram "Current behavior" vs "Expected behavior".
> We find a way to transfer “Current behavior” to “Expected behavior” if
> we want to support pipeline tab “Code Coverage”. Hope it’s helpful for
> you to realize the limitation.
>
> Thanks,
>
> Gua
>
> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>>
> Sent: Wednesday, December 21, 2022 11:11 AM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; Kinney,
> Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Kubacki, Michael
> <michael.kubacki@microsoft.com<mailto:michael.kubacki@microsoft.com>>
> Cc: Barkelew, Bret <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>; Gao, Liming
> <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
> Subject: Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code
> coverage tool
>
> I might be misunderstanding the problem, but can you merge the XML
> file from each package into one and then pass that to the
> "PublishCodeCoverageResult" task?
>
> Perhaps with something like this
>
> https://github.com/danielpalme/ReportGenerator
> <https://github.com/danielpalme/ReportGenerator>
>
> Thanks,
>
> Michael
>
> On 12/20/2022 8:32 PM, Guo, Gua wrote:
>
> > @Kinney, Michael D <mailto:michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> >
>
> > I assume you want to see pipeline "Code Coverage" like below “3.
> Azure
>
> > Pipeline Change” diagram.
>
> >
>
> > But it will have a limitation. Current Edk2 CI/CD design is use
>
> > “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask
> for
>
> > each package. And each package will have itself Coverage Data, but
> the
>
> > design can not be covered by Azure API “PublishCodeCoverageResult”.
>
> >
>
> > That’s why I don’t integrated it into the change.
>
> >
>
> > 1. Azure API doc:
>
> >
> https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/referen
> ce/publish-code-coverage-results-v1?view=azure-pipelines
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/refere
> nce/publish-code-coverage-results-v1?view=azure-pipelines>
>
> >
>
> >
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/refere
>
> > nce/publish-code-coverage-results-v1?view=azure-pipelines>
>
> >
>
> > 2. Code Change:
>
> >
>
> > 3. Azure Pipeline Change:
>
> >
>
> > VS2019
>
> >
>
> >
>
> >
>
> > GCC5
>
> >
>
> >
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 4>
>
> > 22452
>
> >
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
>
> > 422452>
>
> >
>
> >
>
> >
>
> >
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 6>
>
> > 15640
>
> >
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
>
> > 615640>
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&
> _>
>
> >
> a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
>
> > overage-tab
>
> >
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&
>
> >
> _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
>
> > coverage-tab>
>
> >
>
> >
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&
> _>
>
> >
> a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
>
> > overage-tab
>
> >
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&
>
> >
> _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
>
> > coverage-tab>
>
> >
>
> > Thanks,
>
> >
>
> > Gua
>
> >
>
> > -----Original Message-----
>
> > From: Kinney, Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Sent: Wednesday, December 21, 2022 5:23 AM
>
> > To: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com<mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>;
> devel@edk2.groups.io<mailto:devel@edk2.groups.io> <mailto:devel@edk2.groups.io>; Kinney,
>
> > Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>; Sean Brogan
>
> > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com%20%3cmailto:sean.brogan@microsoft.com>>>;
> Kubacki, Michael
>
> > <michael.kubacki@microsoft.com
> <mailto:michael.kubacki@microsoft.com>>
>
> > Cc: Sean Brogan <sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
>
> > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com%20%3cmailto:bret.barkelew@microsoft.com>>>;
> Gao, Liming <gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>;
>
> > Kinney, Michael D <michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>
>
> > Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage
>
> > tool
>
> >
>
> > Hi Gua,
>
> >
>
> > I created a PR with this series.
>
> >
>
> > https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>
>
> > <https://github.com/tianocore/edk2/pull/3795
> <https://github.com/tianocore/edk2/pull/3795>>
>
> >
>
> > When I follow link to Azure Pipelines for the GCC builds, I can see
>
> > the coverage.xml file in the ZIP file of artifacts.
>
> >
>
> >
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&
> v>
>
> > iew=results
>
> >
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&
>
> > view=results>
>
> >
>
> > But I do not see an information from Azure Pielines view on code
>
> > coverage or a way to view the lines covered from the Azure
> Pipelines
>
> > web pages.
>
> >
>
> > Are there additional items to enable?
>
> >
>
> > Mike
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> >
>
> > > Sent: Tuesday, December 13, 2022 7:20 PM
>
> >
>
> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <mailto:devel@edk2.groups.io>
> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io<mailto:devel@edk2.groups.io%20%3cmailto:devel@edk2.groups.io>>>
>
> >
>
> > > Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>; Sean
>
> > Brogan
>
> >
>
> > > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com%20%3cmailto:sean.brogan@microsoft.co
> m>>>;
>
> > Barkelew, Bret
>
> >
>
> > > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com
> <mailto:bret.barkelew@microsoft.com%20%3cmailto:bret.barkelew@microsof
> t.com>>>;
>
> > Kinney, Michael D
>
> >
>
> > > <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com%20%3cmailto:michael.d.kinney@intel.
> com>>>;
>
> > Gao, Liming <gaoliming@byosoft.com.cn
>
> > <mailto:gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>>
>
> >
>
> > > Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage
> tool
>
> >
>
> > >
>
> >
>
> > > From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
>
> >
>
> > >
>
> >
>
> > > For Windows add below tool for code coverage 1. OpenCppCoverage:
>
> >
>
> > > parsing pdb file to generate coverage data 2. pycobertura: show
> up
>
> >
>
> > > html format data for coverage data
>
> >
>
> > >
>
> >
>
> > > For Linux add below tool for code coverage 1. lcov: parsing gcda
>
> > gcno
>
> >
>
> > > file to generate coverage data 2. lcov-cobertura: convert
> coverage
>
> >
>
> > > data to cobertura format 3. pycobertura: show up html format
> data
>
> > for
>
> >
>
> > > coverage data
>
> >
>
> > >
>
> >
>
> > > Cc: Sean Brogan <sean.brogan@microsoft.com
>
> > <mailto:sean.brogan@microsoft.com
> <mailto:sean.brogan@microsoft.com>>>
>
> >
>
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com
>
> > <mailto:Bret.Barkelew@microsoft.com
> <mailto:Bret.Barkelew@microsoft.com>>>
>
> >
>
> > > Cc: Michael D Kinney <michael.d.kinney@intel.com
>
> > <mailto:michael.d.kinney@intel.com
> <mailto:michael.d.kinney@intel.com>>>
>
> >
>
> > > Cc: Liming Gao <gaoliming@byosoft.com.cn
>
> > <mailto:gaoliming@byosoft.com.cn
> <mailto:gaoliming@byosoft.com.cn>>>
>
> >
>
> > > Signed-off-by: Gua Guo <gua.guo@intel.com
>
> > <mailto:gua.guo@intel.com <mailto:gua.guo@intel.com<mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>>
>
> >
>
> > > ---
>
> >
>
> > > .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
>
> >
>
> > > .azurepipelines/Windows-VS2019.yml | 5 +++++
>
> >
>
> > > .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
>
> >
>
> > > .azurepipelines/templates/pr-gate-steps.yml | 4 ++++
>
> >
>
> > > pip-requirements.txt | 2 ++
>
> >
>
> > > 5 files changed, 17 insertions(+), 1 deletion(-)
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf
>
> > 100644
>
> >
>
> > > --- a/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > +++ b/.azurepipelines/Ubuntu-GCC5.yml
>
> >
>
> > > @@ -19,4 +19,7 @@ jobs:
>
> >
>
> > > tool_chain_tag: 'GCC5'
>
> >
>
> > >
>
> >
>
> > > vm_image: 'ubuntu-latest'
>
> >
>
> > >
>
> >
>
> > > arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
>
> >
>
> > >
>
> >
>
> > > -
>
> >
>
> > >
>
> >
>
> > > + extra_install_step:
>
> >
>
> > >
>
> >
>
> > > + - bash: sudo apt-get install -y lcov
>
> >
>
> > >
>
> >
>
> > > + displayName: Install Code Coverage Tools
>
> >
>
> > >
>
> >
>
> > > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > b/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > index e4bd4b1d22..c07e5bb434 100644
>
> >
>
> > > --- a/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > +++ b/.azurepipelines/Windows-VS2019.yml
>
> >
>
> > > @@ -18,3 +18,8 @@ jobs:
>
> >
>
> > > tool_chain_tag: 'VS2019'
>
> >
>
> > >
>
> >
>
> > > vm_image: 'windows-2019'
>
> >
>
> > >
>
> >
>
> > > arch_list: "IA32,X64"
>
> >
>
> > >
>
> >
>
> > > + extra_install_step:
>
> >
>
> > >
>
> >
>
> > > + - powershell: choco install opencppcoverage; Write-Host
>
> > "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> >
>
> > >
>
> >
>
> > > + displayName: Install Code Coverage Tool
>
> >
>
> > >
>
> >
>
> > > + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > b/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > index 7f88b41dc8..0ff1ce2952 100644
>
> >
>
> > > --- a/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
>
> >
>
> > > @@ -12,6 +12,7 @@ parameters:
>
> >
>
> > > tool_chain_tag: ''
>
> >
>
> > >
>
> >
>
> > > vm_image: ''
>
> >
>
> > >
>
> >
>
> > > arch_list: ''
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: []
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > # Build step
>
> >
>
> > >
>
> >
>
> > > jobs:
>
> >
>
> > >
>
> >
>
> > > @@ -77,3 +78,4 @@ jobs:
>
> >
>
> > > build_pkgs: $(Build.Pkgs)
>
> >
>
> > >
>
> >
>
> > > build_targets: $(Build.Targets)
>
> >
>
> > >
>
> >
>
> > > build_archs: ${{ parameters.arch_list }}
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: ${{ parameters.extra_install_step }}
>
> >
>
> > >
>
> >
>
> > > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > b/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > index cb431e53fc..080f60aea6 100644
>
> >
>
> > > --- a/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > +++ b/.azurepipelines/templates/pr-gate-steps.yml
>
> >
>
> > > @@ -12,6 +12,7 @@ parameters:
>
> >
>
> > > build_pkgs: ''
>
> >
>
> > >
>
> >
>
> > > build_targets: ''
>
> >
>
> > >
>
> >
>
> > > build_archs: ''
>
> >
>
> > >
>
> >
>
> > > + extra_install_step: []
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > steps:
>
> >
>
> > >
>
> >
>
> > > - checkout: self
>
> >
>
> > >
>
> >
>
> > > @@ -37,6 +38,8 @@ steps:
>
> >
>
> > > displayName: fetch target branch
>
> >
>
> > >
>
> >
>
> > > condition: eq(variables['Build.Reason'], 'PullRequest')
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > +- ${{ parameters.extra_install_step }}
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > # trim the package list if this is a PR
>
> >
>
> > >
>
> >
>
> > > - task: CmdLine@1
>
> >
>
> > >
>
> >
>
> > > displayName: Check if ${{ parameters.build_pkgs }} need
> testing
>
> >
>
> > >
>
> >
>
> > > @@ -125,6 +128,7 @@ steps:
>
> >
>
> > > TestSuites.xml
>
> >
>
> > >
>
> >
>
> > > **/BUILD_TOOLS_REPORT.html
>
> >
>
> > >
>
> >
>
> > > **/OVERRIDELOG.TXT
>
> >
>
> > >
>
> >
>
> > > + coverage.xml
>
> >
>
> > >
>
> >
>
> > > flattenFolders: true
>
> >
>
> > >
>
> >
>
> > > condition: succeededOrFailed()
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > diff --git a/pip-requirements.txt b/pip-requirements.txt index
>
> >
>
> > > be8c7a1c37..4ffcadddd8 100644
>
> >
>
> > > --- a/pip-requirements.txt
>
> >
>
> > > +++ b/pip-requirements.txt
>
> >
>
> > > @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
>
> >
>
> > > edk2-pytool-extensions~=0.20.0
>
> >
>
> > >
>
> >
>
> > > edk2-basetools==0.1.39
>
> >
>
> > >
>
> >
>
> > > antlr4-python3-runtime==4.7.1
>
> >
>
> > >
>
> >
>
> > > +lcov-cobertura==2.0.2
>
> >
>
> > >
>
> >
>
> > > +
>
> >
>
> > >
>
> >
>
> > > --
>
> >
>
> > > 2.31.1.windows.1
>
> >
>
>
[-- Attachment #1.2: Type: text/html, Size: 75437 bytes --]
[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 45068 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-12-21 6:19 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-12-14 3:19 [PATCH v6 0/3] Add code coverage support for GCC/MSVC Guo, Gua
2022-12-14 3:19 ` [PATCH v6 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC Guo, Gua
2022-12-14 3:19 ` [PATCH v6 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
2022-12-14 3:19 ` [PATCH v6 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2022-12-20 21:23 ` Michael D Kinney
2022-12-21 1:32 ` Guo, Gua
2022-12-21 3:11 ` [edk2-devel] " Michael Kubacki
2022-12-21 4:20 ` Guo, Gua
2022-12-21 4:29 ` Michael Kubacki
2022-12-21 6:19 ` Guo, Gua
2022-12-21 3:17 ` Michael D Kinney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox