* [PATCH 0/3] Add code coverage support for GCC
@ 2022-09-30 1:53 Guo, Gua
2022-09-30 1:53 ` [PATCH 1/3] UnitTestFrameworkPkg: " Guo, Gua
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Guo, Gua @ 2022-09-30 1:53 UTC (permalink / raw)
To: devel; +Cc: Gua Guo
From: Gua Guo <gua.guo@intel.com>
V1: Add coverage option for GCC
V2: Add ReadMe.md for how to generate coverage report
V3: Add VS2019 and GCC code coverage support
V4: Add VS2019 and GCC Azure CI/CD support
V5: Fix some typo and some flow issue
Gua Guo (3):
UnitTestFrameworkPkg: Add code coverage support for GCC
BaseTools/Plugin: Add coverage support for Unit Test
.azurepipelines: Install code coverage tool
.azurepipelines/Ubuntu-GCC5.yml | 5 +-
.azurepipelines/Windows-VS2019.yml | 5 +
.../templates/pr-gate-build-job.yml | 2 +
.azurepipelines/templates/pr-gate-steps.yml | 5 +
.../HostBasedUnitTestRunner.py | 119 ++++++++++++++++++
UnitTestFrameworkPkg/ReadMe.md | 37 +++++-
.../UnitTestFrameworkPkg.ci.yaml | 2 +
.../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
pip-requirements.txt | 3 +
9 files changed, 176 insertions(+), 5 deletions(-)
--
2.31.1.windows.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-09-30 1:53 [PATCH 0/3] Add code coverage support for GCC Guo, Gua
@ 2022-09-30 1:53 ` Guo, Gua
2022-12-21 3:41 ` Michael Kubacki
2022-09-30 1:53 ` [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
2022-09-30 1:53 ` [PATCH 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-09-30 1:53 UTC (permalink / raw)
To: devel
Cc: Gua Guo, Michael D Kinney, Sean Brogan, Bret Barkelew,
Michael Kubacki
From: Gua Guo <gua.guo@intel.com>
In order to collect code coverage after running executable
file, generate *.gcda and *.gcno file that require by lcov tool
to generate code coverage report.
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael Kubacki <mikuback@linux.microsoft.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
UnitTestFrameworkPkg/ReadMe.md | 37 +++++++++++++++++--
.../UnitTestFrameworkPkg.ci.yaml | 2 +
.../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
3 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md
index e696412cb3..c00e2411f8 100644
--- a/UnitTestFrameworkPkg/ReadMe.md
+++ b/UnitTestFrameworkPkg/ReadMe.md
@@ -60,7 +60,7 @@ you should be good to go.
See this example in 'SampleUnitTestUefiShell.inf'...
-```
+```inf
[Packages]
MdePkg/MdePkg.dec
@@ -75,7 +75,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
Also, if you want you test to automatically be picked up by the Test Runner plugin, you will need
to make sure that the module `BASE_NAME` contains the word `Test`...
-```
+```inf
[Defines]
BASE_NAME = SampleUnitTestUefiShell
```
@@ -361,7 +361,7 @@ RUNNING TEST SUITE: Int Safe Conversions Test Suite
```
You can also, if you are so inclined, read the output from the exact instance of the test that was run during
-`stuart_ci_build`. The ouput file can be found on a path that looks like:
+`stuart_ci_build`. The output file can be found on a path that looks like:
`Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.result.xml`
@@ -399,6 +399,37 @@ CMOCKA_XML_FILE=<absolute or relative path to output file>
This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
+### Code Coverage
+
+Host based Unit Tests will automatically enable coverage data.
+
+For Windows, This is primarily leverage for pipeline builds, but this can be leveraged locally using the
+OpenCppCoverage windows tool, and pycobertura is used to covert this coverage data to a human readable HTML
+file. These tools must be installed to parse code coverage.
+
+- Windows Prerequisite
+ ```bash
+ Download and install https://github.com/OpenCppCoverage/OpenCppCoverage/tags
+ python -m pip install --upgrade -r ./pip-requirements.txt
+ stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
+ Open Build/coverage.html
+ ```
+
+
+For Linux, This is primarily leveraged for pipeline builds, but this can be leveraged locally using the
+lcov linux tool, and parsed using the lcov_cobertura python tool. pycobertura is used to
+covert this coverage data to a human readable HTML file. These tools must be installed
+to parse code coverage.
+
+- Linux Prerequisite
+ ```bash
+ sudo apt-get install -y lcov
+ python -m pip install --upgrade -r ./pip-requirements.txt
+ stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
+ Open Build/coverage.html
+ ```
+
+
### Important Note
This works on both Windows and Linux, but is currently limited to x64 architectures. Working on getting others, but we
diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
index 77d51e1348..f68882e1cf 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
@@ -90,6 +90,8 @@
"pytools",
"NOFAILURE",
"cmockery",
+ "cobertura",
+ "pycobertura",
"DHAVE", # build flag for cmocka in the INF
"corthon", # Contact GitHub account in Readme
"mdkinney", # Contact GitHub account in Readme
diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
index 4dd8d4ac67..ca46b8513a 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
@@ -25,7 +25,8 @@
GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
!endif
-
+ GCC:*_GCC5_*_CC_FLAGS = --coverage
+ GCC:*_GCC5_*_DLINK_FLAGS = --coverage
[BuildOptions.common.EDKII.HOST_APPLICATION]
#
# MSFT
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test
2022-09-30 1:53 [PATCH 0/3] Add code coverage support for GCC Guo, Gua
2022-09-30 1:53 ` [PATCH 1/3] UnitTestFrameworkPkg: " Guo, Gua
@ 2022-09-30 1:53 ` Guo, Gua
2022-12-21 3:42 ` [edk2-devel] " Michael Kubacki
2022-09-30 1:53 ` [PATCH 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-09-30 1:53 UTC (permalink / raw)
To: devel
Cc: Gua Guo, Bob Feng, Bret Barkelew, Liming Gao, Michael D Kinney,
Sean Brogan
From: Gua Guo <gua.guo@intel.com>
For GCC, use lcov to generate Unit Test code coverage
report
For VS2019, use OpenCppCoverage to generate code
coverage report
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Liming Gao <gaoliming@bysoft.com.cn>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
.../HostBasedUnitTestRunner.py | 119 ++++++++++++++++++
1 file changed, 119 insertions(+)
diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index c1eeaf2625..d92de236dc 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -112,4 +112,123 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
" %s - %s" % (case.attrib['name'], result.text))
failure_count += 1
+ if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "GCC5":
+ self.gen_code_coverage_gcc(thebuilder)
+ elif thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "VS2019":
+ self.gen_code_coverage_msvc(thebuilder)
+ else:
+ logging.info("Skipping code coverage. Only supported on GCC.")
+
return failure_count
+
+ def gen_code_coverage_gcc(self, thebuilder):
+ logging.info("Generating UnitTest code coverage")
+
+ buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
+ workspace = thebuilder.env.GetValue("WORKSPACE")
+
+ # Generate base code coverage for all source files
+ ret = RunCmd("lcov", f"--no-external --capture --initial --directory {buildOutputBase} --output-file {buildOutputBase}/cov-base.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to build initial coverage data.")
+ return 1
+
+ # Coverage data for tested files only
+ ret = RunCmd("lcov", f"--capture --directory {buildOutputBase}/ --output-file {buildOutputBase}/coverage-test.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to build coverage data for tested files.")
+ return 1
+
+ # Aggregate all coverage data
+ ret = RunCmd("lcov", f"--add-tracefile {buildOutputBase}/cov-base.info --add-tracefile {buildOutputBase}/coverage-test.info --output-file {buildOutputBase}/total-coverage.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to aggregate coverage data.")
+ return 1
+
+ # Generate coverage XML
+ ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info -o {buildOutputBase}/compare.xml")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate coverage XML.")
+ return 1
+
+ # Filter out auto-generated and test code
+ ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {buildOutputBase}/coverage.xml")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed generate filtered coverage XML.")
+ return 1
+
+ # Generate all coverage file
+ testCoverageList = glob.glob (f"{workspace}/Build/**/total-coverage.info", recursive=True)
+
+ coverageFile = ""
+ for testCoverage in testCoverageList:
+ coverageFile += " --add-tracefile " + testCoverage
+ ret = RunCmd("lcov", f"{coverageFile} --output-file {workspace}/Build/all-coverage.info --rc lcov_branch_coverage=1")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed generate all coverage file.")
+ return 1
+
+ # Generate and HTML file if requested.by each package
+ ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/coverage.html {buildOutputBase}/coverage.xml --source {workspace}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate HTML in single package..")
+
+ # Generate and HTML file if requested.for all package
+ if os.path.isfile(f"{workspace}/Build/coverage.xml"):
+ os.remove(f"{workspace}/Build/coverage.xml")
+ ret = RunCmd("lcov_cobertura",f"{workspace}/Build/all-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {workspace}/Build/coverage.xml")
+
+ if os.path.isfile(f"{workspace}/Build/coverage.html"):
+ os.remove(f"{workspace}/Build/coverage.html")
+ ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate HTML.")
+
+ return 0
+
+
+ def gen_code_coverage_msvc(self, thebuilder):
+ logging.info("Generating UnitTest code coverage")
+
+
+ buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
+ testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
+ workspace = thebuilder.env.GetValue("WORKSPACE")
+
+ # Generate coverage file
+ coverageFile = ""
+ for testFile in testList:
+ ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
+ coverageFile += " --input_coverage=" + testFile + ".cov"
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to collect coverage data.")
+ return 1
+
+ # Generate and HTML file if requested.by each package
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}/Build {coverageFile}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
+ return 1
+
+ ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/cverage.html {buildOutputBase}/coverage.xml --source {workspace}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate HTML in single package.")
+ return 1
+
+ # Generate total report HTML file for all package
+ testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
+ coverageFile = ""
+ for testCoverage in testCoverageList:
+ coverageFile += " --input_coverage=" + testCoverage
+
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}/Build/coverage.xml --working_dir={workspace}/Build {coverageFile}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
+ return 1
+
+ ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
+ if(ret != 0):
+ logging.error("UnitTest Coverage: Failed to generate HTML.")
+ return 1
+
+ return 0
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 3/3] .azurepipelines: Install code coverage tool
2022-09-30 1:53 [PATCH 0/3] Add code coverage support for GCC Guo, Gua
2022-09-30 1:53 ` [PATCH 1/3] UnitTestFrameworkPkg: " Guo, Gua
2022-09-30 1:53 ` [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
@ 2022-09-30 1:53 ` Guo, Gua
2022-12-21 3:42 ` [edk2-devel] " Michael Kubacki
2 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-09-30 1:53 UTC (permalink / raw)
To: devel; +Cc: Gua Guo, Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
From: Gua Guo <gua.guo@intel.com>
For Windows add below tool for code coverage
1. OpenCppCoverage: parsing pdb file to generate coverage
data
2. pycobertura: show up html format data for coverage data
For Linux add below tool for code coverage
1. lcov: parsing gcda gcno file to generate coverage data
2. lcov-cobertura: convert coverage data to cobertura format
3. pycobertura: show up html format data for coverage data
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
.azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
.azurepipelines/Windows-VS2019.yml | 5 +++++
.azurepipelines/templates/pr-gate-build-job.yml | 2 ++
.azurepipelines/templates/pr-gate-steps.yml | 5 +++++
.../HostBasedUnitTestRunner.py | 12 ++++++------
pip-requirements.txt | 3 +++
6 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
index 3760c6efe1..9d53528063 100644
--- a/.azurepipelines/Ubuntu-GCC5.yml
+++ b/.azurepipelines/Ubuntu-GCC5.yml
@@ -18,4 +18,7 @@ jobs:
tool_chain_tag: 'GCC5'
vm_image: 'ubuntu-latest'
arch_list: "IA32,X64,ARM,AARCH64,RISCV64"
-
+ extra_install_step:
+ - bash: sudo apt-get install -y lcov
+ displayName: Install Code Coverage Tools
+ condition: and(gt(variables.pkg_count, 0), succeeded())
diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
index e4bd4b1d22..c07e5bb434 100644
--- a/.azurepipelines/Windows-VS2019.yml
+++ b/.azurepipelines/Windows-VS2019.yml
@@ -18,3 +18,8 @@ jobs:
tool_chain_tag: 'VS2019'
vm_image: 'windows-2019'
arch_list: "IA32,X64"
+ extra_install_step:
+ - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
+ displayName: Install Code Coverage Tool
+ condition: and(gt(variables.pkg_count, 0), succeeded())
+
diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
index 0e4ad019bf..0162ea97cb 100644
--- a/.azurepipelines/templates/pr-gate-build-job.yml
+++ b/.azurepipelines/templates/pr-gate-build-job.yml
@@ -12,6 +12,7 @@ parameters:
tool_chain_tag: ''
vm_image: ''
arch_list: ''
+ extra_install_step: []
# Build step
jobs:
@@ -70,3 +71,4 @@ jobs:
build_pkgs: $(Build.Pkgs)
build_targets: $(Build.Targets)
build_archs: ${{ parameters.arch_list }}
+ extra_install_step: ${{ parameters.extra_install_step }}
diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
index cb431e53fc..0568941399 100644
--- a/.azurepipelines/templates/pr-gate-steps.yml
+++ b/.azurepipelines/templates/pr-gate-steps.yml
@@ -12,6 +12,7 @@ parameters:
build_pkgs: ''
build_targets: ''
build_archs: ''
+ extra_install_step: []
steps:
- checkout: self
@@ -37,6 +38,8 @@ steps:
displayName: fetch target branch
condition: eq(variables['Build.Reason'], 'PullRequest')
+- ${{ parameters.extra_install_step }}
+
# trim the package list if this is a PR
- task: CmdLine@1
displayName: Check if ${{ parameters.build_pkgs }} need testing
@@ -125,6 +128,8 @@ steps:
TestSuites.xml
**/BUILD_TOOLS_REPORT.html
**/OVERRIDELOG.TXT
+ coverage.xml
+ coverage.html
flattenFolders: true
condition: succeededOrFailed()
diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index d92de236dc..c2821cde7d 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -194,7 +194,7 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
workspace = thebuilder.env.GetValue("WORKSPACE")
-
+ workspace = (workspace + os.sep) if workspace[-1] != os.sep else workspace
# Generate coverage file
coverageFile = ""
for testFile in testList:
@@ -204,13 +204,14 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
logging.error("UnitTest Coverage: Failed to collect coverage data.")
return 1
+ DiskName = workspace[:workspace.find (":\\") + 2]
# Generate and HTML file if requested.by each package
- ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}/Build {coverageFile}")
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}Build {coverageFile}")
if(ret != 0):
logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
return 1
- ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/cverage.html {buildOutputBase}/coverage.xml --source {workspace}")
+ ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/coverage.html {buildOutputBase}/coverage.xml --source {DiskName}")
if(ret != 0):
logging.error("UnitTest Coverage: Failed to generate HTML in single package.")
return 1
@@ -221,12 +222,11 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
for testCoverage in testCoverageList:
coverageFile += " --input_coverage=" + testCoverage
- ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}/Build/coverage.xml --working_dir={workspace}/Build {coverageFile}")
+ ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
if(ret != 0):
logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
return 1
-
- ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
+ ret = RunCmd("pycobertura", f"show --format html --output {workspace}Build/coverage.html {workspace}Build/coverage.xml --source {DiskName}")
if(ret != 0):
logging.error("UnitTest Coverage: Failed to generate HTML.")
return 1
diff --git a/pip-requirements.txt b/pip-requirements.txt
index 967da7cb37..18f5afb9c5 100644
--- a/pip-requirements.txt
+++ b/pip-requirements.txt
@@ -16,3 +16,6 @@ edk2-pytool-library==0.11.2
edk2-pytool-extensions~=0.16.0
edk2-basetools==0.1.29
antlr4-python3-runtime==4.7.1
+pycobertura==2.1.0
+lcov-cobertura==2.0.2
+
--
2.31.1.windows.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-09-30 1:53 ` [PATCH 1/3] UnitTestFrameworkPkg: " Guo, Gua
@ 2022-12-21 3:41 ` Michael Kubacki
2022-12-21 3:53 ` Guo, Gua
0 siblings, 1 reply; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 3:41 UTC (permalink / raw)
To: gua.guo, devel; +Cc: Michael D Kinney, Sean Brogan, Bret Barkelew
I'm not sure if you added some extra new lines in the markdown file to
help with readability in the raw markdown or they were accidental, but I
think they can be eliminated.
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
On 9/29/2022 9:53 PM, gua.guo@intel.com wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> In order to collect code coverage after running executable
> file, generate *.gcda and *.gcno file that require by lcov tool
> to generate code coverage report.
>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael Kubacki <mikuback@linux.microsoft.com>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> UnitTestFrameworkPkg/ReadMe.md | 37 +++++++++++++++++--
> .../UnitTestFrameworkPkg.ci.yaml | 2 +
> .../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
> 3 files changed, 38 insertions(+), 4 deletions(-)
>
> diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md
> index e696412cb3..c00e2411f8 100644
> --- a/UnitTestFrameworkPkg/ReadMe.md
> +++ b/UnitTestFrameworkPkg/ReadMe.md
> @@ -60,7 +60,7 @@ you should be good to go.
>
>
> See this example in 'SampleUnitTestUefiShell.inf'...
>
>
>
> -```
>
> +```inf
>
> [Packages]
>
> MdePkg/MdePkg.dec
>
>
>
> @@ -75,7 +75,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
> Also, if you want you test to automatically be picked up by the Test Runner plugin, you will need
>
> to make sure that the module `BASE_NAME` contains the word `Test`...
>
>
>
> -```
>
> +```inf
>
> [Defines]
>
> BASE_NAME = SampleUnitTestUefiShell
>
> ```
>
> @@ -361,7 +361,7 @@ RUNNING TEST SUITE: Int Safe Conversions Test Suite
> ```
>
>
>
> You can also, if you are so inclined, read the output from the exact instance of the test that was run during
>
> -`stuart_ci_build`. The ouput file can be found on a path that looks like:
>
> +`stuart_ci_build`. The output file can be found on a path that looks like:
>
>
>
> `Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.result.xml`
>
>
>
> @@ -399,6 +399,37 @@ CMOCKA_XML_FILE=<absolute or relative path to output file>
>
>
> This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
>
>
>
> +### Code Coverage
>
> +
>
> +Host based Unit Tests will automatically enable coverage data.
>
> +
>
> +For Windows, This is primarily leverage for pipeline builds, but this can be leveraged locally using the
>
> +OpenCppCoverage windows tool, and pycobertura is used to covert this coverage data to a human readable HTML
>
> +file. These tools must be installed to parse code coverage.
>
> +
>
> +- Windows Prerequisite
>
> + ```bash
>
> + Download and install https://github.com/OpenCppCoverage/OpenCppCoverage/tags
>
> + python -m pip install --upgrade -r ./pip-requirements.txt
>
> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
>
> + Open Build/coverage.html
>
> + ```
>
> +
>
> +
>
> +For Linux, This is primarily leveraged for pipeline builds, but this can be leveraged locally using the
>
> +lcov linux tool, and parsed using the lcov_cobertura python tool. pycobertura is used to
>
> +covert this coverage data to a human readable HTML file. These tools must be installed
>
> +to parse code coverage.
>
> +
>
> +- Linux Prerequisite
>
> + ```bash
>
> + sudo apt-get install -y lcov
>
> + python -m pip install --upgrade -r ./pip-requirements.txt
>
> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
>
> + Open Build/coverage.html
>
> + ```
>
> +
>
> +
>
> ### Important Note
>
>
>
> This works on both Windows and Linux, but is currently limited to x64 architectures. Working on getting others, but we
>
> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> index 77d51e1348..f68882e1cf 100644
> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> @@ -90,6 +90,8 @@
> "pytools",
>
> "NOFAILURE",
>
> "cmockery",
>
> + "cobertura",
>
> + "pycobertura",
>
> "DHAVE", # build flag for cmocka in the INF
>
> "corthon", # Contact GitHub account in Readme
>
> "mdkinney", # Contact GitHub account in Readme
>
> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> index 4dd8d4ac67..ca46b8513a 100644
> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> @@ -25,7 +25,8 @@
> GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>
> XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>
> !endif
>
> -
>
> + GCC:*_GCC5_*_CC_FLAGS = --coverage
>
> + GCC:*_GCC5_*_DLINK_FLAGS = --coverage
>
> [BuildOptions.common.EDKII.HOST_APPLICATION]
>
> #
>
> # MSFT
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test
2022-09-30 1:53 ` [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
@ 2022-12-21 3:42 ` Michael Kubacki
0 siblings, 0 replies; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 3:42 UTC (permalink / raw)
To: devel, gua.guo
Cc: Bob Feng, Bret Barkelew, Liming Gao, Michael D Kinney,
Sean Brogan
Since you have direct access to the UEFI builder object, I think you can
use "thebuilder.ws" to get the workspace path instead of looking it up
in the build vars.
---
I know many of these pre-existing files place parentheses around
conditions. This is not really Pythonic and I suggest new code avoid it.
The code additions in this patch have mixed usage.
This has no parentheses:
if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "GCC5":
This does:
if(ret != 0):
So, it would be:
if ret != 0:
This is probably not worth sending a new patch over alone but something
to consider if making other updates.
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
On 9/29/2022 9:53 PM, Guo, Gua wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> For GCC, use lcov to generate Unit Test code coverage
> report
>
> For VS2019, use OpenCppCoverage to generate code
> coverage report
>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Liming Gao <gaoliming@bysoft.com.cn>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> .../HostBasedUnitTestRunner.py | 119 ++++++++++++++++++
> 1 file changed, 119 insertions(+)
>
> diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index c1eeaf2625..d92de236dc 100644
> --- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> @@ -112,4 +112,123 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> " %s - %s" % (case.attrib['name'], result.text))
>
> failure_count += 1
>
>
>
> + if thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "GCC5":
>
> + self.gen_code_coverage_gcc(thebuilder)
>
> + elif thebuilder.env.GetValue("TOOL_CHAIN_TAG") == "VS2019":
>
> + self.gen_code_coverage_msvc(thebuilder)
>
> + else:
>
> + logging.info("Skipping code coverage. Only supported on GCC.")
>
> +
>
> return failure_count
>
> +
>
> + def gen_code_coverage_gcc(self, thebuilder):
>
> + logging.info("Generating UnitTest code coverage")
>
> +
>
> + buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
>
> + workspace = thebuilder.env.GetValue("WORKSPACE")
>
> +
>
> + # Generate base code coverage for all source files
>
> + ret = RunCmd("lcov", f"--no-external --capture --initial --directory {buildOutputBase} --output-file {buildOutputBase}/cov-base.info --rc lcov_branch_coverage=1")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to build initial coverage data.")
>
> + return 1
>
> +
>
> + # Coverage data for tested files only
>
> + ret = RunCmd("lcov", f"--capture --directory {buildOutputBase}/ --output-file {buildOutputBase}/coverage-test.info --rc lcov_branch_coverage=1")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to build coverage data for tested files.")
>
> + return 1
>
> +
>
> + # Aggregate all coverage data
>
> + ret = RunCmd("lcov", f"--add-tracefile {buildOutputBase}/cov-base.info --add-tracefile {buildOutputBase}/coverage-test.info --output-file {buildOutputBase}/total-coverage.info --rc lcov_branch_coverage=1")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to aggregate coverage data.")
>
> + return 1
>
> +
>
> + # Generate coverage XML
>
> + ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info -o {buildOutputBase}/compare.xml")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate coverage XML.")
>
> + return 1
>
> +
>
> + # Filter out auto-generated and test code
>
> + ret = RunCmd("lcov_cobertura",f"{buildOutputBase}/total-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {buildOutputBase}/coverage.xml")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed generate filtered coverage XML.")
>
> + return 1
>
> +
>
> + # Generate all coverage file
>
> + testCoverageList = glob.glob (f"{workspace}/Build/**/total-coverage.info", recursive=True)
>
> +
>
> + coverageFile = ""
>
> + for testCoverage in testCoverageList:
>
> + coverageFile += " --add-tracefile " + testCoverage
>
> + ret = RunCmd("lcov", f"{coverageFile} --output-file {workspace}/Build/all-coverage.info --rc lcov_branch_coverage=1")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed generate all coverage file.")
>
> + return 1
>
> +
>
> + # Generate and HTML file if requested.by each package
>
> + ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/coverage.html {buildOutputBase}/coverage.xml --source {workspace}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate HTML in single package..")
>
> +
>
> + # Generate and HTML file if requested.for all package
>
> + if os.path.isfile(f"{workspace}/Build/coverage.xml"):
>
> + os.remove(f"{workspace}/Build/coverage.xml")
>
> + ret = RunCmd("lcov_cobertura",f"{workspace}/Build/all-coverage.info --excludes ^.*UnitTest\|^.*MU\|^.*Mock\|^.*DEBUG -o {workspace}/Build/coverage.xml")
>
> +
>
> + if os.path.isfile(f"{workspace}/Build/coverage.html"):
>
> + os.remove(f"{workspace}/Build/coverage.html")
>
> + ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate HTML.")
>
> +
>
> + return 0
>
> +
>
> +
>
> + def gen_code_coverage_msvc(self, thebuilder):
>
> + logging.info("Generating UnitTest code coverage")
>
> +
>
> +
>
> + buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
>
> + testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
>
> + workspace = thebuilder.env.GetValue("WORKSPACE")
>
> +
>
> + # Generate coverage file
>
> + coverageFile = ""
>
> + for testFile in testList:
>
> + ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
>
> + coverageFile += " --input_coverage=" + testFile + ".cov"
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to collect coverage data.")
>
> + return 1
>
> +
>
> + # Generate and HTML file if requested.by each package
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
>
> + return 1
>
> +
>
> + ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/cverage.html {buildOutputBase}/coverage.xml --source {workspace}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate HTML in single package.")
>
> + return 1
>
> +
>
> + # Generate total report HTML file for all package
>
> + testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
>
> + coverageFile = ""
>
> + for testCoverage in testCoverageList:
>
> + coverageFile += " --input_coverage=" + testCoverage
>
> +
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}/Build/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
>
> + return 1
>
> +
>
> + ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
>
> + if(ret != 0):
>
> + logging.error("UnitTest Coverage: Failed to generate HTML.")
>
> + return 1
>
> +
>
> + return 0
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH 3/3] .azurepipelines: Install code coverage tool
2022-09-30 1:53 ` [PATCH 3/3] .azurepipelines: Install code coverage tool Guo, Gua
@ 2022-12-21 3:42 ` Michael Kubacki
2022-12-21 6:35 ` Guo, Gua
0 siblings, 1 reply; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 3:42 UTC (permalink / raw)
To: devel, gua.guo; +Cc: Sean Brogan, Bret Barkelew, Michael D Kinney, Liming Gao
The "DiskName" variable in this line does not appear to be used (in
gen_code_coverage_msvc()):
DiskName = workspace[:workspace.find (":\\") + 2]
---
Can you move the HostBasedUnitRunner changes to patch 2? That was scoped
for BaseTools and this patch is scoped for .azurepipelines.
On 9/29/2022 9:53 PM, Guo, Gua wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> For Windows add below tool for code coverage
> 1. OpenCppCoverage: parsing pdb file to generate coverage
> data
> 2. pycobertura: show up html format data for coverage data
>
> For Linux add below tool for code coverage
> 1. lcov: parsing gcda gcno file to generate coverage data
> 2. lcov-cobertura: convert coverage data to cobertura format
> 3. pycobertura: show up html format data for coverage data
>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
> .azurepipelines/Windows-VS2019.yml | 5 +++++
> .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> .azurepipelines/templates/pr-gate-steps.yml | 5 +++++
> .../HostBasedUnitTestRunner.py | 12 ++++++------
> pip-requirements.txt | 3 +++
> 6 files changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml
> index 3760c6efe1..9d53528063 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -18,4 +18,7 @@ jobs:
> tool_chain_tag: 'GCC5'
>
> vm_image: 'ubuntu-latest'
>
> arch_list: "IA32,X64,ARM,AARCH64,RISCV64"
>
> -
>
> + extra_install_step:
>
> + - bash: sudo apt-get install -y lcov
>
> + displayName: Install Code Coverage Tools
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml
> index e4bd4b1d22..c07e5bb434 100644
> --- a/.azurepipelines/Windows-VS2019.yml
> +++ b/.azurepipelines/Windows-VS2019.yml
> @@ -18,3 +18,8 @@ jobs:
> tool_chain_tag: 'VS2019'
>
> vm_image: 'windows-2019'
>
> arch_list: "IA32,X64"
>
> + extra_install_step:
>
> + - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> + displayName: Install Code Coverage Tool
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> +
>
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml
> index 0e4ad019bf..0162ea97cb 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
> tool_chain_tag: ''
>
> vm_image: ''
>
> arch_list: ''
>
> + extra_install_step: []
>
>
>
> # Build step
>
> jobs:
>
> @@ -70,3 +71,4 @@ jobs:
> build_pkgs: $(Build.Pkgs)
>
> build_targets: $(Build.Targets)
>
> build_archs: ${{ parameters.arch_list }}
>
> + extra_install_step: ${{ parameters.extra_install_step }}
>
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fc..0568941399 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
> build_pkgs: ''
>
> build_targets: ''
>
> build_archs: ''
>
> + extra_install_step: []
>
>
>
> steps:
>
> - checkout: self
>
> @@ -37,6 +38,8 @@ steps:
> displayName: fetch target branch
>
> condition: eq(variables['Build.Reason'], 'PullRequest')
>
>
>
> +- ${{ parameters.extra_install_step }}
>
> +
>
> # trim the package list if this is a PR
>
> - task: CmdLine@1
>
> displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> @@ -125,6 +128,8 @@ steps:
> TestSuites.xml
>
> **/BUILD_TOOLS_REPORT.html
>
> **/OVERRIDELOG.TXT
>
> + coverage.xml
>
> + coverage.html
>
> flattenFolders: true
>
> condition: succeededOrFailed()
>
>
>
> diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index d92de236dc..c2821cde7d 100644
> --- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> @@ -194,7 +194,7 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> buildOutputBase = thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
>
> testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
>
> workspace = thebuilder.env.GetValue("WORKSPACE")
>
> -
>
> + workspace = (workspace + os.sep) if workspace[-1] != os.sep else workspace
>
> # Generate coverage file
>
> coverageFile = ""
>
> for testFile in testList:
>
> @@ -204,13 +204,14 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> logging.error("UnitTest Coverage: Failed to collect coverage data.")
>
> return 1
>
>
>
> + DiskName = workspace[:workspace.find (":\\") + 2]
>
> # Generate and HTML file if requested.by each package
>
> - ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}Build {coverageFile}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
>
> return 1
>
>
>
> - ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/cverage.html {buildOutputBase}/coverage.xml --source {workspace}")
>
> + ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/coverage.html {buildOutputBase}/coverage.xml --source {DiskName}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate HTML in single package.")
>
> return 1
>
> @@ -221,12 +222,11 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> for testCoverage in testCoverageList:
>
> coverageFile += " --input_coverage=" + testCoverage
>
>
>
> - ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}/Build/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
>
> return 1
>
> -
>
> - ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
>
> + ret = RunCmd("pycobertura", f"show --format html --output {workspace}Build/coverage.html {workspace}Build/coverage.xml --source {DiskName}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate HTML.")
>
> return 1
>
> diff --git a/pip-requirements.txt b/pip-requirements.txt
> index 967da7cb37..18f5afb9c5 100644
> --- a/pip-requirements.txt
> +++ b/pip-requirements.txt
> @@ -16,3 +16,6 @@ edk2-pytool-library==0.11.2
> edk2-pytool-extensions~=0.16.0
>
> edk2-basetools==0.1.29
>
> antlr4-python3-runtime==4.7.1
>
> +pycobertura==2.1.0
>
> +lcov-cobertura==2.0.2
>
> +
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-12-21 3:41 ` Michael Kubacki
@ 2022-12-21 3:53 ` Guo, Gua
2022-12-21 3:58 ` Michael Kubacki
0 siblings, 1 reply; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 3:53 UTC (permalink / raw)
To: Michael Kubacki, devel@edk2.groups.io
Cc: Kinney, Michael D, Sean Brogan, Barkelew, Bret
You means "```" to "```inf ", it just make markdown file more readability only.
Thanks,
Gua
-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 21, 2022 11:41 AM
To: Guo, Gua <gua.guo@intel.com>; devel@edk2.groups.io
Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>
Subject: Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
I'm not sure if you added some extra new lines in the markdown file to help with readability in the raw markdown or they were accidental, but I think they can be eliminated.
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
On 9/29/2022 9:53 PM, gua.guo@intel.com wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> In order to collect code coverage after running executable file,
> generate *.gcda and *.gcno file that require by lcov tool to generate
> code coverage report.
>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael Kubacki <mikuback@linux.microsoft.com>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> UnitTestFrameworkPkg/ReadMe.md | 37 +++++++++++++++++--
> .../UnitTestFrameworkPkg.ci.yaml | 2 +
> .../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
> 3 files changed, 38 insertions(+), 4 deletions(-)
>
> diff --git a/UnitTestFrameworkPkg/ReadMe.md
> b/UnitTestFrameworkPkg/ReadMe.md index e696412cb3..c00e2411f8 100644
> --- a/UnitTestFrameworkPkg/ReadMe.md
> +++ b/UnitTestFrameworkPkg/ReadMe.md
> @@ -60,7 +60,7 @@ you should be good to go.
>
>
> See this example in 'SampleUnitTestUefiShell.inf'...
>
>
>
> -```
>
> +```inf
>
> [Packages]
>
> MdePkg/MdePkg.dec
>
>
>
> @@ -75,7 +75,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
> Also, if you want you test to automatically be picked up by the Test
> Runner plugin, you will need
>
> to make sure that the module `BASE_NAME` contains the word `Test`...
>
>
>
> -```
>
> +```inf
>
> [Defines]
>
> BASE_NAME = SampleUnitTestUefiShell
>
> ```
>
> @@ -361,7 +361,7 @@ RUNNING TEST SUITE: Int Safe Conversions Test Suite
> ```
>
>
>
> You can also, if you are so inclined, read the output from the exact
> instance of the test that was run during
>
> -`stuart_ci_build`. The ouput file can be found on a path that looks like:
>
> +`stuart_ci_build`. The output file can be found on a path that looks like:
>
>
>
>
> `Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.res
> ult.xml`
>
>
>
> @@ -399,6 +399,37 @@ CMOCKA_XML_FILE=<absolute or relative path to
> output file>
>
>
> This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
>
>
>
> +### Code Coverage
>
> +
>
> +Host based Unit Tests will automatically enable coverage data.
>
> +
>
> +For Windows, This is primarily leverage for pipeline builds, but this
> +can be leveraged locally using the
>
> +OpenCppCoverage windows tool, and pycobertura is used to covert this
> +coverage data to a human readable HTML
>
> +file. These tools must be installed to parse code coverage.
>
> +
>
> +- Windows Prerequisite
>
> + ```bash
>
> + Download and install
> + https://github.com/OpenCppCoverage/OpenCppCoverage/tags
>
> + python -m pip install --upgrade -r ./pip-requirements.txt
>
> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
> + TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
>
> + Open Build/coverage.html
>
> + ```
>
> +
>
> +
>
> +For Linux, This is primarily leveraged for pipeline builds, but this
> +can be leveraged locally using the
>
> +lcov linux tool, and parsed using the lcov_cobertura python tool.
> +pycobertura is used to
>
> +covert this coverage data to a human readable HTML file. These tools
> +must be installed
>
> +to parse code coverage.
>
> +
>
> +- Linux Prerequisite
>
> + ```bash
>
> + sudo apt-get install -y lcov
>
> + python -m pip install --upgrade -r ./pip-requirements.txt
>
> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
> + TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
>
> + Open Build/coverage.html
>
> + ```
>
> +
>
> +
>
> ### Important Note
>
>
>
> This works on both Windows and Linux, but is currently limited to
> x64 architectures. Working on getting others, but we
>
> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> index 77d51e1348..f68882e1cf 100644
> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
> @@ -90,6 +90,8 @@
> "pytools",
>
> "NOFAILURE",
>
> "cmockery",
>
> + "cobertura",
>
> + "pycobertura",
>
> "DHAVE", # build flag for cmocka in the INF
>
> "corthon", # Contact GitHub account in Readme
>
> "mdkinney", # Contact GitHub account in Readme
>
> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> index 4dd8d4ac67..ca46b8513a 100644
> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
> @@ -25,7 +25,8 @@
> GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>
> XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>
> !endif
>
> -
>
> + GCC:*_GCC5_*_CC_FLAGS = --coverage
>
> + GCC:*_GCC5_*_DLINK_FLAGS = --coverage
>
> [BuildOptions.common.EDKII.HOST_APPLICATION]
>
> #
>
> # MSFT
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-12-21 3:53 ` Guo, Gua
@ 2022-12-21 3:58 ` Michael Kubacki
2022-12-21 4:02 ` Guo, Gua
0 siblings, 1 reply; 11+ messages in thread
From: Michael Kubacki @ 2022-12-21 3:58 UTC (permalink / raw)
To: Guo, Gua, devel@edk2.groups.io
Cc: Kinney, Michael D, Sean Brogan, Barkelew, Bret
It might cause problems depending on markdown linter settings, but if it
passes CI, it's fine.
Thanks,
Michael
On 12/20/2022 10:53 PM, Guo, Gua wrote:
> You means "```" to "```inf ", it just make markdown file more readability only.
>
> Thanks,
> Gua
> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com>
> Sent: Wednesday, December 21, 2022 11:41 AM
> To: Guo, Gua <gua.guo@intel.com>; devel@edk2.groups.io
> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>
> Subject: Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
>
> I'm not sure if you added some extra new lines in the markdown file to help with readability in the raw markdown or they were accidental, but I think they can be eliminated.
>
> Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
>
> On 9/29/2022 9:53 PM, gua.guo@intel.com wrote:
>> From: Gua Guo <gua.guo@intel.com>
>>
>> In order to collect code coverage after running executable file,
>> generate *.gcda and *.gcno file that require by lcov tool to generate
>> code coverage report.
>>
>> Cc: Michael D Kinney <michael.d.kinney@intel.com>
>> Cc: Sean Brogan <sean.brogan@microsoft.com>
>> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
>> Cc: Michael Kubacki <mikuback@linux.microsoft.com>
>> Signed-off-by: Gua Guo <gua.guo@intel.com>
>> ---
>> UnitTestFrameworkPkg/ReadMe.md | 37 +++++++++++++++++--
>> .../UnitTestFrameworkPkg.ci.yaml | 2 +
>> .../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
>> 3 files changed, 38 insertions(+), 4 deletions(-)
>>
>> diff --git a/UnitTestFrameworkPkg/ReadMe.md
>> b/UnitTestFrameworkPkg/ReadMe.md index e696412cb3..c00e2411f8 100644
>> --- a/UnitTestFrameworkPkg/ReadMe.md
>> +++ b/UnitTestFrameworkPkg/ReadMe.md
>> @@ -60,7 +60,7 @@ you should be good to go.
>>
>>
>> See this example in 'SampleUnitTestUefiShell.inf'...
>>
>>
>>
>> -```
>>
>> +```inf
>>
>> [Packages]
>>
>> MdePkg/MdePkg.dec
>>
>>
>>
>> @@ -75,7 +75,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
>> Also, if you want you test to automatically be picked up by the Test
>> Runner plugin, you will need
>>
>> to make sure that the module `BASE_NAME` contains the word `Test`...
>>
>>
>>
>> -```
>>
>> +```inf
>>
>> [Defines]
>>
>> BASE_NAME = SampleUnitTestUefiShell
>>
>> ```
>>
>> @@ -361,7 +361,7 @@ RUNNING TEST SUITE: Int Safe Conversions Test Suite
>> ```
>>
>>
>>
>> You can also, if you are so inclined, read the output from the exact
>> instance of the test that was run during
>>
>> -`stuart_ci_build`. The ouput file can be found on a path that looks like:
>>
>> +`stuart_ci_build`. The output file can be found on a path that looks like:
>>
>>
>>
>>
>> `Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.res
>> ult.xml`
>>
>>
>>
>> @@ -399,6 +399,37 @@ CMOCKA_XML_FILE=<absolute or relative path to
>> output file>
>>
>>
>> This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
>>
>>
>>
>> +### Code Coverage
>>
>> +
>>
>> +Host based Unit Tests will automatically enable coverage data.
>>
>> +
>>
>> +For Windows, This is primarily leverage for pipeline builds, but this
>> +can be leveraged locally using the
>>
>> +OpenCppCoverage windows tool, and pycobertura is used to covert this
>> +coverage data to a human readable HTML
>>
>> +file. These tools must be installed to parse code coverage.
>>
>> +
>>
>> +- Windows Prerequisite
>>
>> + ```bash
>>
>> + Download and install
>> + https://github.com/OpenCppCoverage/OpenCppCoverage/tags
>>
>> + python -m pip install --upgrade -r ./pip-requirements.txt
>>
>> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
>> + TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
>>
>> + Open Build/coverage.html
>>
>> + ```
>>
>> +
>>
>> +
>>
>> +For Linux, This is primarily leveraged for pipeline builds, but this
>> +can be leveraged locally using the
>>
>> +lcov linux tool, and parsed using the lcov_cobertura python tool.
>> +pycobertura is used to
>>
>> +covert this coverage data to a human readable HTML file. These tools
>> +must be installed
>>
>> +to parse code coverage.
>>
>> +
>>
>> +- Linux Prerequisite
>>
>> + ```bash
>>
>> + sudo apt-get install -y lcov
>>
>> + python -m pip install --upgrade -r ./pip-requirements.txt
>>
>> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
>> + TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
>>
>> + Open Build/coverage.html
>>
>> + ```
>>
>> +
>>
>> +
>>
>> ### Important Note
>>
>>
>>
>> This works on both Windows and Linux, but is currently limited to
>> x64 architectures. Working on getting others, but we
>>
>> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> index 77d51e1348..f68882e1cf 100644
>> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> @@ -90,6 +90,8 @@
>> "pytools",
>>
>> "NOFAILURE",
>>
>> "cmockery",
>>
>> + "cobertura",
>>
>> + "pycobertura",
>>
>> "DHAVE", # build flag for cmocka in the INF
>>
>> "corthon", # Contact GitHub account in Readme
>>
>> "mdkinney", # Contact GitHub account in Readme
>>
>> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> index 4dd8d4ac67..ca46b8513a 100644
>> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> @@ -25,7 +25,8 @@
>> GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>>
>> XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>>
>> !endif
>>
>> -
>>
>> + GCC:*_GCC5_*_CC_FLAGS = --coverage
>>
>> + GCC:*_GCC5_*_DLINK_FLAGS = --coverage
>>
>> [BuildOptions.common.EDKII.HOST_APPLICATION]
>>
>> #
>>
>> # MSFT
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
2022-12-21 3:58 ` Michael Kubacki
@ 2022-12-21 4:02 ` Guo, Gua
0 siblings, 0 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 4:02 UTC (permalink / raw)
To: Michael Kubacki, devel@edk2.groups.io
Cc: Kinney, Michael D, Sean Brogan, Barkelew, Bret
[-- Attachment #1.1: Type: text/plain, Size: 7864 bytes --]
From CI side, it looks good.
https://github.com/tianocore/edk2/pull/3349
[cid:image001.png@01D91534.0BD436C0]
From my local side it looks good also.
[cid:image002.png@01D91534.0BD436C0]
-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 21, 2022 11:59 AM
To: Guo, Gua <gua.guo@intel.com>; devel@edk2.groups.io
Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>
Subject: Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage support for GCC
It might cause problems depending on markdown linter settings, but if it passes CI, it's fine.
Thanks,
Michael
On 12/20/2022 10:53 PM, Guo, Gua wrote:
> You means "```" to "```inf ", it just make markdown file more readability only.
>
> Thanks,
> Gua
> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>>
> Sent: Wednesday, December 21, 2022 11:41 AM
> To: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
> Cc: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Sean Brogan
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
> <bret.barkelew@microsoft.com<mailto:bret.barkelew@microsoft.com>>
> Subject: Re: [PATCH 1/3] UnitTestFrameworkPkg: Add code coverage
> support for GCC
>
> I'm not sure if you added some extra new lines in the markdown file to help with readability in the raw markdown or they were accidental, but I think they can be eliminated.
>
> Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com<mailto:michael.kubacki@microsoft.com>>
>
> On 9/29/2022 9:53 PM, gua.guo@intel.com<mailto:gua.guo@intel.com> wrote:
>> From: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
>>
>> In order to collect code coverage after running executable file,
>> generate *.gcda and *.gcno file that require by lcov tool to generate
>> code coverage report.
>>
>> Cc: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
>> Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
>> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
>> Cc: Michael Kubacki <mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>>
>> Signed-off-by: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>
>> ---
>> UnitTestFrameworkPkg/ReadMe.md | 37 +++++++++++++++++--
>> .../UnitTestFrameworkPkg.ci.yaml | 2 +
>> .../UnitTestFrameworkPkgHost.dsc.inc | 3 +-
>> 3 files changed, 38 insertions(+), 4 deletions(-)
>>
>> diff --git a/UnitTestFrameworkPkg/ReadMe.md
>> b/UnitTestFrameworkPkg/ReadMe.md index e696412cb3..c00e2411f8 100644
>> --- a/UnitTestFrameworkPkg/ReadMe.md
>> +++ b/UnitTestFrameworkPkg/ReadMe.md
>> @@ -60,7 +60,7 @@ you should be good to go.
>>
>>
>> See this example in 'SampleUnitTestUefiShell.inf'...
>>
>>
>>
>> -```
>>
>> +```inf
>>
>> [Packages]
>>
>> MdePkg/MdePkg.dec
>>
>>
>>
>> @@ -75,7 +75,7 @@ See this example in 'SampleUnitTestUefiShell.inf'...
>> Also, if you want you test to automatically be picked up by the
>> Test Runner plugin, you will need
>>
>> to make sure that the module `BASE_NAME` contains the word `Test`...
>>
>>
>>
>> -```
>>
>> +```inf
>>
>> [Defines]
>>
>> BASE_NAME = SampleUnitTestUefiShell
>>
>> ```
>>
>> @@ -361,7 +361,7 @@ RUNNING TEST SUITE: Int Safe Conversions Test Suite
>> ```
>>
>>
>>
>> You can also, if you are so inclined, read the output from the
>> exact instance of the test that was run during
>>
>> -`stuart_ci_build`. The ouput file can be found on a path that looks like:
>>
>> +`stuart_ci_build`. The output file can be found on a path that looks like:
>>
>>
>>
>>
>> `Build/<Package>/HostTest/<Arch>/<TestName>.<TestSuiteName>.<Arch>.re
>> s
>> ult.xml`
>>
>>
>>
>> @@ -399,6 +399,37 @@ CMOCKA_XML_FILE=<absolute or relative path to
>> output file>
>>
>>
>> This mode is used by the test running plugin to aggregate the results for CI test status reporting in the web view.
>>
>>
>>
>> +### Code Coverage
>>
>> +
>>
>> +Host based Unit Tests will automatically enable coverage data.
>>
>> +
>>
>> +For Windows, This is primarily leverage for pipeline builds, but
>> +this can be leveraged locally using the
>>
>> +OpenCppCoverage windows tool, and pycobertura is used to covert this
>> +coverage data to a human readable HTML
>>
>> +file. These tools must be installed to parse code coverage.
>>
>> +
>>
>> +- Windows Prerequisite
>>
>> + ```bash
>>
>> + Download and install
>> + https://github.com/OpenCppCoverage/OpenCppCoverage/tags
>>
>> + python -m pip install --upgrade -r ./pip-requirements.txt
>>
>> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
>> + TOOL_CHAIN_TAG=VS2019 -p MdeModulePkg
>>
>> + Open Build/coverage.html
>>
>> + ```
>>
>> +
>>
>> +
>>
>> +For Linux, This is primarily leveraged for pipeline builds, but this
>> +can be leveraged locally using the
>>
>> +lcov linux tool, and parsed using the lcov_cobertura python tool.
>> +pycobertura is used to
>>
>> +covert this coverage data to a human readable HTML file. These tools
>> +must be installed
>>
>> +to parse code coverage.
>>
>> +
>>
>> +- Linux Prerequisite
>>
>> + ```bash
>>
>> + sudo apt-get install -y lcov
>>
>> + python -m pip install --upgrade -r ./pip-requirements.txt
>>
>> + stuart_ci_build -c .pytool/CISettings.py -t NOOPT
>> + TOOL_CHAIN_TAG=GCC5 -p MdeModulePkg
>>
>> + Open Build/coverage.html
>>
>> + ```
>>
>> +
>>
>> +
>>
>> ### Important Note
>>
>>
>>
>> This works on both Windows and Linux, but is currently limited to
>> x64 architectures. Working on getting others, but we
>>
>> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> index 77d51e1348..f68882e1cf 100644
>> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
>> @@ -90,6 +90,8 @@
>> "pytools",
>>
>> "NOFAILURE",
>>
>> "cmockery",
>>
>> + "cobertura",
>>
>> + "pycobertura",
>>
>> "DHAVE", # build flag for cmocka in the INF
>>
>> "corthon", # Contact GitHub account in Readme
>>
>> "mdkinney", # Contact GitHub account in Readme
>>
>> diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> index 4dd8d4ac67..ca46b8513a 100644
>> --- a/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> +++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc
>> @@ -25,7 +25,8 @@
>> GCC:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>>
>> XCODE:*_*_*_CC_FLAGS = -D UNIT_TESTING_DEBUG=1
>>
>> !endif
>>
>> -
>>
>> + GCC:*_GCC5_*_CC_FLAGS = --coverage
>>
>> + GCC:*_GCC5_*_DLINK_FLAGS = --coverage
>>
>> [BuildOptions.common.EDKII.HOST_APPLICATION]
>>
>> #
>>
>> # MSFT
>>
[-- Attachment #1.2: Type: text/html, Size: 23831 bytes --]
[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 53420 bytes --]
[-- Attachment #3: image002.png --]
[-- Type: image/png, Size: 183150 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [edk2-devel] [PATCH 3/3] .azurepipelines: Install code coverage tool
2022-12-21 3:42 ` [edk2-devel] " Michael Kubacki
@ 2022-12-21 6:35 ` Guo, Gua
0 siblings, 0 replies; 11+ messages in thread
From: Guo, Gua @ 2022-12-21 6:35 UTC (permalink / raw)
To: Michael Kubacki, devel@edk2.groups.io
Cc: Sean Brogan, Barkelew, Bret, Kinney, Michael D, Gao, Liming
Will follow up in my next patch.
-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 21, 2022 11:42 AM
To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>; Barkelew, Bret <bret.barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>
Subject: Re: [edk2-devel] [PATCH 3/3] .azurepipelines: Install code coverage tool
The "DiskName" variable in this line does not appear to be used (in
gen_code_coverage_msvc()):
DiskName = workspace[:workspace.find (":\\") + 2]
---
Can you move the HostBasedUnitRunner changes to patch 2? That was scoped for BaseTools and this patch is scoped for .azurepipelines.
On 9/29/2022 9:53 PM, Guo, Gua wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> For Windows add below tool for code coverage 1. OpenCppCoverage:
> parsing pdb file to generate coverage data 2. pycobertura: show up
> html format data for coverage data
>
> For Linux add below tool for code coverage 1. lcov: parsing gcda gcno
> file to generate coverage data 2. lcov-cobertura: convert coverage
> data to cobertura format 3. pycobertura: show up html format data for
> coverage data
>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
> .azurepipelines/Ubuntu-GCC5.yml | 5 ++++-
> .azurepipelines/Windows-VS2019.yml | 5 +++++
> .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> .azurepipelines/templates/pr-gate-steps.yml | 5 +++++
> .../HostBasedUnitTestRunner.py | 12 ++++++------
> pip-requirements.txt | 3 +++
> 6 files changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/.azurepipelines/Ubuntu-GCC5.yml
> b/.azurepipelines/Ubuntu-GCC5.yml index 3760c6efe1..9d53528063 100644
> --- a/.azurepipelines/Ubuntu-GCC5.yml
> +++ b/.azurepipelines/Ubuntu-GCC5.yml
> @@ -18,4 +18,7 @@ jobs:
> tool_chain_tag: 'GCC5'
>
> vm_image: 'ubuntu-latest'
>
> arch_list: "IA32,X64,ARM,AARCH64,RISCV64"
>
> -
>
> + extra_install_step:
>
> + - bash: sudo apt-get install -y lcov
>
> + displayName: Install Code Coverage Tools
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> diff --git a/.azurepipelines/Windows-VS2019.yml
> b/.azurepipelines/Windows-VS2019.yml
> index e4bd4b1d22..c07e5bb434 100644
> --- a/.azurepipelines/Windows-VS2019.yml
> +++ b/.azurepipelines/Windows-VS2019.yml
> @@ -18,3 +18,8 @@ jobs:
> tool_chain_tag: 'VS2019'
>
> vm_image: 'windows-2019'
>
> arch_list: "IA32,X64"
>
> + extra_install_step:
>
> + - powershell: choco install opencppcoverage; Write-Host "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
>
> + displayName: Install Code Coverage Tool
>
> + condition: and(gt(variables.pkg_count, 0), succeeded())
>
> +
>
> diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> b/.azurepipelines/templates/pr-gate-build-job.yml
> index 0e4ad019bf..0162ea97cb 100644
> --- a/.azurepipelines/templates/pr-gate-build-job.yml
> +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> @@ -12,6 +12,7 @@ parameters:
> tool_chain_tag: ''
>
> vm_image: ''
>
> arch_list: ''
>
> + extra_install_step: []
>
>
>
> # Build step
>
> jobs:
>
> @@ -70,3 +71,4 @@ jobs:
> build_pkgs: $(Build.Pkgs)
>
> build_targets: $(Build.Targets)
>
> build_archs: ${{ parameters.arch_list }}
>
> + extra_install_step: ${{ parameters.extra_install_step }}
>
> diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> b/.azurepipelines/templates/pr-gate-steps.yml
> index cb431e53fc..0568941399 100644
> --- a/.azurepipelines/templates/pr-gate-steps.yml
> +++ b/.azurepipelines/templates/pr-gate-steps.yml
> @@ -12,6 +12,7 @@ parameters:
> build_pkgs: ''
>
> build_targets: ''
>
> build_archs: ''
>
> + extra_install_step: []
>
>
>
> steps:
>
> - checkout: self
>
> @@ -37,6 +38,8 @@ steps:
> displayName: fetch target branch
>
> condition: eq(variables['Build.Reason'], 'PullRequest')
>
>
>
> +- ${{ parameters.extra_install_step }}
>
> +
>
> # trim the package list if this is a PR
>
> - task: CmdLine@1
>
> displayName: Check if ${{ parameters.build_pkgs }} need testing
>
> @@ -125,6 +128,8 @@ steps:
> TestSuites.xml
>
> **/BUILD_TOOLS_REPORT.html
>
> **/OVERRIDELOG.TXT
>
> + coverage.xml
>
> + coverage.html
>
> flattenFolders: true
>
> condition: succeededOrFailed()
>
>
>
> diff --git
> a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index d92de236dc..c2821cde7d 100644
> ---
> a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner
> +++ .py
> @@ -194,7 +194,7 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> buildOutputBase =
> thebuilder.env.GetValue("BUILD_OUTPUT_BASE")
>
> testList = glob.glob(os.path.join(buildOutputBase,
> "**","*Test*.exe"), recursive=True)
>
> workspace = thebuilder.env.GetValue("WORKSPACE")
>
> -
>
> + workspace = (workspace + os.sep) if workspace[-1] != os.sep
> + else workspace
>
> # Generate coverage file
>
> coverageFile = ""
>
> for testFile in testList:
>
> @@ -204,13 +204,14 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> logging.error("UnitTest Coverage: Failed to collect
> coverage data.")
>
> return 1
>
>
>
> + DiskName = workspace[:workspace.find (":\\") + 2]
>
> # Generate and HTML file if requested.by each package
>
> - ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{buildOutputBase}/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type
> + cobertura:{buildOutputBase}/coverage.xml
> + --working_dir={workspace}Build {coverageFile}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate
> cobertura format xml in single package.")
>
> return 1
>
>
>
> - ret = RunCmd("pycobertura", f"show --format html --output {buildOutputBase}/cverage.html {buildOutputBase}/coverage.xml --source {workspace}")
>
> + ret = RunCmd("pycobertura", f"show --format html --output
> + {buildOutputBase}/coverage.html {buildOutputBase}/coverage.xml
> + --source {DiskName}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate
> HTML in single package.")
>
> return 1
>
> @@ -221,12 +222,11 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
> for testCoverage in testCoverageList:
>
> coverageFile += " --input_coverage=" + testCoverage
>
>
>
> - ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}/Build/coverage.xml --working_dir={workspace}/Build {coverageFile}")
>
> + ret = RunCmd("OpenCppCoverage", f"--export_type
> + cobertura:{workspace}Build/coverage.xml
> + --working_dir={workspace}Build {coverageFile}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate
> cobertura format xml.")
>
> return 1
>
> -
>
> - ret = RunCmd("pycobertura", f"show --format html --output {workspace}/Build/coverage.html {workspace}/Build/coverage.xml --source {workspace}")
>
> + ret = RunCmd("pycobertura", f"show --format html --output
> + {workspace}Build/coverage.html {workspace}Build/coverage.xml
> + --source {DiskName}")
>
> if(ret != 0):
>
> logging.error("UnitTest Coverage: Failed to generate
> HTML.")
>
> return 1
>
> diff --git a/pip-requirements.txt b/pip-requirements.txt index
> 967da7cb37..18f5afb9c5 100644
> --- a/pip-requirements.txt
> +++ b/pip-requirements.txt
> @@ -16,3 +16,6 @@ edk2-pytool-library==0.11.2
> edk2-pytool-extensions~=0.16.0
>
> edk2-basetools==0.1.29
>
> antlr4-python3-runtime==4.7.1
>
> +pycobertura==2.1.0
>
> +lcov-cobertura==2.0.2
>
> +
>
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-12-21 6:36 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-09-30 1:53 [PATCH 0/3] Add code coverage support for GCC Guo, Gua
2022-09-30 1:53 ` [PATCH 1/3] UnitTestFrameworkPkg: " Guo, Gua
2022-12-21 3:41 ` Michael Kubacki
2022-12-21 3:53 ` Guo, Gua
2022-12-21 3:58 ` Michael Kubacki
2022-12-21 4:02 ` Guo, Gua
2022-09-30 1:53 ` [PATCH 2/3] BaseTools/Plugin: Add coverage support for Unit Test Guo, Gua
2022-12-21 3:42 ` [edk2-devel] " Michael Kubacki
2022-09-30 1:53 ` [PATCH 3/3] .azurepipelines: Install code coverage tool Guo, Gua
2022-12-21 3:42 ` [edk2-devel] " Michael Kubacki
2022-12-21 6:35 ` Guo, Gua
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox