Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code coverage tool

["Michael Kubacki" <mikuback@linux.microsoft.com>]

I understand the problem. Have you looked into tools like the one I 
linked? It also has a DevOps extension - 
https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator.

I have not looked at it closely, but at a high-level, it appears to at 
least attempt to address this problem.

Thanks,
Michael

On 12/20/2022 11:20 PM, Guo, Gua wrote:
> @Kubacki, Michael <mailto:michael.kubacki@microsoft.com>
> 
> Try to see the block diagram "Current behavior" vs "Expected behavior". 
> We find a way to transfer “Current behavior” to “Expected behavior” if 
> we want to support pipeline tab “Code Coverage”. Hope it’s helpful for 
> you to realize the limitation.
> 
> Thanks,
> 
> Gua
> 
> -----Original Message-----
> From: Michael Kubacki <mikuback@linux.microsoft.com>
> Sent: Wednesday, December 21, 2022 11:11 AM
> To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, Michael 
> D <michael.d.kinney@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; 
> Kubacki, Michael <michael.kubacki@microsoft.com>
> Cc: Barkelew, Bret <bret.barkelew@microsoft.com>; Gao, Liming 
> <gaoliming@byosoft.com.cn>
> Subject: Re: [edk2-devel] [PATCH v6 3/3] .azurepipelines: Install code 
> coverage tool
> 
> I might be misunderstanding the problem, but can you merge the XML file 
> from each package into one and then pass that to the 
> "PublishCodeCoverageResult" task?
> 
> Perhaps with something like this
> 
> https://github.com/danielpalme/ReportGenerator 
> <https://github.com/danielpalme/ReportGenerator>
> 
> Thanks,
> 
> Michael
> 
> On 12/20/2022 8:32 PM, Guo, Gua wrote:
> 
>  > @Kinney, Michael D <mailto:michael.d.kinney@intel.com 
> <mailto:michael.d.kinney@intel.com>>
> 
>  >
> 
>  > I assume you want to see pipeline "Code Coverage" like below “3. Azure
> 
>  > Pipeline Change” diagram.
> 
>  >
> 
>  > But it will have a limitation. Current Edk2 CI/CD design is use
> 
>  > “Ubuntu GCC PR5” and “Windows VS2019 PR” that include each subtask for
> 
>  > each package. And each package will have itself Coverage Data, but the
> 
>  > design can not be covered by Azure API “PublishCodeCoverageResult”.
> 
>  >
> 
>  > That’s why I don’t integrated it into the change.
> 
>  >
> 
>  >  1. Azure API doc:
> 
>  > 
> https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines 
> <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/publish-code-coverage-results-v1?view=azure-pipelines>
> 
>  >
> 
>  > <https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/refere
> 
>  > nce/publish-code-coverage-results-v1?view=azure-pipelines>
> 
>  >
> 
>  >  2. Code Change:
> 
>  >
> 
>  >  3. Azure Pipeline Change:
> 
>  >
> 
>  > VS2019
> 
>  >
> 
>  >
> 
>  >
> 
>  > GCC5
> 
>  >
> 
>  > 
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204 
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102204>
> 
>  > 22452
> 
>  > <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 
>  > 422452>
> 
>  >
> 
>  >
> 
>  >
> 
>  > 
> https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206 
> <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=102206>
> 
>  > 15640
> 
>  > <https://github.com/tianocore/edk2/pull/3349/checks?check_run_id=10220
> 
>  > 615640>
> 
>  >
> 
>  > 
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_ 
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&_>
> 
>  > a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
> 
>  > overage-tab
> 
>  > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73952&
> 
>  > _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
> 
>  > coverage-tab>
> 
>  >
> 
>  >
> 
>  >
> 
>  > 
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_ 
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&_>
> 
>  > a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=codec
> 
>  > overage-tab
> 
>  > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73951&
> 
>  > _a=summary&tab=ms.vss-codecoverage-web.code-coverage-details&view=code
> 
>  > coverage-tab>
> 
>  >
> 
>  > Thanks,
> 
>  >
> 
>  > Gua
> 
>  >
> 
>  > -----Original Message-----
> 
>  > From: Kinney, Michael D <michael.d.kinney@intel.com 
> <mailto:michael.d.kinney@intel.com>>
> 
>  > Sent: Wednesday, December 21, 2022 5:23 AM
> 
>  > To: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com>>; 
> devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney,
> 
>  > Michael D <michael.d.kinney@intel.com 
> <mailto:michael.d.kinney@intel.com>>; Sean Brogan
> 
>  > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com>>; 
> Kubacki, Michael
> 
>  > <michael.kubacki@microsoft.com <mailto:michael.kubacki@microsoft.com>>
> 
>  > Cc: Sean Brogan <sean.brogan@microsoft.com 
> <mailto:sean.brogan@microsoft.com>>; Barkelew, Bret
> 
>  > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com>>; 
> Gao, Liming <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>;
> 
>  > Kinney, Michael D <michael.d.kinney@intel.com 
> <mailto:michael.d.kinney@intel.com>>
> 
>  > Subject: RE: [PATCH v6 3/3] .azurepipelines: Install code coverage
> 
>  > tool
> 
>  >
> 
>  > Hi Gua,
> 
>  >
> 
>  > I created a PR with this series.
> 
>  >
> 
>  > https://github.com/tianocore/edk2/pull/3795 
> <https://github.com/tianocore/edk2/pull/3795>
> 
>  > <https://github.com/tianocore/edk2/pull/3795 
> <https://github.com/tianocore/edk2/pull/3795>>
> 
>  >
> 
>  > When I follow link to Azure Pipelines for the GCC builds, I can see
> 
>  > the coverage.xml file in the ZIP file of artifacts.
> 
>  >
> 
>  > 
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v 
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&v>
> 
>  > iew=results
> 
>  > <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=73664&
> 
>  > view=results>
> 
>  >
> 
>  > But I do not see an information from Azure Pielines view on code
> 
>  > coverage or a way to view the lines covered from the Azure Pipelines
> 
>  > web pages.
> 
>  >
> 
>  > Are there additional items to enable?
> 
>  >
> 
>  > Mike
> 
>  >
> 
>  >  > -----Original Message-----
> 
>  >
> 
>  >  > From: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com 
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
> 
>  >
> 
>  >  > Sent: Tuesday, December 13, 2022 7:20 PM
> 
>  >
> 
>  >  > To: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
> <mailto:devel@edk2.groups.io <mailto:devel@edk2.groups.io>>
> 
>  >
> 
>  >  > Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com 
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>; Sean
> 
>  > Brogan
> 
>  >
> 
>  >  > <sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com 
> <mailto:sean.brogan@microsoft.com%20%3cmailto:sean.brogan@microsoft.com>>>;
> 
>  > Barkelew, Bret
> 
>  >
> 
>  >  > <bret.barkelew@microsoft.com <mailto:bret.barkelew@microsoft.com 
> <mailto:bret.barkelew@microsoft.com%20%3cmailto:bret.barkelew@microsoft.com>>>;
> 
>  > Kinney, Michael D
> 
>  >
> 
>  >  > <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com 
> <mailto:michael.d.kinney@intel.com%20%3cmailto:michael.d.kinney@intel.com>>>;
> 
>  > Gao, Liming <gaoliming@byosoft.com.cn
> 
>  > <mailto:gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>>
> 
>  >
> 
>  >  > Subject: [PATCH v6 3/3] .azurepipelines: Install code coverage tool
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com 
> <mailto:gua.guo@intel.com%20%3cmailto:gua.guo@intel.com>>>
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > For Windows add below tool for code coverage 1. OpenCppCoverage:
> 
>  >
> 
>  >  > parsing pdb file to generate coverage data 2. pycobertura: show up
> 
>  >
> 
>  >  > html format data for coverage data
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > For Linux add below tool for code coverage 1. lcov: parsing gcda
> 
>  > gcno
> 
>  >
> 
>  >  > file to generate coverage data 2. lcov-cobertura: convert coverage
> 
>  >
> 
>  >  > data to cobertura format 3. pycobertura: show up html format data
> 
>  > for
> 
>  >
> 
>  >  > coverage data
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > Cc: Sean Brogan <sean.brogan@microsoft.com
> 
>  > <mailto:sean.brogan@microsoft.com <mailto:sean.brogan@microsoft.com>>>
> 
>  >
> 
>  >  > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com
> 
>  > <mailto:Bret.Barkelew@microsoft.com 
> <mailto:Bret.Barkelew@microsoft.com>>>
> 
>  >
> 
>  >  > Cc: Michael D Kinney <michael.d.kinney@intel.com
> 
>  > <mailto:michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>>
> 
>  >
> 
>  >  > Cc: Liming Gao <gaoliming@byosoft.com.cn
> 
>  > <mailto:gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>>
> 
>  >
> 
>  >  > Signed-off-by: Gua Guo <gua.guo@intel.com
> 
>  > <mailto:gua.guo@intel.com <mailto:gua.guo@intel.com>>>
> 
>  >
> 
>  >  > ---
> 
>  >
> 
>  >  >  .azurepipelines/Ubuntu-GCC5.yml                 | 5 ++++-
> 
>  >
> 
>  >  >  .azurepipelines/Windows-VS2019.yml              | 5 +++++
> 
>  >
> 
>  >  >  .azurepipelines/templates/pr-gate-build-job.yml | 2 ++
> 
>  >
> 
>  >  >  .azurepipelines/templates/pr-gate-steps.yml     | 4 ++++
> 
>  >
> 
>  >  >  pip-requirements.txt                            | 2 ++
> 
>  >
> 
>  >  >  5 files changed, 17 insertions(+), 1 deletion(-)
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > diff --git a/.azurepipelines/Ubuntu-GCC5.yml
> 
>  >
> 
>  >  > b/.azurepipelines/Ubuntu-GCC5.yml index 1acd8d2a46..f83951eeaf
> 
>  > 100644
> 
>  >
> 
>  >  > --- a/.azurepipelines/Ubuntu-GCC5.yml
> 
>  >
> 
>  >  > +++ b/.azurepipelines/Ubuntu-GCC5.yml
> 
>  >
> 
>  >  > @@ -19,4 +19,7 @@ jobs:
> 
>  >
> 
>  >  >      tool_chain_tag: 'GCC5'
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >      vm_image: 'ubuntu-latest'
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >      arch_list: "IA32,X64,ARM,AARCH64,RISCV64,LOONGARCH64"
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > -
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +    extra_install_step:
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +    - bash: sudo apt-get install -y lcov
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      displayName: Install Code Coverage Tools
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      condition: and(gt(variables.pkg_count, 0), succeeded())
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > diff --git a/.azurepipelines/Windows-VS2019.yml
> 
>  >
> 
>  >  > b/.azurepipelines/Windows-VS2019.yml
> 
>  >
> 
>  >  > index e4bd4b1d22..c07e5bb434 100644
> 
>  >
> 
>  >  > --- a/.azurepipelines/Windows-VS2019.yml
> 
>  >
> 
>  >  > +++ b/.azurepipelines/Windows-VS2019.yml
> 
>  >
> 
>  >  > @@ -18,3 +18,8 @@ jobs:
> 
>  >
> 
>  >  >      tool_chain_tag: 'VS2019'
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >      vm_image: 'windows-2019'
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >      arch_list: "IA32,X64"
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +    extra_install_step:
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +    - powershell: choco install opencppcoverage; Write-Host
> 
>  > "##vso[task.prependpath]C:\Program Files\OpenCppCoverage"
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      displayName: Install Code Coverage Tool
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      condition: and(gt(variables.pkg_count, 0), succeeded())
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > diff --git a/.azurepipelines/templates/pr-gate-build-job.yml
> 
>  >
> 
>  >  > b/.azurepipelines/templates/pr-gate-build-job.yml
> 
>  >
> 
>  >  > index 7f88b41dc8..0ff1ce2952 100644
> 
>  >
> 
>  >  > --- a/.azurepipelines/templates/pr-gate-build-job.yml
> 
>  >
> 
>  >  > +++ b/.azurepipelines/templates/pr-gate-build-job.yml
> 
>  >
> 
>  >  > @@ -12,6 +12,7 @@ parameters:
> 
>  >
> 
>  >  >    tool_chain_tag: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    vm_image: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    arch_list: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +  extra_install_step: []
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  # Build step
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  jobs:
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > @@ -77,3 +78,4 @@ jobs:
> 
>  >
> 
>  >  >        build_pkgs: $(Build.Pkgs)
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >        build_targets: $(Build.Targets)
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >        build_archs: ${{ parameters.arch_list }}
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      extra_install_step: ${{ parameters.extra_install_step }}
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > diff --git a/.azurepipelines/templates/pr-gate-steps.yml
> 
>  >
> 
>  >  > b/.azurepipelines/templates/pr-gate-steps.yml
> 
>  >
> 
>  >  > index cb431e53fc..080f60aea6 100644
> 
>  >
> 
>  >  > --- a/.azurepipelines/templates/pr-gate-steps.yml
> 
>  >
> 
>  >  > +++ b/.azurepipelines/templates/pr-gate-steps.yml
> 
>  >
> 
>  >  > @@ -12,6 +12,7 @@ parameters:
> 
>  >
> 
>  >  >    build_pkgs: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    build_targets: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    build_archs: ''
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +  extra_install_step: []
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  steps:
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  - checkout: self
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > @@ -37,6 +38,8 @@ steps:
> 
>  >
> 
>  >  >    displayName: fetch target branch
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    condition: eq(variables['Build.Reason'], 'PullRequest')
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +- ${{ parameters.extra_install_step }}
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  # trim the package list if this is a PR
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  - task: CmdLine@1
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    displayName: Check if ${{ parameters.build_pkgs }} need testing
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > @@ -125,6 +128,7 @@ steps:
> 
>  >
> 
>  >  >        TestSuites.xml
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >        **/BUILD_TOOLS_REPORT.html
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >        **/OVERRIDELOG.TXT
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +      coverage.xml
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >      flattenFolders: true
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >    condition: succeededOrFailed()
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > diff --git a/pip-requirements.txt b/pip-requirements.txt index
> 
>  >
> 
>  >  > be8c7a1c37..4ffcadddd8 100644
> 
>  >
> 
>  >  > --- a/pip-requirements.txt
> 
>  >
> 
>  >  > +++ b/pip-requirements.txt
> 
>  >
> 
>  >  > @@ -16,3 +16,5 @@ edk2-pytool-library==0.12.1
> 
>  >
> 
>  >  >  edk2-pytool-extensions~=0.20.0
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  edk2-basetools==0.1.39
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  >  antlr4-python3-runtime==4.7.1
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +lcov-cobertura==2.0.2
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > +
> 
>  >
> 
>  >  >
> 
>  >
> 
>  >  > --
> 
>  >
> 
>  >  > 2.31.1.windows.1
> 
>  >
> 
>