Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Guo, Gua]

[-- Attachment #1.1: Type: text/plain, Size: 2983 bytes --]

@Kinney, Michael D<mailto:michael.d.kinney@intel.com> and @Michael Kubacki<mailto:mikuback@linux.microsoft.com> everything fix now.



Leverage ReportGenerator https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator to create code coverage pipeline.



PR: https://github.com/tianocore/edk2/pull/3349



"Job2 (Build Agent : windows-2019)" must require for Windows build agent because ReportGenerator is "Windows dot-net" specific. So I add some dependency to let it support for both GCC5 and VS2019.





  *   Windows-VS2019 flow
     *   Summary URL: https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75021&view=results

[cid:image003.png@01D91EE2.B62A06B0]

     *   CI/CD Flow:

[cid:image001.png@01D91EE0.6BDEEE60]







  *   Ubuntu-GCC5 flow
     *   Summary URL: https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75020&view=results

[cid:image004.png@01D91EE4.4A236270]

     *   CI/CD Flow:

[cid:image002.png@01D91EE0.D7B262C0]







Thanks,

Gua

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Guo, Gua
Sent: Monday, January 2, 2023 7:25 PM
To: devel@edk2.groups.io
Cc: Guo, Gua <gua.guo@intel.com>
Subject: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC



From: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>



V1: Add coverage option for GCC

V2: Add ReadMe.md for how to generate coverage report

V3: Add VS2019 and GCC code coverage support

V4: Add VS2019 and GCC Azure CI/CD support

V5: Fix some typo and some flow issue

V6: Remove html coverage information

  - Due to python 3.11 install lxml will be failure,

  pycobertura need it to convert cobertura format to

  html file.

  - Add section for developer how to use OpenCppCoverage

  on IDE Visual Studio

V7: Remove redundant code and add code coverage pipeline support

  - Remove redundant code on HostBasedUnitTestRunner.py

  - Unify coding rule on HostBasedUnitTestRunner.py

  - Add CodeCoverage Azure pipeline support for GCC5 and VS2019



Gua Guo (3):

  UnitTestFrameworkPkg: Add code coverage support for GCC

  BaseTools/Plugin: Add coverage support for Unit Test

  .azurepipelines: Install code coverage tool



.azurepipelines/Ubuntu-GCC5.yml               |   5 +-

.azurepipelines/Windows-VS2019.yml            |   5 +

.../templates/pr-gate-build-job.yml           |  36 +++++++

.azurepipelines/templates/pr-gate-steps.yml   |   4 +

.../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-

UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++

.../UnitTestFrameworkPkg.ci.yaml              |   1 +

.../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-

pip-requirements.txt                          |   2 +

9 files changed, 195 insertions(+), 3 deletions(-)



--

2.31.1.windows.1













[-- Attachment #1.2: Type: text/html, Size: 12590 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 35196 bytes --]

[-- Attachment #3: image002.png --]
[-- Type: image/png, Size: 35672 bytes --]

[-- Attachment #4: image003.png --]
[-- Type: image/png, Size: 49263 bytes --]

[-- Attachment #5: image004.png --]
[-- Type: image/png, Size: 54363 bytes --]

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Michael D Kinney]

[-- Attachment #1.1: Type: text/plain, Size: 3754 bytes --]

Hi Gua,

Thanks!!!  The results for both VS and GCC builds in Azure look really good.

It even visually shows some branches and APIs calls missing coverage in the unit tests for the SafeIntLib.

https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75036&view=codecoverage-tab

Search for “SafeIntLib”

Mike

From: Guo, Gua <gua.guo@intel.com>
Sent: Monday, January 2, 2023 3:57 AM
To: devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki <mikuback@linux.microsoft.com>
Subject: RE: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC


@Kinney, Michael D<mailto:michael.d.kinney@intel.com> and @Michael Kubacki<mailto:mikuback@linux.microsoft.com> everything fix now.



Leverage ReportGenerator https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator to create code coverage pipeline.



PR: https://github.com/tianocore/edk2/pull/3349



“Job2 (Build Agent : windows-2019)” must require for Windows build agent because ReportGenerator is “Windows dot-net” specific. So I add some dependency to let it support for both GCC5 and VS2019.





  *   Windows-VS2019 flow

     *   Summary URL: https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75021&view=results

[cid:image003.png@01D91E91.3AB4F810]

     *   CI/CD Flow:

[cid:image005.png@01D91E91.3AB4F810]







  *   Ubuntu-GCC5 flow

     *   Summary URL: https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75020&view=results

[cid:image006.png@01D91E91.3AB4F810]

     *   CI/CD Flow:

[cid:image007.png@01D91E91.3AB4F810]







Thanks,

Gua

-----Original Message-----
From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Guo, Gua
Sent: Monday, January 2, 2023 7:25 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Cc: Guo, Gua <gua.guo@intel.com<mailto:gua.guo@intel.com>>
Subject: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC



From: Gua Guo <gua.guo@intel.com<mailto:gua.guo@intel.com>>



V1: Add coverage option for GCC

V2: Add ReadMe.md for how to generate coverage report

V3: Add VS2019 and GCC code coverage support

V4: Add VS2019 and GCC Azure CI/CD support

V5: Fix some typo and some flow issue

V6: Remove html coverage information

  - Due to python 3.11 install lxml will be failure,

  pycobertura need it to convert cobertura format to

  html file.

  - Add section for developer how to use OpenCppCoverage

  on IDE Visual Studio

V7: Remove redundant code and add code coverage pipeline support

  - Remove redundant code on HostBasedUnitTestRunner.py

  - Unify coding rule on HostBasedUnitTestRunner.py

  - Add CodeCoverage Azure pipeline support for GCC5 and VS2019



Gua Guo (3):

  UnitTestFrameworkPkg: Add code coverage support for GCC

  BaseTools/Plugin: Add coverage support for Unit Test

  .azurepipelines: Install code coverage tool



.azurepipelines/Ubuntu-GCC5.yml               |   5 +-

.azurepipelines/Windows-VS2019.yml            |   5 +

.../templates/pr-gate-build-job.yml           |  36 +++++++

.azurepipelines/templates/pr-gate-steps.yml   |   4 +

.../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-

UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++

.../UnitTestFrameworkPkg.ci.yaml              |   1 +

.../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-

pip-requirements.txt                          |   2 +

9 files changed, 195 insertions(+), 3 deletions(-)



--

2.31.1.windows.1













[-- Attachment #1.2: Type: text/html, Size: 61581 bytes --]

[-- Attachment #2: image003.png --]
[-- Type: image/png, Size: 49263 bytes --]

[-- Attachment #3: image005.png --]
[-- Type: image/png, Size: 35196 bytes --]

[-- Attachment #4: image006.png --]
[-- Type: image/png, Size: 54363 bytes --]

[-- Attachment #5: image007.png --]
[-- Type: image/png, Size: 35672 bytes --]

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Michael D Kinney]

Series Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Guo, Gua
> Sent: Monday, January 2, 2023 3:25 AM
> To: devel@edk2.groups.io
> Cc: Guo, Gua <gua.guo@intel.com>
> Subject: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC
> 
> From: Gua Guo <gua.guo@intel.com>
> 
> V1: Add coverage option for GCC
> V2: Add ReadMe.md for how to generate coverage report
> V3: Add VS2019 and GCC code coverage support
> V4: Add VS2019 and GCC Azure CI/CD support
> V5: Fix some typo and some flow issue
> V6: Remove html coverage information
>   - Due to python 3.11 install lxml will be failure,
>   pycobertura need it to convert cobertura format to
>   html file.
>   - Add section for developer how to use OpenCppCoverage
>   on IDE Visual Studio
> V7: Remove redundant code and add code coverage pipeline support
>   - Remove redundant code on HostBasedUnitTestRunner.py
>   - Unify coding rule on HostBasedUnitTestRunner.py
>   - Add CodeCoverage Azure pipeline support for GCC5 and VS2019
> 
> Gua Guo (3):
>   UnitTestFrameworkPkg: Add code coverage support for GCC
>   BaseTools/Plugin: Add coverage support for Unit Test
>   .azurepipelines: Install code coverage tool
> 
>  .azurepipelines/Ubuntu-GCC5.yml               |   5 +-
>  .azurepipelines/Windows-VS2019.yml            |   5 +
>  .../templates/pr-gate-build-job.yml           |  36 +++++++
>  .azurepipelines/templates/pr-gate-steps.yml   |   4 +
>  .../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-
>  UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++
>  .../UnitTestFrameworkPkg.ci.yaml              |   1 +
>  .../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-
>  pip-requirements.txt                          |   2 +
>  9 files changed, 195 insertions(+), 3 deletions(-)
> 
> --
> 2.31.1.windows.1
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Michael Kubacki]

I agree, the results look great.

On 1/2/2023 1:01 PM, Michael D Kinney wrote:
> Hi Gua,
> 
> Thanks!!!The results for both VS and GCC builds in Azure look really good.
> 
> It even visually shows some branches and APIs calls missing coverage in 
> the unit tests for the SafeIntLib.
> 
> https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75036&view=codecoverage-tab 
> <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75036&view=codecoverage-tab>
> 
> Search for “SafeIntLib”
> 
> Mike
> 
> *From:*Guo, Gua <gua.guo@intel.com>
> *Sent:* Monday, January 2, 2023 3:57 AM
> *To:* devel@edk2.groups.io; Guo, Gua <gua.guo@intel.com>; Kinney, 
> Michael D <michael.d.kinney@intel.com>; Michael Kubacki 
> <mikuback@linux.microsoft.com>
> *Subject:* RE: [edk2-devel] [PATCH v8 0/3] Add code coverage support for 
> GCC/MSVC
> 
> @Kinney, Michael D <mailto:michael.d.kinney@intel.com> and @Michael 
> Kubacki <mailto:mikuback@linux.microsoft.com> everything fix now.
> 
> Leverage ReportGenerator 
> https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator 
> <https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator> 
> to create code coverage pipeline.
> 
> PR: https://github.com/tianocore/edk2/pull/3349 
> <https://github.com/tianocore/edk2/pull/3349>
> 
> “Job2 (Build Agent : windows-2019)” must require for Windows build agent 
> because ReportGenerator is “Windows dot-net” specific. So I add some 
> dependency to let it support for both GCC5 and VS2019.
> 
>   * Windows-VS2019 flow
> 
>       o Summary URL:
>         https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75021&view=results
>         <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75021&view=results>
> 
>       o CI/CD Flow: 
> 
>   * Ubuntu-GCC5 flow
> 
>       o Summary URL:
>         https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75020&view=results
>         <https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=75020&view=results>
> 
>       o CI/CD Flow: 
> 
> Thanks,
> 
> Gua
> 
> -----Original Message-----
> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Guo, Gua
> Sent: Monday, January 2, 2023 7:25 PM
> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>
> Cc: Guo, Gua <gua.guo@intel.com <mailto:gua.guo@intel.com>>
> Subject: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC
> 
> From: Gua Guo <gua.guo@intel.com <mailto:gua.guo@intel.com>>
> 
> V1: Add coverage option for GCC
> 
> V2: Add ReadMe.md for how to generate coverage report
> 
> V3: Add VS2019 and GCC code coverage support
> 
> V4: Add VS2019 and GCC Azure CI/CD support
> 
> V5: Fix some typo and some flow issue
> 
> V6: Remove html coverage information
> 
>    - Due to python 3.11 install lxml will be failure,
> 
>    pycobertura need it to convert cobertura format to
> 
>    html file.
> 
>    - Add section for developer how to use OpenCppCoverage
> 
>    on IDE Visual Studio
> 
> V7: Remove redundant code and add code coverage pipeline support
> 
>    - Remove redundant code on HostBasedUnitTestRunner.py
> 
>    - Unify coding rule on HostBasedUnitTestRunner.py
> 
>    - Add CodeCoverage Azure pipeline support for GCC5 and VS2019
> 
> Gua Guo (3):
> 
>    UnitTestFrameworkPkg: Add code coverage support for GCC
> 
>    BaseTools/Plugin: Add coverage support for Unit Test
> 
>    .azurepipelines: Install code coverage tool
> 
> .azurepipelines/Ubuntu-GCC5.yml               |   5 +-
> 
> .azurepipelines/Windows-VS2019.yml            |   5 +
> 
> .../templates/pr-gate-build-job.yml           |  36 +++++++
> 
> .azurepipelines/templates/pr-gate-steps.yml   |   4 +
> 
> .../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-
> 
> UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++
> 
> .../UnitTestFrameworkPkg.ci.yaml              |   1 +
> 
> .../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-
> 
> pip-requirements.txt                          |   2 +
> 
> 9 files changed, 195 insertions(+), 3 deletions(-)
> 
> -- 
> 
> 2.31.1.windows.1
> 
> 

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Michael Kubacki]

For the series:
Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/2/2023 6:24 AM, Guo, Gua wrote:
> From: Gua Guo <gua.guo@intel.com>
> 
> V1: Add coverage option for GCC
> V2: Add ReadMe.md for how to generate coverage report
> V3: Add VS2019 and GCC code coverage support
> V4: Add VS2019 and GCC Azure CI/CD support
> V5: Fix some typo and some flow issue
> V6: Remove html coverage information
>    - Due to python 3.11 install lxml will be failure,
>    pycobertura need it to convert cobertura format to
>    html file.
>    - Add section for developer how to use OpenCppCoverage
>    on IDE Visual Studio
> V7: Remove redundant code and add code coverage pipeline support
>    - Remove redundant code on HostBasedUnitTestRunner.py
>    - Unify coding rule on HostBasedUnitTestRunner.py
>    - Add CodeCoverage Azure pipeline support for GCC5 and VS2019
> 
> Gua Guo (3):
>    UnitTestFrameworkPkg: Add code coverage support for GCC
>    BaseTools/Plugin: Add coverage support for Unit Test
>    .azurepipelines: Install code coverage tool
> 
>   .azurepipelines/Ubuntu-GCC5.yml               |   5 +-
>   .azurepipelines/Windows-VS2019.yml            |   5 +
>   .../templates/pr-gate-build-job.yml           |  36 +++++++
>   .azurepipelines/templates/pr-gate-steps.yml   |   4 +
>   .../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-
>   UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++
>   .../UnitTestFrameworkPkg.ci.yaml              |   1 +
>   .../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-
>   pip-requirements.txt                          |   2 +
>   9 files changed, 195 insertions(+), 3 deletions(-)
> 

Re: [edk2-devel] [PATCH v8 0/3] Add code coverage support for GCC/MSVC

[Sean]

Looks like a great first step for code coverage collection.

The container support patch series will need to be updated (since it 
hasn't yet been merged) and it will need to account for the new 
requirements for code coverage.

In the future i would like to see the local experience improved (HTML 
reports generated) and metrics tracked per package instead of build 
wide.  These topics can be discussed at the Monday tools meeting but I 
think this series as is should be merged now to start collecting this data.

Thanks for all your efforts.


Series Reviewed-by: Sean Brogan<sean.brogan@microsoft.com>



On 1/2/2023 3:24 AM, Guo, Gua wrote:
> From: Gua Guo <gua.guo@intel.com>
>
> V1: Add coverage option for GCC
> V2: Add ReadMe.md for how to generate coverage report
> V3: Add VS2019 and GCC code coverage support
> V4: Add VS2019 and GCC Azure CI/CD support
> V5: Fix some typo and some flow issue
> V6: Remove html coverage information
>    - Due to python 3.11 install lxml will be failure,
>    pycobertura need it to convert cobertura format to
>    html file.
>    - Add section for developer how to use OpenCppCoverage
>    on IDE Visual Studio
> V7: Remove redundant code and add code coverage pipeline support
>    - Remove redundant code on HostBasedUnitTestRunner.py
>    - Unify coding rule on HostBasedUnitTestRunner.py
>    - Add CodeCoverage Azure pipeline support for GCC5 and VS2019
>
> Gua Guo (3):
>    UnitTestFrameworkPkg: Add code coverage support for GCC
>    BaseTools/Plugin: Add coverage support for Unit Test
>    .azurepipelines: Install code coverage tool
>
>   .azurepipelines/Ubuntu-GCC5.yml               |   5 +-
>   .azurepipelines/Windows-VS2019.yml            |   5 +
>   .../templates/pr-gate-build-job.yml           |  36 +++++++
>   .azurepipelines/templates/pr-gate-steps.yml   |   4 +
>   .../HostBasedUnitTestRunner.py                | 101 +++++++++++++++++-
>   UnitTestFrameworkPkg/ReadMe.md                |  41 +++++++
>   .../UnitTestFrameworkPkg.ci.yaml              |   1 +
>   .../UnitTestFrameworkPkgHost.dsc.inc          |   3 +-
>   pip-requirements.txt                          |   2 +
>   9 files changed, 195 insertions(+), 3 deletions(-)
>