[PATCH v1 1/1] BaseTools/Plugin: Clarify code coverage failure message

[PATCH v1 1/1] BaseTools/Plugin: Clarify code coverage failure message

[Michael Kubacki]

From: Michael Kubacki <michael.kubacki@microsoft.com>

HostBasedUnitTestRunner.py is a build plugin responsible for locating
and executing host-based unit tests.

Recently, commit 6bb00aa introduced support for the plugin to
generate code coverage reports via lcov and OpenCppCoverage.

The plugin has discovered unit tests by searching for executables
with "Test" in the name for a while. However, the test coverage
change makes assumptions about test presence when crafting the
OpenCppCoverage command that ultimately fails with an ambiguous error
message if no host-based unit tests are discovered (see "ERROR").

```
SECTION - Run Host based Unit Tests
SUBSECTION - Testing for architecture: X64
ERROR - UnitTest Coverage: Failed to generate cobertura format xml in
        single package.
PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
```

This change preempts that message with a check in the plugin to
determine if any host-based tests were discovered. If not, a message
is printed with more guidance about how the developer should proceed
to either (1) fix their tests so code coverage is generated as
expected or (2) prevent the error message.

New message:

```
SECTION - Run Host based Unit Tests
SUBSECTION - Testing for architecture: X64
WARNING - UnitTest Coverage:
  No unit tests discovered. Test coverage will not be generated.

  Prevent this message by:
  1. Adding host-based unit tests to this package
  2. Ensuring tests have the word "Test" in their name
  3. Disabling HostUnitTestCompilerPlugin in the package CI YAML file
PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
```

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
---
 BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index 0e013c5f1a7f..a384b556294c 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -16,6 +16,7 @@ import edk2toollib.windows.locate_tools as locate_tools
 from edk2toolext.environment import shell_environment
 from edk2toollib.utility_functions import RunCmd
 from edk2toollib.utility_functions import GetHostInfo
+from textwrap import dedent
 
 
 class HostBasedUnitTestRunner(IUefiBuildPlugin):
@@ -84,6 +85,18 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
             else:
                 raise NotImplementedError("Unsupported Operating System")
 
+            if not testList:
+                logging.warning(dedent("""
+                    UnitTest Coverage:
+                      No unit tests discovered. Test coverage will not be generated.
+
+                      Prevent this message by:
+                      1. Adding host-based unit tests to this package
+                      2. Ensuring tests have the word "Test" in their name
+                      3. Disabling HostUnitTestCompilerPlugin in the package CI YAML file
+                    """).strip())
+                return 0
+
             for test in testList:
                 # Configure output name if test uses cmocka.
                 shell_env.set_shell_var(
-- 
2.40.0.windows.1

回复: [PATCH v1 1/1] BaseTools/Plugin: Clarify code coverage failure message

[gaoliming]

Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>

> -----邮件原件-----
> 发件人: mikuback@linux.microsoft.com <mikuback@linux.microsoft.com>
> 发送时间: 2023年4月19日 0:11
> 收件人: devel@edk2.groups.io
> 抄送: Rebecca Cran <rebecca@bsdio.com>; Liming Gao
> <gaoliming@byosoft.com.cn>; Bob Feng <bob.c.feng@intel.com>; Yuwei Chen
> <yuwei.chen@intel.com>; Michael D Kinney <michael.d.kinney@intel.com>;
> Sean Brogan <sean.brogan@microsoft.com>
> 主题: [PATCH v1 1/1] BaseTools/Plugin: Clarify code coverage failure
message
> 
> From: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> HostBasedUnitTestRunner.py is a build plugin responsible for locating
> and executing host-based unit tests.
> 
> Recently, commit 6bb00aa introduced support for the plugin to
> generate code coverage reports via lcov and OpenCppCoverage.
> 
> The plugin has discovered unit tests by searching for executables
> with "Test" in the name for a while. However, the test coverage
> change makes assumptions about test presence when crafting the
> OpenCppCoverage command that ultimately fails with an ambiguous error
> message if no host-based unit tests are discovered (see "ERROR").
> 
> ```
> SECTION - Run Host based Unit Tests
> SUBSECTION - Testing for architecture: X64
> ERROR - UnitTest Coverage: Failed to generate cobertura format xml in
>         single package.
> PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
> ```
> 
> This change preempts that message with a check in the plugin to
> determine if any host-based tests were discovered. If not, a message
> is printed with more guidance about how the developer should proceed
> to either (1) fix their tests so code coverage is generated as
> expected or (2) prevent the error message.
> 
> New message:
> 
> ```
> SECTION - Run Host based Unit Tests
> SUBSECTION - Testing for architecture: X64
> WARNING - UnitTest Coverage:
>   No unit tests discovered. Test coverage will not be generated.
> 
>   Prevent this message by:
>   1. Adding host-based unit tests to this package
>   2. Ensuring tests have the word "Test" in their name
>   3. Disabling HostUnitTestCompilerPlugin in the package CI YAML file
> PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
> ```
> 
> Cc: Rebecca Cran <rebecca@bsdio.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
> ---
>  BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py |
> 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git
> a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index 0e013c5f1a7f..a384b556294c 100644
> ---
> a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++
> b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> @@ -16,6 +16,7 @@ import edk2toollib.windows.locate_tools as locate_tools
>  from edk2toolext.environment import shell_environment
>  from edk2toollib.utility_functions import RunCmd
>  from edk2toollib.utility_functions import GetHostInfo
> +from textwrap import dedent
> 
> 
>  class HostBasedUnitTestRunner(IUefiBuildPlugin):
> @@ -84,6 +85,18 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
>              else:
>                  raise NotImplementedError("Unsupported Operating
> System")
> 
> +            if not testList:
> +                logging.warning(dedent("""
> +                    UnitTest Coverage:
> +                      No unit tests discovered. Test coverage will not
> be generated.
> +
> +                      Prevent this message by:
> +                      1. Adding host-based unit tests to this package
> +                      2. Ensuring tests have the word "Test" in their
> name
> +                      3. Disabling HostUnitTestCompilerPlugin in the
> package CI YAML file
> +                    """).strip())
> +                return 0
> +
>              for test in testList:
>                  # Configure output name if test uses cmocka.
>                  shell_env.set_shell_var(
> --
> 2.40.0.windows.1

Re: [PATCH v1 1/1] BaseTools/Plugin: Clarify code coverage failure message

[Rebecca Cran]

Reviewed-by: Rebecca Cran <rebecca@bsdio.com>


On 4/18/23 10:11 AM, mikuback@linux.microsoft.com wrote:
> From: Michael Kubacki <michael.kubacki@microsoft.com>
>
> HostBasedUnitTestRunner.py is a build plugin responsible for locating
> and executing host-based unit tests.
>
> Recently, commit 6bb00aa introduced support for the plugin to
> generate code coverage reports via lcov and OpenCppCoverage.
>
> The plugin has discovered unit tests by searching for executables
> with "Test" in the name for a while. However, the test coverage
> change makes assumptions about test presence when crafting the
> OpenCppCoverage command that ultimately fails with an ambiguous error
> message if no host-based unit tests are discovered (see "ERROR").
>
> ```
> SECTION - Run Host based Unit Tests
> SUBSECTION - Testing for architecture: X64
> ERROR - UnitTest Coverage: Failed to generate cobertura format xml in
>          single package.
> PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
> ```
>
> This change preempts that message with a check in the plugin to
> determine if any host-based tests were discovered. If not, a message
> is printed with more guidance about how the developer should proceed
> to either (1) fix their tests so code coverage is generated as
> expected or (2) prevent the error message.
>
> New message:
>
> ```
> SECTION - Run Host based Unit Tests
> SUBSECTION - Testing for architecture: X64
> WARNING - UnitTest Coverage:
>    No unit tests discovered. Test coverage will not be generated.
>
>    Prevent this message by:
>    1. Adding host-based unit tests to this package
>    2. Ensuring tests have the word "Test" in their name
>    3. Disabling HostUnitTestCompilerPlugin in the package CI YAML file
> PROGRESS - --->Test Success: Host Unit Test Compiler Plugin NOOPT
> ```
>
> Cc: Rebecca Cran <rebecca@bsdio.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
> ---
>   BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
>
> diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index 0e013c5f1a7f..a384b556294c 100644
> --- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> @@ -16,6 +16,7 @@ import edk2toollib.windows.locate_tools as locate_tools
>   from edk2toolext.environment import shell_environment
>   from edk2toollib.utility_functions import RunCmd
>   from edk2toollib.utility_functions import GetHostInfo
> +from textwrap import dedent
>   
>   
>   class HostBasedUnitTestRunner(IUefiBuildPlugin):
> @@ -84,6 +85,18 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
>               else:
>                   raise NotImplementedError("Unsupported Operating System")
>   
> +            if not testList:
> +                logging.warning(dedent("""
> +                    UnitTest Coverage:
> +                      No unit tests discovered. Test coverage will not be generated.
> +
> +                      Prevent this message by:
> +                      1. Adding host-based unit tests to this package
> +                      2. Ensuring tests have the word "Test" in their name
> +                      3. Disabling HostUnitTestCompilerPlugin in the package CI YAML file
> +                    """).strip())
> +                return 0
> +
>               for test in testList:
>                   # Configure output name if test uses cmocka.
>                   shell_env.set_shell_var(