[PATCH v5 0/1] BaseTools/Plugin: Too many execute files cause "cmd too long" failure

[PATCH v5 0/1] BaseTools/Plugin: Too many execute files cause "cmd too long" failure

[Guo, Gua]

From: Gua Guo <gua.guo@intel.com>

V4:
Add Reviewed-by Info from Michael Kubacki
Add more people into Cc List
> BaseTools
> F: BaseTools/
> M: Rebecca Cran <rebecca@bsdio.com> [bcran]
> M: Liming Gao <gaoliming@byosoft.com.cn> [lgao4]
> R: Bob Feng <bob.c.feng@intel.com> [BobCF]
> R: Yuwei Chen <yuwei.chen@intel.com> [YuweiChen1110]

V3:
Michael Kubacki
Open1:
I understand it was common in some parts of the code base at one point to wrap all expressions in parentheses, but can it be avoided in newly added code?
And the parentheses are unnecessary in this case and don't follow typical Python style.

Solution: Fixed, I will record it in my personal guideline to prevent happen again in next time that I change python code on Edk2.

Open2:
Due to the number of references, it would be easier to follow if this path were assigned to a variable:
os.path.join(buildOutputBase, 'coverage.cov')

Solution: Fixed, Use variable to reduce repetitive data.

Open3:
Solution: my local test, performance time increase 1.4% each build. I'm not sure whether the data is reasonable or not. But if won't fix it, I think one day unittest implement more and more on Edk2 part.
It will break azurepipe line build in the future.

V2/V1:
Mike Kinney:
Open1: Make commit message more clearly.
Solution: Change commit message to use real case to describe the issue I encounter.

Gua Guo (1):
  BaseTools/Plugin: Too many execute files cause "cmd too long" failure

 .../HostBasedUnitTestRunner.py                | 46 +++++++++++++++++--
 1 file changed, 41 insertions(+), 5 deletions(-)

--
2.39.2.windows.1

[PATCH v5 1/1] BaseTools/Plugin: Too many execute files cause "cmd too long" failure

[Guo, Gua]

From: Gua Guo <gua.guo@intel.com>

Windows command prompt have 8191 characters limitation,
enhance it to make command too long can be resloved.

Provide an example, if have too many cov files, it causes to run single
command over the 8191 characters limitation.
> OpenCppCoverage
>  --export_type binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=AAA.cov
>  ...
>  --input_coverage=NNN.cov

The solution is passing many coverage files in single command line to
breaking it up into many command lines with one coverage file per
command line in order to prevent single line is over to 8191 characters.

- Command Line 1
> OpenCppCoverage
>  --export_type binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=AAA.cov
>  --input_coverage=coverage.cov
...

- Command Line N
> OpenCppCoverage
>  --export_type. binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=NNN.cov
>  --input_coverage=coverage.cov

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Michael Kubacki <mikuback@linux.microsoft.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
Reviewed-by: Michael Kubacki <mikuback@linux.microsoft.com>
---
 .../HostBasedUnitTestRunner.py                | 46 +++++++++++++++++--
 1 file changed, 41 insertions(+), 5 deletions(-)

diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index d993de9412..2e5c462cd2 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -205,28 +205,64 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
         testList = glob.glob(os.path.join(buildOutputBase, "**","*Test*.exe"), recursive=True)
         workspace = thebuilder.env.GetValue("WORKSPACE")
         workspace = (workspace + os.sep) if workspace[-1] != os.sep else workspace
+        workspaceBuild = os.path.join(workspace, 'Build')
         # Generate coverage file
         coverageFile = ""
         for testFile in testList:
             ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
-            coverageFile += " --input_coverage=" + testFile + ".cov"
+            if ret != 0:
+                logging.error("UnitTest Coverage: Failed to collect coverage data.")
+                return 1
+
+            coverageFile  = f" --input_coverage={testFile}.cov"
+            totalCoverageFile = os.path.join(buildOutputBase, 'coverage.cov')
+            if os.path.isfile(totalCoverageFile):
+                coverageFile += f" --input_coverage={totalCoverageFile}"
+            ret = RunCmd(
+                "OpenCppCoverage",
+                f"--export_type binary:{totalCoverageFile} " +
+                f"--working_dir={workspaceBuild} " +
+                f"{coverageFile}"
+                )
             if ret != 0:
                 logging.error("UnitTest Coverage: Failed to collect coverage data.")
                 return 1
 
         # Generate and XML file if requested.by each package
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} --working_dir={workspace}Build {coverageFile}")
+        ret = RunCmd(
+            "OpenCppCoverage",
+            f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} " +
+            f"--working_dir={workspaceBuild} " +
+            f"--input_coverage={totalCoverageFile} "
+            )
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
             return 1
 
         # Generate total report XML file for all package
-        testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
+        testCoverageList = glob.glob(os.path.join(workspace, "Build", "**", "*Test*.exe.cov"), recursive=True)
         coverageFile = ""
+        totalCoverageFile = os.path.join(workspaceBuild, 'coverage.cov')
         for testCoverage in testCoverageList:
-            coverageFile += " --input_coverage=" + testCoverage
+            coverageFile  = f" --input_coverage={testCoverage}"
+            if os.path.isfile(totalCoverageFile):
+                coverageFile += f" --input_coverage={totalCoverageFile}"
+            ret = RunCmd(
+                "OpenCppCoverage",
+                f"--export_type binary:{totalCoverageFile} " +
+                f"--working_dir={workspaceBuild} " +
+                f"{coverageFile}"
+                )
+            if ret != 0:
+                logging.error("UnitTest Coverage: Failed to collect coverage data.")
+                return 1
 
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
+        ret = RunCmd(
+            "OpenCppCoverage",
+            f"--export_type cobertura:{os.path.join(workspaceBuild, 'coverage.xml')} " +
+            f"--working_dir={workspaceBuild} " +
+            f"--input_coverage={totalCoverageFile}"
+            )
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
             return 1
-- 
2.39.2.windows.1