[Patch 1/1] BaseTools: Resolve a issue of Incremental build

[Patch 1/1] BaseTools: Resolve a issue of Incremental build

[Bob Feng]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311

In patch set 13c5e34a - 0c3e8e99, we implemented incremental build with
using compiler/pre-processor generate dependent header file function.

A issue is found for MSVC compiler, that the cl.exe /showIncludes
build option generate header file list to either stdout or stderr.
For .c file, the header file list is print out to stdout while for
.vfr, .aslc and .nasm file, the file list is print out to stderr.

The build tool use two threads to process the message from stdout and
stderr, but to generate correct *.deps file, build tool need to 
combine the header file list from stderr and other messages from stdout
together with correct time sequence order.

So this patch is trying to combine the stdout and stderr together for
the process which is for calling make program.

The impact of this patch is that the output message of build with -q
will be changed. The compiler error message will not print out.
The build behavior of other log level setting will not be impacted.


Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
---
 BaseTools/Source/Python/build/build.py | 11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)

diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index 8a8e32e496f8..5263c54e5ae0 100755
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -22,11 +22,11 @@ import time
 import platform
 import traceback
 import multiprocessing
 from threading import Thread,Event,BoundedSemaphore
 import threading
-from subprocess import Popen,PIPE
+from subprocess import Popen,PIPE, STDOUT
 from collections import OrderedDict, defaultdict
 from Common.buildoptions import BuildOption,BuildTarget
 from AutoGen.PlatformAutoGen import PlatformAutoGen
 from AutoGen.ModuleAutoGen import ModuleAutoGen
 from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen
@@ -227,26 +227,21 @@ def LaunchCommand(Command, WorkingDir,ModuleAuto = None):
 
     Proc = None
     EndOfProcedure = None
     try:
         # launch the command
-        Proc = MakeSubProc(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)
+        Proc = MakeSubProc(Command, stdout=PIPE, stderr=STDOUT, env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)
 
         # launch two threads to read the STDOUT and STDERR
         EndOfProcedure = Event()
         EndOfProcedure.clear()
         if Proc.stdout:
             StdOutThread = Thread(target=ReadMessage, args=(Proc.stdout, EdkLogger.info, EndOfProcedure,Proc.ProcOut))
             StdOutThread.setName("STDOUT-Redirector")
             StdOutThread.setDaemon(False)
             StdOutThread.start()
 
-        if Proc.stderr:
-            StdErrThread = Thread(target=ReadMessage, args=(Proc.stderr, EdkLogger.quiet, EndOfProcedure,Proc.ProcOut))
-            StdErrThread.setName("STDERR-Redirector")
-            StdErrThread.setDaemon(False)
-            StdErrThread.start()
 
         # waiting for program exit
         Proc.wait()
     except: # in case of aborting
         # terminate the threads redirecting the program output
@@ -258,12 +253,10 @@ def LaunchCommand(Command, WorkingDir,ModuleAuto = None):
                 Command = " ".join(Command)
             EdkLogger.error("build", COMMAND_FAILURE, "Failed to start command", ExtraData="%s [%s]" % (Command, WorkingDir))
 
     if Proc.stdout:
         StdOutThread.join()
-    if Proc.stderr:
-        StdErrThread.join()
 
     # check the return code of the program
     if Proc.returncode != 0:
         if not isinstance(Command, type("")):
             Command = " ".join(Command)
-- 
2.20.1.windows.1

Re: [Patch 1/1] BaseTools: Resolve a issue of Incremental build

[Liming Gao]

Reviewed-by: Liming Gao <liming.gao@intel.com>

>-----Original Message-----
>From: Feng, Bob C
>Sent: Tuesday, December 17, 2019 9:55 AM
>To: devel@edk2.groups.io
>Cc: Gao, Liming <liming.gao@intel.com>; Shi, Steven <steven.shi@intel.com>;
>Kinney, Michael D <michael.d.kinney@intel.com>
>Subject: [Patch 1/1] BaseTools: Resolve a issue of Incremental build
>
>BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2311
>
>In patch set 13c5e34a - 0c3e8e99, we implemented incremental build with
>using compiler/pre-processor generate dependent header file function.
>
>A issue is found for MSVC compiler, that the cl.exe /showIncludes
>build option generate header file list to either stdout or stderr.
>For .c file, the header file list is print out to stdout while for
>.vfr, .aslc and .nasm file, the file list is print out to stderr.
>
>The build tool use two threads to process the message from stdout and
>stderr, but to generate correct *.deps file, build tool need to
>combine the header file list from stderr and other messages from stdout
>together with correct time sequence order.
>
>So this patch is trying to combine the stdout and stderr together for
>the process which is for calling make program.
>
>The impact of this patch is that the output message of build with -q
>will be changed. The compiler error message will not print out.
>The build behavior of other log level setting will not be impacted.
>
>
>Cc: Liming Gao <liming.gao@intel.com>
>Cc: Steven Shi <steven.shi@intel.com>
>Cc: Michael D Kinney <michael.d.kinney@intel.com>
>Signed-off-by: Bob Feng <bob.c.feng@intel.com>
>---
> BaseTools/Source/Python/build/build.py | 11 ++---------
> 1 file changed, 2 insertions(+), 9 deletions(-)
>
>diff --git a/BaseTools/Source/Python/build/build.py
>b/BaseTools/Source/Python/build/build.py
>index 8a8e32e496f8..5263c54e5ae0 100755
>--- a/BaseTools/Source/Python/build/build.py
>+++ b/BaseTools/Source/Python/build/build.py
>@@ -22,11 +22,11 @@ import time
> import platform
> import traceback
> import multiprocessing
> from threading import Thread,Event,BoundedSemaphore
> import threading
>-from subprocess import Popen,PIPE
>+from subprocess import Popen,PIPE, STDOUT
> from collections import OrderedDict, defaultdict
> from Common.buildoptions import BuildOption,BuildTarget
> from AutoGen.PlatformAutoGen import PlatformAutoGen
> from AutoGen.ModuleAutoGen import ModuleAutoGen
> from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen
>@@ -227,26 +227,21 @@ def LaunchCommand(Command,
>WorkingDir,ModuleAuto = None):
>
>     Proc = None
>     EndOfProcedure = None
>     try:
>         # launch the command
>-        Proc = MakeSubProc(Command, stdout=PIPE, stderr=PIPE,
>env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)
>+        Proc = MakeSubProc(Command, stdout=PIPE, stderr=STDOUT,
>env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)
>
>         # launch two threads to read the STDOUT and STDERR
>         EndOfProcedure = Event()
>         EndOfProcedure.clear()
>         if Proc.stdout:
>             StdOutThread = Thread(target=ReadMessage, args=(Proc.stdout,
>EdkLogger.info, EndOfProcedure,Proc.ProcOut))
>             StdOutThread.setName("STDOUT-Redirector")
>             StdOutThread.setDaemon(False)
>             StdOutThread.start()
>
>-        if Proc.stderr:
>-            StdErrThread = Thread(target=ReadMessage, args=(Proc.stderr,
>EdkLogger.quiet, EndOfProcedure,Proc.ProcOut))
>-            StdErrThread.setName("STDERR-Redirector")
>-            StdErrThread.setDaemon(False)
>-            StdErrThread.start()
>
>         # waiting for program exit
>         Proc.wait()
>     except: # in case of aborting
>         # terminate the threads redirecting the program output
>@@ -258,12 +253,10 @@ def LaunchCommand(Command,
>WorkingDir,ModuleAuto = None):
>                 Command = " ".join(Command)
>             EdkLogger.error("build", COMMAND_FAILURE, "Failed to start
>command", ExtraData="%s [%s]" % (Command, WorkingDir))
>
>     if Proc.stdout:
>         StdOutThread.join()
>-    if Proc.stderr:
>-        StdErrThread.join()
>
>     # check the return code of the program
>     if Proc.returncode != 0:
>         if not isinstance(Command, type("")):
>             Command = " ".join(Command)
>--
>2.20.1.windows.1