From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.115; helo=mga14.intel.com; envelope-from=jaben.carsey@intel.com; receiver=edk2-devel@lists.01.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 80E9C210FC0A0 for ; Tue, 28 Aug 2018 09:12:43 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 Aug 2018 09:12:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,300,1531810800"; d="scan'208";a="66642258" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga008.fm.intel.com with ESMTP; 28 Aug 2018 09:09:59 -0700 Received: from fmsmsx155.amr.corp.intel.com (10.18.116.71) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 28 Aug 2018 09:09:59 -0700 Received: from fmsmsx103.amr.corp.intel.com ([169.254.2.126]) by FMSMSX155.amr.corp.intel.com ([169.254.5.29]) with mapi id 14.03.0319.002; Tue, 28 Aug 2018 09:09:58 -0700 From: "Carsey, Jaben" To: "Gao, Liming" CC: "edk2-devel@lists.01.org" , "Guo, Dongao" Thread-Topic: [edk2] [Patch] BaseTools: Support multi thread build Basetool on Windows Thread-Index: AQHUPuPJkHJEvg6GqUqOxs4KcdBX5aTVVVtv Date: Tue, 28 Aug 2018 16:09:58 +0000 Message-ID: References: <20180828152656.17596-1-liming.gao@intel.com> In-Reply-To: <20180828152656.17596-1-liming.gao@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Support multi thread build Basetool on Windows X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 16:12:43 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable This looks like a great change. Why not delete the bat file? Reviewed-by Jaben Carsey: Jaben > On Aug 28, 2018, at 8:28 AM, Liming Gao wrote: >=20 > From: Dongao Guo >=20 > Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This scri= pt will > invoke nmake in multi thread mode. It can save more than half time of Bas= eTools > C clean build. > GCC make supports multiple thread in make phase. So, GNNmakefile doesn't = need apply > this script. >=20 > single task or job=3D1: > just single thread and invoke subprocess,subprocess will use > system.stdout to print output. > multi task: > thread number is logic cpu count.All subprocess output will pass to > python script by PIPE and then script print it to system.stdout. >=20 > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Dongao Guo > Reviewed-by: Liming Gao > Test-by: Liming Gao > --- > BaseTools/Makefile | 12 +- > BaseTools/Source/C/Makefile | 14 +-- > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 169 ++++++++++++++++++++++= +++++ > 3 files changed, 182 insertions(+), 13 deletions(-) > create mode 100644 BaseTools/Source/C/Makefiles/NmakeSubdirs.py >=20 > diff --git a/BaseTools/Makefile b/BaseTools/Makefile > index 3736d85..b98cd85 100644 > --- a/BaseTools/Makefile > +++ b/BaseTools/Makefile > @@ -1,7 +1,7 @@ > ## @file > # Windows makefile for Base Tools project build. > # > -# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved.
> +# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.
> # This program and the accompanying materials > # are licensed and made available under the terms and conditions of the B= SD License > # which accompanies this distribution. The full text of the license may = be found at > @@ -20,19 +20,19 @@ SUBDIRS =3D $(BASE_TOOLS_PATH)\Source\C $(BASE_TOOLS_= PATH)\Source\Python > all: c python >=20 > c : > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOO= LS_PATH)\Source\C > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nmake= Subdirs.py all $(BASE_TOOLS_PATH)\Source\C >=20 > python: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_TOO= LS_PATH)\Source\Python > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nmake= Subdirs.py all $(BASE_TOOLS_PATH)\Source\Python >=20 > subdirs: $(SUBDIRS) > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $** > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nmake= Subdirs.py all $** >=20 > .PHONY: clean > clean: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBDIR= S) > + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeS= ubdirs.py clean $(SUBDIRS) >=20 > .PHONY: cleanall > cleanall: > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall $(SUB= DIRS) > + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeS= ubdirs.py cleanall $(SUBDIRS) >=20 > diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile > index 5428180..1246d23 100644 > --- a/BaseTools/Source/C/Makefile > +++ b/BaseTools/Source/C/Makefile > @@ -1,7 +1,7 @@ > ## @file > # Windows makefile for C tools build. > # > -# Copyright (c) 2009 - 2017, Intel Corporation. All rights reserved.
> +# Copyright (c) 2009 - 2018, Intel Corporation. All rights reserved.
> # This program and the accompanying materials > # are licensed and made available under the terms and conditions of the B= SD License > # which accompanies this distribution. The full text of the license may = be found at > @@ -16,7 +16,7 @@ HOST_ARCH =3D IA32 >=20 > LIBRARIES =3D Common > APPLICATIONS =3D \ > - BootSectImage \ > + VfrCompile \ > BrotliCompress \ > EfiLdrImage \ > EfiRom \ > @@ -32,7 +32,7 @@ APPLICATIONS =3D \ > Split \ > TianoCompress \ > VolInfo \ > - VfrCompile \ > + BootSectImage \ > DevicePath >=20 > all: libs apps install > @@ -43,7 +43,7 @@ libs: $(LIBRARIES) > @echo # Build libraries > @echo ###################### > @if not exist $(LIB_PATH) mkdir $(LIB_PATH) > - @Makefiles\NmakeSubdirs.bat all $** > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** >=20 > apps: $(APPLICATIONS) > @echo. > @@ -51,7 +51,7 @@ apps: $(APPLICATIONS) > @echo # Build executables > @echo ###################### > @if not exist $(BIN_PATH) mkdir $(BIN_PATH) > - @Makefiles\NmakeSubdirs.bat all $** > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** >=20 > install: $(LIB_PATH) $(BIN_PATH) > @echo. > @@ -65,11 +65,11 @@ install: $(LIB_PATH) $(BIN_PATH) >=20 > .PHONY: clean > clean: > - @Makefiles\NmakeSubdirs.bat clean $(LIBRARIES) $(APPLICATIONS) > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py clean $(LIBRARIES= ) $(APPLICATIONS) >=20 > .PHONY: cleanall > cleanall: > - @Makefiles\NmakeSubdirs.bat cleanall $(LIBRARIES) $(APPLICATIONS) > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py cleanall $(LIBRAR= IES) $(APPLICATIONS) >=20 > !INCLUDE Makefiles\ms.rule >=20 > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py b/BaseTools/Sou= rce/C/Makefiles/NmakeSubdirs.py > new file mode 100644 > index 0000000..29bb5df > --- /dev/null > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > @@ -0,0 +1,169 @@ > +# @file NmakeSubdirs.py > +# This script support parallel build for nmake in windows environment. > +# It supports Python2.x and Python3.x both. > +# > +# Copyright (c) 2018, Intel Corporation. All rights reserved.
> +# > +# This program and the accompanying materials > +# are licensed and made available under the terms and conditions of the= BSD License > +# which accompanies this distribution. The full text of the license ma= y be found at > +# http://opensource.org/licenses/bsd-license.php > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, > +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR = IMPLIED. > +# > + > +# > +# Import Modules > +# > + > +from __future__ import print_function > +import argparse > +import threading > +import time > +import os > +import subprocess > +import multiprocessing > +import copy > +import sys > +__prog__ =3D 'NmakeSubdirs' > +__version__ =3D '%s Version %s' % (__prog__, '0.10 ') > +__copyright__ =3D 'Copyright (c) 2018, Intel Corporation. All rights r= eserved.' > +__description__ =3D 'Replace for NmakeSubdirs.bat in windows ,support pa= rallel build for nmake.\n' > + > +cpu_count =3D multiprocessing.cpu_count() > +output_lock =3D threading.Lock() > +def RunCommand(WorkDir=3DNone, *Args, **kwargs): > + if WorkDir is None: > + WorkDir =3D os.curdir > + if "stderr" not in kwargs: > + kwargs["stderr"] =3D subprocess.STDOUT > + if "stdout" not in kwargs: > + kwargs["stdout"] =3D subprocess.PIPE > + p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr"= ], stdout=3Dkwargs["stdout"]) > + stdout, stderr =3D p.communicate() > + message =3D "" > + if stdout is not None: > + message =3D stdout.decode() #for compatibility in python 2 and 3 > + > + if p.returncode !=3D 0: > + raise RuntimeError("Error while execute command \'{0}\' in direc= otry {1}\n{2}".format(" ".join(Args), WorkDir, message)) > + > + output_lock.acquire(True) > + print("execute command \"{0}\" in directory {1}".format(" ".join(Arg= s), WorkDir)) > + print(message) > + output_lock.release() > + > + return p.returncode, stdout > + > +class TaskUnit(object): > + def __init__(self, func, args, kwargs): > + self.func =3D func > + self.args =3D args > + self.kwargs =3D kwargs > + > + def __eq__(self, other): > + return id(self).__eq__(id(other)) > + > + def run(self): > + return self.func(*self.args, **self.kwargs) > + > + def __str__(self): > + para =3D list(self.args) > + para.extend("{0}=3D{1}".format(k, v)for k, v in self.kwargs.item= s()) > + > + return "{0}({1})".format(self.func.__name__, ",".join(para)) > + > +class ThreadControl(object): > + > + def __init__(self, maxthread): > + self._processNum =3D maxthread > + self.pending =3D [] > + self.running =3D [] > + self.pendingLock =3D threading.Lock() > + self.runningLock =3D threading.Lock() > + self.error =3D False > + self.errorLock =3D threading.Lock() > + self.errorMsg =3D "errorMsg" > + > + def addTask(self, func, *args, **kwargs): > + self.pending.append(TaskUnit(func, args, kwargs)) > + > + def waitComplete(self): > + self._schedule.join() > + > + def startSchedule(self): > + self._schedule =3D threading.Thread(target=3Dself.Schedule) > + self._schedule.start() > + > + def Schedule(self): > + for i in range(self._processNum): > + task =3D threading.Thread(target=3Dself.startTask) > + task.daemon =3D False > + self.running.append(task) > + > + self.runningLock.acquire(True) > + for thread in self.running: > + thread.start() > + self.runningLock.release() > + > + while len(self.running) > 0: > + time.sleep(0.1) > + if self.error: > + print("subprocess not exit sucessfully") > + print(self.errorMsg) > + > + def startTask(self): > + while True: > + if self.error: > + break > + self.pendingLock.acquire(True) > + if len(self.pending) =3D=3D 0: > + self.pendingLock.release() > + break > + task =3D self.pending.pop(0) > + self.pendingLock.release() > + try: > + task.run() > + except RuntimeError as e: > + if self.error: break > + self.errorLock.acquire(True) > + self.error =3D True > + self.errorMsg =3D str(e) > + time.sleep(0.1) > + self.errorLock.release() > + break > + > + self.runningLock.acquire(True) > + self.running.remove(threading.currentThread()) > + self.runningLock.release() > + > +def Run(): > + curdir =3D os.path.abspath(os.curdir) > + if len(args.subdirs) =3D=3D 1: > + args.jobs =3D 1 > + if args.jobs =3D=3D 1: > + try: > + for dir in args.subdirs: > + RunCommand(os.path.join(curdir, dir), "nmake", args.targ= et, stdout=3Dsys.stdout, stderr=3Dsubprocess.STDOUT) > + except RuntimeError: > + exit(1) > + else: > + controller =3D ThreadControl(args.jobs) > + for dir in args.subdirs: > + controller.addTask(RunCommand, os.path.join(curdir, dir), "n= make", args.target) > + controller.startSchedule() > + controller.waitComplete() > + if controller.error: > + exit(1) > + > +if __name__ =3D=3D "__main__": > + parser =3D argparse.ArgumentParser(prog=3D__prog__, description=3D__= description__ + __copyright__, conflict_handler=3D'resolve') > + > + parser.add_argument("target", help=3D"the target for nmake") > + parser.add_argument("subdirs", nargs=3D"+", help=3D"the relative dir= path of makefile") > + parser.add_argument("--jobs", type=3Dint, dest=3D"jobs", default=3Dc= pu_count, help=3D"thread number") > + parser.add_argument('--version', action=3D'version', version=3D__ver= sion__) > + args =3D parser.parse_args() > + Run() > + > --=20 > 2.10.0.windows.1 >=20 > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel