From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=134.134.136.100; helo=mga07.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 096F221106C6C for ; Wed, 29 Aug 2018 07:48:08 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Aug 2018 07:48:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,303,1531810800"; d="scan'208";a="258211444" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by fmsmga005.fm.intel.com with ESMTP; 29 Aug 2018 07:47:35 -0700 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 29 Aug 2018 07:47:37 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.143]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.150]) with mapi id 14.03.0319.002; Wed, 29 Aug 2018 22:47:34 +0800 From: "Gao, Liming" To: "Carsey, Jaben" CC: "edk2-devel@lists.01.org" , "Guo, Dongao" Thread-Topic: [edk2] [Patch] BaseTools: Support multi thread build Basetool on Windows Thread-Index: AQHUPuPKY8IL1d9IikmVy9nQToTCYqTUzz0AgAIA7jA= Date: Wed, 29 Aug 2018 14:47:33 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E2EACA8@SHSMSX104.ccr.corp.intel.com> References: <20180828152656.17596-1-liming.gao@intel.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMzI1Y2E2NjgtMWU3ZS00NzdmLTllOGMtNjhlZGJkZDZmNTVkIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoicFQyTjBBeDlZRzVUZ01nNWppNm0rdnNjVFN4SUR1YjdcL2dKZTd1dk1seTZvamxSY2FzWDJmYkpLcFdGQzd3Q0IifQ== dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [Patch] BaseTools: Support multi thread build Basetool on Windows X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 14:48:09 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Jaben: NmakeSubdirs.bat will be removed once NmakeSubdirs.py is stable. Now, bat= is kept for a while. Thanks Liming > -----Original Message----- > From: Carsey, Jaben > Sent: Wednesday, August 29, 2018 12:10 AM > To: Gao, Liming > Cc: edk2-devel@lists.01.org; Guo, Dongao > Subject: Re: [edk2] [Patch] BaseTools: Support multi thread build Basetoo= l on Windows >=20 > This looks like a great change. Why not delete the bat file? >=20 > Reviewed-by Jaben Carsey: >=20 > Jaben >=20 > > On Aug 28, 2018, at 8:28 AM, Liming Gao wrote: > > > > From: Dongao Guo > > > > Add NmakeSubdirs.py to replace NmakeSubdirs.bat in VS Makefile. This sc= ript will > > invoke nmake in multi thread mode. It can save more than half time of B= aseTools > > C clean build. > > GCC make supports multiple thread in make phase. So, GNNmakefile doesn'= t need apply > > this script. > > > > single task or job=3D1: > > just single thread and invoke subprocess,subprocess will use > > system.stdout to print output. > > multi task: > > thread number is logic cpu count.All subprocess output will pass to > > python script by PIPE and then script print it to system.stdout. > > > > Contributed-under: TianoCore Contribution Agreement 1.1 > > Signed-off-by: Dongao Guo > > Reviewed-by: Liming Gao > > Test-by: Liming Gao > > --- > > BaseTools/Makefile | 12 +- > > BaseTools/Source/C/Makefile | 14 +-- > > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 169 ++++++++++++++++++++= +++++++ > > 3 files changed, 182 insertions(+), 13 deletions(-) > > create mode 100644 BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > > diff --git a/BaseTools/Makefile b/BaseTools/Makefile > > index 3736d85..b98cd85 100644 > > --- a/BaseTools/Makefile > > +++ b/BaseTools/Makefile > > @@ -1,7 +1,7 @@ > > ## @file > > # Windows makefile for Base Tools project build. > > # > > -# Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved. > > +# Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved. > > # This program and the accompanying materials > > # are licensed and made available under the terms and conditions of the= BSD License > > # which accompanies this distribution. The full text of the license ma= y be found at > > @@ -20,19 +20,19 @@ SUBDIRS =3D $(BASE_TOOLS_PATH)\Source\C $(BASE_TOOL= S_PATH)\Source\Python > > all: c python > > > > c : > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_T= OOLS_PATH)\Source\C > > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nma= keSubdirs.py all > $(BASE_TOOLS_PATH)\Source\C > > > > python: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $(BASE_T= OOLS_PATH)\Source\Python > > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nma= keSubdirs.py all > $(BASE_TOOLS_PATH)\Source\Python > > > > subdirs: $(SUBDIRS) > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat all $** > > + @$(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nma= keSubdirs.py all $** > > > > .PHONY: clean > > clean: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat clean $(SUBD= IRS) > > + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nmak= eSubdirs.py clean $(SUBDIRS) > > > > .PHONY: cleanall > > cleanall: > > - @$(BASE_TOOLS_PATH)\Source\C\Makefiles\NmakeSubdirs.bat cleanall $(S= UBDIRS) > > + $(PYTHON_HOME)\python.exe $(BASE_TOOLS_PATH)\Source\C\Makefiles\Nmak= eSubdirs.py cleanall $(SUBDIRS) > > > > diff --git a/BaseTools/Source/C/Makefile b/BaseTools/Source/C/Makefile > > index 5428180..1246d23 100644 > > --- a/BaseTools/Source/C/Makefile > > +++ b/BaseTools/Source/C/Makefile > > @@ -1,7 +1,7 @@ > > ## @file > > # Windows makefile for C tools build. > > # > > -# Copyright (c) 2009 - 2017, Intel Corporation. All rights reserved. > > +# Copyright (c) 2009 - 2018, Intel Corporation. All rights reserved. > > # This program and the accompanying materials > > # are licensed and made available under the terms and conditions of the= BSD License > > # which accompanies this distribution. The full text of the license ma= y be found at > > @@ -16,7 +16,7 @@ HOST_ARCH =3D IA32 > > > > LIBRARIES =3D Common > > APPLICATIONS =3D \ > > - BootSectImage \ > > + VfrCompile \ > > BrotliCompress \ > > EfiLdrImage \ > > EfiRom \ > > @@ -32,7 +32,7 @@ APPLICATIONS =3D \ > > Split \ > > TianoCompress \ > > VolInfo \ > > - VfrCompile \ > > + BootSectImage \ > > DevicePath > > > > all: libs apps install > > @@ -43,7 +43,7 @@ libs: $(LIBRARIES) > > @echo # Build libraries > > @echo ###################### > > @if not exist $(LIB_PATH) mkdir $(LIB_PATH) > > - @Makefiles\NmakeSubdirs.bat all $** > > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** > > > > apps: $(APPLICATIONS) > > @echo. > > @@ -51,7 +51,7 @@ apps: $(APPLICATIONS) > > @echo # Build executables > > @echo ###################### > > @if not exist $(BIN_PATH) mkdir $(BIN_PATH) > > - @Makefiles\NmakeSubdirs.bat all $** > > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py all $** > > > > install: $(LIB_PATH) $(BIN_PATH) > > @echo. > > @@ -65,11 +65,11 @@ install: $(LIB_PATH) $(BIN_PATH) > > > > .PHONY: clean > > clean: > > - @Makefiles\NmakeSubdirs.bat clean $(LIBRARIES) $(APPLICATIONS) > > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py clean $(LIBRARI= ES) $(APPLICATIONS) > > > > .PHONY: cleanall > > cleanall: > > - @Makefiles\NmakeSubdirs.bat cleanall $(LIBRARIES) $(APPLICATIONS) > > + @$(PYTHON_HOME)\python.exe Makefiles\NmakeSubdirs.py cleanall $(LIBR= ARIES) $(APPLICATIONS) > > > > !INCLUDE Makefiles\ms.rule > > > > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py b/BaseTools/S= ource/C/Makefiles/NmakeSubdirs.py > > new file mode 100644 > > index 0000000..29bb5df > > --- /dev/null > > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > @@ -0,0 +1,169 @@ > > +# @file NmakeSubdirs.py > > +# This script support parallel build for nmake in windows environment. > > +# It supports Python2.x and Python3.x both. > > +# > > +# Copyright (c) 2018, Intel Corporation. All rights reserved.
> > +# > > +# This program and the accompanying materials > > +# are licensed and made available under the terms and conditions of t= he BSD License > > +# which accompanies this distribution. The full text of the license = may be found at > > +# http://opensource.org/licenses/bsd-license.php > > +# > > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASI= S, > > +# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS O= R IMPLIED. > > +# > > + > > +# > > +# Import Modules > > +# > > + > > +from __future__ import print_function > > +import argparse > > +import threading > > +import time > > +import os > > +import subprocess > > +import multiprocessing > > +import copy > > +import sys > > +__prog__ =3D 'NmakeSubdirs' > > +__version__ =3D '%s Version %s' % (__prog__, '0.10 ') > > +__copyright__ =3D 'Copyright (c) 2018, Intel Corporation. All rights= reserved.' > > +__description__ =3D 'Replace for NmakeSubdirs.bat in windows ,support = parallel build for nmake.\n' > > + > > +cpu_count =3D multiprocessing.cpu_count() > > +output_lock =3D threading.Lock() > > +def RunCommand(WorkDir=3DNone, *Args, **kwargs): > > + if WorkDir is None: > > + WorkDir =3D os.curdir > > + if "stderr" not in kwargs: > > + kwargs["stderr"] =3D subprocess.STDOUT > > + if "stdout" not in kwargs: > > + kwargs["stdout"] =3D subprocess.PIPE > > + p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stder= r"], stdout=3Dkwargs["stdout"]) > > + stdout, stderr =3D p.communicate() > > + message =3D "" > > + if stdout is not None: > > + message =3D stdout.decode() #for compatibility in python 2 and= 3 > > + > > + if p.returncode !=3D 0: > > + raise RuntimeError("Error while execute command \'{0}\' in dir= ecotry {1}\n{2}".format(" ".join(Args), WorkDir, message)) > > + > > + output_lock.acquire(True) > > + print("execute command \"{0}\" in directory {1}".format(" ".join(A= rgs), WorkDir)) > > + print(message) > > + output_lock.release() > > + > > + return p.returncode, stdout > > + > > +class TaskUnit(object): > > + def __init__(self, func, args, kwargs): > > + self.func =3D func > > + self.args =3D args > > + self.kwargs =3D kwargs > > + > > + def __eq__(self, other): > > + return id(self).__eq__(id(other)) > > + > > + def run(self): > > + return self.func(*self.args, **self.kwargs) > > + > > + def __str__(self): > > + para =3D list(self.args) > > + para.extend("{0}=3D{1}".format(k, v)for k, v in self.kwargs.it= ems()) > > + > > + return "{0}({1})".format(self.func.__name__, ",".join(para)) > > + > > +class ThreadControl(object): > > + > > + def __init__(self, maxthread): > > + self._processNum =3D maxthread > > + self.pending =3D [] > > + self.running =3D [] > > + self.pendingLock =3D threading.Lock() > > + self.runningLock =3D threading.Lock() > > + self.error =3D False > > + self.errorLock =3D threading.Lock() > > + self.errorMsg =3D "errorMsg" > > + > > + def addTask(self, func, *args, **kwargs): > > + self.pending.append(TaskUnit(func, args, kwargs)) > > + > > + def waitComplete(self): > > + self._schedule.join() > > + > > + def startSchedule(self): > > + self._schedule =3D threading.Thread(target=3Dself.Schedule) > > + self._schedule.start() > > + > > + def Schedule(self): > > + for i in range(self._processNum): > > + task =3D threading.Thread(target=3Dself.startTask) > > + task.daemon =3D False > > + self.running.append(task) > > + > > + self.runningLock.acquire(True) > > + for thread in self.running: > > + thread.start() > > + self.runningLock.release() > > + > > + while len(self.running) > 0: > > + time.sleep(0.1) > > + if self.error: > > + print("subprocess not exit sucessfully") > > + print(self.errorMsg) > > + > > + def startTask(self): > > + while True: > > + if self.error: > > + break > > + self.pendingLock.acquire(True) > > + if len(self.pending) =3D=3D 0: > > + self.pendingLock.release() > > + break > > + task =3D self.pending.pop(0) > > + self.pendingLock.release() > > + try: > > + task.run() > > + except RuntimeError as e: > > + if self.error: break > > + self.errorLock.acquire(True) > > + self.error =3D True > > + self.errorMsg =3D str(e) > > + time.sleep(0.1) > > + self.errorLock.release() > > + break > > + > > + self.runningLock.acquire(True) > > + self.running.remove(threading.currentThread()) > > + self.runningLock.release() > > + > > +def Run(): > > + curdir =3D os.path.abspath(os.curdir) > > + if len(args.subdirs) =3D=3D 1: > > + args.jobs =3D 1 > > + if args.jobs =3D=3D 1: > > + try: > > + for dir in args.subdirs: > > + RunCommand(os.path.join(curdir, dir), "nmake", args.ta= rget, stdout=3Dsys.stdout, stderr=3Dsubprocess.STDOUT) > > + except RuntimeError: > > + exit(1) > > + else: > > + controller =3D ThreadControl(args.jobs) > > + for dir in args.subdirs: > > + controller.addTask(RunCommand, os.path.join(curdir, dir), = "nmake", args.target) > > + controller.startSchedule() > > + controller.waitComplete() > > + if controller.error: > > + exit(1) > > + > > +if __name__ =3D=3D "__main__": > > + parser =3D argparse.ArgumentParser(prog=3D__prog__, description=3D= __description__ + __copyright__, conflict_handler=3D'resolve') > > + > > + parser.add_argument("target", help=3D"the target for nmake") > > + parser.add_argument("subdirs", nargs=3D"+", help=3D"the relative d= ir path of makefile") > > + parser.add_argument("--jobs", type=3Dint, dest=3D"jobs", default= =3Dcpu_count, help=3D"thread number") > > + parser.add_argument('--version', action=3D'version', version=3D__v= ersion__) > > + args =3D parser.parse_args() > > + Run() > > + > > -- > > 2.10.0.windows.1 > > > > _______________________________________________ > > edk2-devel mailing list > > edk2-devel@lists.01.org > > https://lists.01.org/mailman/listinfo/edk2-devel