From: "Bob Feng" <bob.c.feng@intel.com>
To: devel@edk2.groups.io
Cc: Liming Gao <liming.gao@intel.com>, Bob Feng <bob.c.feng@intel.com>
Subject: [Patch 06/10 V3] BaseTools: Add shared data for processes
Date: Tue, 23 Jul 2019 11:58:14 +0800 [thread overview]
Message-ID: <20190723035818.21112-7-bob.c.feng@intel.com> (raw)
In-Reply-To: <20190723035818.21112-1-bob.c.feng@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
Add shared data for autogen processes.
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
---
BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 3 ++-
BaseTools/Source/Python/build/build.py | 10 ++++++----
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
index edec346abd06..1419d92b5e39 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -52,18 +52,19 @@ class AutoGenManager(threading.Thread):
for w in self.autogen_workers:
if w.is_alive():
w.terminate()
class AutoGenWorkerInProcess(mp.Process):
- def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_lock):
+ def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_lock, share_data):
mp.Process.__init__(self)
self.module_queue = module_queue
self.data_pipe_file_path =data_pipe_file_path
self.data_pipe = None
self.feedback_q = feedback_q
self.PlatformMetaFileSet = {}
self.file_lock = file_lock
+ self.share_data = share_data
def GetPlatformMetaFile(self,filepath,root):
try:
return self.PlatformMetaFileSet[(filepath,root)]
except:
self.PlatformMetaFileSet[(filepath,root)] = filepath
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index 590584b3c67f..d49554ec0282 100644
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -50,10 +50,11 @@ from GenPatchPcdTable.GenPatchPcdTable import PeImageClass,parsePcdInfoFromMapFi
from PatchPcdValue.PatchPcdValue import PatchBinaryFile
import Common.GlobalData as GlobalData
from GenFds.GenFds import GenFds, GenFdsApi
import multiprocessing as mp
+from multiprocessing import Manager
# Version and Copyright
VersionNumber = "0.60" + ' ' + gBUILD_VERSION
__version__ = "%prog Version " + VersionNumber
__copyright__ = "Copyright (c) 2007 - 2018, Intel Corporation All rights reserved."
@@ -826,16 +827,17 @@ class Build():
self.InitBuild()
self.AutoGenMgr = None
EdkLogger.info("")
os.chdir(self.WorkspaceDir)
- def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList):
+ self.share_data = Manager().dict()
+ def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_data):
if SkipAutoGen:
return
feedback_q = mp.Queue()
file_lock = mp.Lock()
- auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,file_lock) for _ in range(self.ThreadNumber)]
+ auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,file_lock,share_data) for _ in range(self.ThreadNumber)]
self.AutoGenMgr = AutoGenManager(auto_workers,feedback_q)
self.AutoGenMgr.start()
for w in auto_workers:
w.start()
if PcdMaList is not None:
@@ -1228,11 +1230,11 @@ class Build():
AutoGenObject.DataPipe.DataContainer = {"FfsCommand":FfsCommand}
self.Progress.Start("Generating makefile and code")
data_pipe_file = os.path.join(AutoGenObject.BuildDir, "GlobalVar_%s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch))
AutoGenObject.DataPipe.dump(data_pipe_file)
- autogen_rt = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList)
+ autogen_rt = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
self.Progress.Stop("done!")
return autogen_rt
else:
# always recreate top/platform makefile when clean, just in case of inconsistency
AutoGenObject.CreateCodeFile(False)
@@ -2060,11 +2062,11 @@ class Build():
mqueue = mp.Queue()
for m in Pa.GetAllModuleInfo:
mqueue.put(m)
data_pipe_file = os.path.join(Pa.BuildDir, "GlobalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch))
Pa.DataPipe.dump(data_pipe_file)
- autogen_rt = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList)
+ autogen_rt = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
self.Progress.Stop("done!")
self.AutoGenTime += int(round((time.time() - AutoGenStart)))
if not autogen_rt:
return
for Arch in Wa.ArchList:
--
2.20.1.windows.1
next prev parent reply other threads:[~2019-07-23 3:58 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-23 3:58 [Patch 00/10 V3] Enable multiple process AutoGen Bob Feng
2019-07-23 3:58 ` [Patch 01/10 V3] BaseTools: Singleton the object to handle build conf file Bob Feng
2019-07-23 3:58 ` [Patch 02/10 V3] BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions Bob Feng
2019-07-23 3:58 ` [Patch 03/10 V3] BaseTools: Add functions to get platform scope build options Bob Feng
2019-07-23 3:58 ` [Patch 04/10 V3] BaseTools: Decouple AutoGen Objects Bob Feng
2019-07-23 3:58 ` [Patch 05/10 V3] BaseTools: Enable Multiple Process AutoGen Bob Feng
2019-07-23 3:58 ` Bob Feng [this message]
2019-07-23 3:58 ` [Patch 07/10 V3] BaseTools: Add LogAgent to support multiple process Autogen Bob Feng
2019-07-23 3:58 ` [Patch 08/10 V3] BaseTools: Move BuildOption parser out of build.py Bob Feng
2019-07-23 3:58 ` [Patch 09/10 V3] BaseTools: Add the support for python 2 Bob Feng
2019-07-23 3:58 ` [Patch 10/10 V3] BaseTools: Enable block queue log agent Bob Feng
2019-07-23 10:11 ` [edk2-devel] " Laszlo Ersek
2019-07-23 12:00 ` Bob Feng
2019-07-23 11:30 ` [edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen Laszlo Ersek
2019-07-30 13:38 ` Bob Feng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190723035818.21112-7-bob.c.feng@intel.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox