public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
From: "Bob Feng" <bob.c.feng@intel.com>
To: devel@edk2.groups.io
Cc: Liming Gao <liming.gao@intel.com>, Bob Feng <bob.c.feng@intel.com>
Subject: [Patch 06/10 V6] BaseTools: Add shared data for processes
Date: Thu,  1 Aug 2019 11:03:29 +0800	[thread overview]
Message-ID: <20190801030333.8588-7-bob.c.feng@intel.com> (raw)
In-Reply-To: <20190801030333.8588-1-bob.c.feng@intel.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

Add shared data for autogen processes.

Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
---
 BaseTools/Source/Python/AutoGen/AutoGenWorker.py |  3 ++-
 BaseTools/Source/Python/build/build.py           | 10 ++++++----
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
index 8736fde50ef7..fa2852cf2c6b 100644
--- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py
@@ -62,18 +62,19 @@ class AutoGenManager(threading.Thread):
     def TerminateWorkers(self):
         self.error_event.set()
     def kill(self):
         self.feedback_q.put(None)
 class AutoGenWorkerInProcess(mp.Process):
-    def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_lock,error_event):
+    def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_lock, share_data,error_event):
         mp.Process.__init__(self)
         self.module_queue = module_queue
         self.data_pipe_file_path =data_pipe_file_path
         self.data_pipe = None
         self.feedback_q = feedback_q
         self.PlatformMetaFileSet = {}
         self.file_lock = file_lock
+        self.share_data = share_data
         self.error_event = error_event
     def GetPlatformMetaFile(self,filepath,root):
         try:
             return self.PlatformMetaFileSet[(filepath,root)]
         except:
diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
index 579b57a4e958..5eed57bf6404 100644
--- a/BaseTools/Source/Python/build/build.py
+++ b/BaseTools/Source/Python/build/build.py
@@ -50,10 +50,11 @@ from GenPatchPcdTable.GenPatchPcdTable import PeImageClass,parsePcdInfoFromMapFi
 from PatchPcdValue.PatchPcdValue import PatchBinaryFile
 
 import Common.GlobalData as GlobalData
 from GenFds.GenFds import GenFds, GenFdsApi
 import multiprocessing as mp
+from multiprocessing import Manager
 
 # Version and Copyright
 VersionNumber = "0.60" + ' ' + gBUILD_VERSION
 __version__ = "%prog Version " + VersionNumber
 __copyright__ = "Copyright (c) 2007 - 2018, Intel Corporation  All rights reserved."
@@ -826,18 +827,19 @@ class Build():
             self.InitBuild()
 
         self.AutoGenMgr = None
         EdkLogger.info("")
         os.chdir(self.WorkspaceDir)
-    def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList):
+        self.share_data = Manager().dict()
+    def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_data):
         try:
             if SkipAutoGen:
                 return True,0
             feedback_q = mp.Queue()
             file_lock = mp.Lock()
             error_event = mp.Event()
-            auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,file_lock,error_event) for _ in range(self.ThreadNumber)]
+            auto_workers = [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file,feedback_q,file_lock,share_data,error_event) for _ in range(self.ThreadNumber)]
             self.AutoGenMgr = AutoGenManager(auto_workers,feedback_q,error_event)
             self.AutoGenMgr.start()
             for w in auto_workers:
                 w.start()
             if PcdMaList is not None:
@@ -1229,11 +1231,11 @@ class Build():
 
             AutoGenObject.DataPipe.DataContainer = {"FfsCommand":FfsCommand}
             self.Progress.Start("Generating makefile and code")
             data_pipe_file = os.path.join(AutoGenObject.BuildDir, "GlobalVar_%s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch))
             AutoGenObject.DataPipe.dump(data_pipe_file)
-            autogen_rt,errorcode = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList)
+            autogen_rt, errorcode = self.StartAutoGen(mqueue, AutoGenObject.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
             self.Progress.Stop("done!")
             if not autogen_rt:
                 self.AutoGenMgr.TerminateWorkers()
                 self.AutoGenMgr.join(0.1)
                 raise FatalError(errorcode)
@@ -2066,11 +2068,11 @@ class Build():
                     mqueue = mp.Queue()
                     for m in Pa.GetAllModuleInfo:
                         mqueue.put(m)
                     data_pipe_file = os.path.join(Pa.BuildDir, "GlobalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch))
                     Pa.DataPipe.dump(data_pipe_file)
-                    autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList)
+                    autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data)
                     self.Progress.Stop("done!")
                     self.AutoGenTime += int(round((time.time() - AutoGenStart)))
                     if not autogen_rt:
                         self.AutoGenMgr.TerminateWorkers()
                         self.AutoGenMgr.join(0.1)
-- 
2.20.1.windows.1


  parent reply	other threads:[~2019-08-01  3:03 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-01  3:03 [Patch 00/10 V6] Enable multiple process AutoGen Bob Feng
2019-08-01  3:03 ` [Patch 01/10 V6] BaseTools: Singleton the object to handle build conf file Bob Feng
2019-08-01  3:03 ` [Patch 02/10 V6] BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions Bob Feng
2019-08-01  3:03 ` [Patch 03/10 V6] BaseTools: Add functions to get platform scope build options Bob Feng
2019-08-01  3:03 ` [Patch 04/10 V6] BaseTools: Decouple AutoGen Objects Bob Feng
2019-08-01  3:03 ` [Patch 05/10 V6] BaseTools: Enable Multiple Process AutoGen Bob Feng
2019-08-01  3:03 ` Bob Feng [this message]
2019-08-01  3:03 ` [Patch 07/10 V6] BaseTools: Add LogAgent to support multiple process Autogen Bob Feng
2019-08-01  3:03 ` [Patch 08/10 V6] BaseTools: Move BuildOption parser out of build.py Bob Feng
2019-08-01  3:03 ` [Patch 09/10 V6] BaseTools: Add the support for python 2 Bob Feng
2019-08-01  3:03 ` [Patch 10/10 V6] BaseTools: Enable block queue log agent Bob Feng

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-list from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190801030333.8588-7-bob.c.feng@intel.com \
    --to=devel@edk2.groups.io \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox