From: "Bob Feng" <bob.c.feng@intel.com>
To: devel@edk2.groups.io
Subject: [Patch 00/11 V4] Enable multiple process AutoGen
Date: Mon, 29 Jul 2019 16:44:45 +0800 [thread overview]
Message-ID: <20190729084456.18844-1-bob.c.feng@intel.com> (raw)
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
In order to improve the build performance, we implemented
multiple-processes AutoGen. This change will reduce 20% time
for AutoGen phase.
The design document can be got from:
https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf
This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages
under Edk2 repository and intel client and server platforms.
V4:
Add one more patch 11/11 to enhance this feature. 1-10 are the same as V3
1. Set Log queue maxsize as thread number * 10
2. enhance ModuleUniqueBaseName function
3. fix bugs of build option pcd in sub Process
4. enhance error handling. Handle the exception of
KeyboardInterrup and exceptions happen in subprocess.
5. fix the issue of shared fixed pcd between module and lib.
6. fix bug in the function of duplicate modules handling.
V3:
1. Fixed incremental build issue.
2. Set AutoGen worker number to be align with "-n THREADNUMBER"
3. Enable block log queue.
V2:
1. The first version missed autogen related commit
from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit
back.
2. Move CreateAsBuildInf into AutoGenWorker process
3. Save GlobalVar_<platform guid>_<arch>.bin to build folder.
4. Regenerate patches based on master bb824f685d
Feng, Bob C (11):
BaseTools: Singleton the object to handle build conf file
BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
BaseTools: Add functions to get platform scope build options
BaseTools: Decouple AutoGen Objects
BaseTools: Enable Multiple Process AutoGen
BaseTools: Add shared data for processes
BaseTools: Add LogAgent to support multiple process Autogen
BaseTools: Move BuildOption parser out of build.py
BaseTools: Add the support for python 2
BaseTools: Enable block queue log agent.
BaseTools: Enhance Multiple-Process AutoGen
BaseTools/Source/Python/AutoGen/AutoGen.py | 4227 +----------------
.../Source/Python/AutoGen/AutoGenWorker.py | 257 +
.../Source/Python/AutoGen/BuildEngine.py | 22 +
BaseTools/Source/Python/AutoGen/DataPipe.py | 160 +
BaseTools/Source/Python/AutoGen/GenC.py | 6 +-
.../Source/Python/AutoGen/ModuleAutoGen.py | 1903 ++++++++
.../Python/AutoGen/ModuleAutoGenHelper.py | 619 +++
.../Source/Python/AutoGen/PlatformAutoGen.py | 1512 ++++++
.../Source/Python/AutoGen/WorkspaceAutoGen.py | 907 ++++
BaseTools/Source/Python/Common/EdkLogger.py | 119 +-
BaseTools/Source/Python/Common/Misc.py | 1 -
.../Python/Common/TargetTxtClassObject.py | 28 +-
.../Python/Common/ToolDefClassObject.py | 6 +-
BaseTools/Source/Python/GenFds/GenFds.py | 4 +-
.../Python/GenFds/GenFdsGlobalVariable.py | 54 +-
.../Python/PatchPcdValue/PatchPcdValue.py | 1 -
.../Source/Python/Workspace/DscBuildData.py | 38 +-
.../Source/Python/Workspace/InfBuildData.py | 39 +
.../Python/Workspace/WorkspaceCommon.py | 4 +
.../Python/Workspace/WorkspaceDatabase.py | 3 +
BaseTools/Source/Python/build/BuildReport.py | 4 +-
BaseTools/Source/Python/build/build.py | 370 +-
BaseTools/Source/Python/build/buildoptions.py | 92 +
23 files changed, 5921 insertions(+), 4455 deletions(-)
create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py
create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py
create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
create mode 100644 BaseTools/Source/Python/build/buildoptions.py
--
2.20.1.windows.1
next reply other threads:[~2019-07-29 8:45 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-29 8:44 Bob Feng [this message]
2019-07-29 8:44 ` [Patch 01/11] BaseTools: Singleton the object to handle build conf file Bob Feng
2019-07-29 8:44 ` [Patch 02/11] BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions Bob Feng
2019-07-29 15:03 ` [edk2-devel] " Philippe Mathieu-Daudé
2019-07-30 2:10 ` Bob Feng
2019-07-30 12:38 ` Philippe Mathieu-Daudé
2019-07-29 8:44 ` [Patch 03/11] BaseTools: Add functions to get platform scope build options Bob Feng
2019-07-29 8:44 ` [Patch 04/11] BaseTools: Decouple AutoGen Objects Bob Feng
2019-07-29 8:44 ` [Patch 05/11] BaseTools: Enable Multiple Process AutoGen Bob Feng
2019-07-29 8:44 ` [Patch 06/11] BaseTools: Add shared data for processes Bob Feng
2019-07-29 8:44 ` [Patch 07/11] BaseTools: Add LogAgent to support multiple process Autogen Bob Feng
2019-07-29 8:44 ` [Patch 08/11] BaseTools: Move BuildOption parser out of build.py Bob Feng
2019-07-29 8:44 ` [Patch 09/11] BaseTools: Add the support for python 2 Bob Feng
2019-07-29 8:44 ` [Patch 10/11] BaseTools: Enable block queue log agent Bob Feng
2019-07-29 8:44 ` [Patch 11/11] BaseTools: Enhance Multiple-Process AutoGen Bob Feng
2019-07-29 10:10 ` [edk2-devel] [Patch 00/11 V4] Enable multiple process AutoGen Laszlo Ersek
2019-07-30 7:31 ` Bob Feng
2019-07-30 14:02 ` Laszlo Ersek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190729084456.18844-1-bob.c.feng@intel.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox