public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
* [Patch 00/10 V7] Enable multiple process AutoGen
@ 2019-08-06  7:15 Bob Feng
  2019-08-06  7:15 ` [Patch 01/10 V7] BaseTools: Singleton the object to handle build conf file Bob Feng
                   ` (9 more replies)
  0 siblings, 10 replies; 11+ messages in thread
From: Bob Feng @ 2019-08-06  7:15 UTC (permalink / raw)
  To: devel

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

In order to improve the build performance, we implemented
multiple-processes AutoGen. This change will reduce 20% time
for AutoGen phase.

The design document can be got from:
https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf

This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages
under Edk2 repository and intel client and server platforms.

V7:
1. Fixed regression issue for build -m, build modules, build libraries.
implement this fix in patch 05/10.

2. Fixed the regression issue for duplicat items in PcdTokenNumber file.
implement this fix in patch 02/10.

V6:

This version fixed a regression issue on incremental build.
The change is in patch 04/10

In the autogen sub-process, we need to use 
PlatformInfo.Pcds rather than PlatformInfo.Platform.Pcds
because we don't want to re-evaluate the Pcds in
the autogen sub-process.

V5:
Restructure the patches of V4. Squash 11/11 patch of V4 into V5 4/10,
5/10 and 10/10.

The details of the changes are as following:

1. Update Log queue maxsize as the value of thread number * 10.
This change is implemented in patch 10/10

2. Fixed the regression issue that the PCDs from build command line
are not passed into autogen sub-process correctly.

3. Fixed the regression issue that the exception raised inside
autogen sub-process is not well handled, including code exception
and Ctrl+C keyboard interrupt.

The above 2 changes are implemented in patch 5/10

4. Fixed the regression issue that multiple builds of the same 
driver/application with FILE_GUID overrides is not handle correctly.

5. Fixed the regression issue that shared fixed PCD between lib
and module is not handle correctly, which cause the autogen.c and 
autogen.h are different from orignal.

The above 2 changes are implemented in patch 4/10

V4:
Add one more patch 11/11 to enhance this feature. 1-10 are the same as V3
1. Set Log queue maxsize as thread number * 10
2. enhance ModuleUniqueBaseName function
3. fix bugs of build option pcd in sub Process
4. enhance error handling. Handle the exception of
KeyboardInterrup and exceptions happen in subprocess.
5. fix the issue of shared fixed pcd between module and lib.
6. fix bug in the function of duplicate modules handling.
V3:
1. Fixed incremental build issue.
2. Set AutoGen worker number to be align with "-n THREADNUMBER"
3. Enable block log queue.
V2: 
1. The first version missed autogen related commit 
from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit
back.
2. Move CreateAsBuildInf into AutoGenWorker process
3. Save GlobalVar_<platform guid>_<arch>.bin to build folder.
4. Regenerate patches based on master bb824f685d
Feng, Bob C (10):
  BaseTools: Singleton the object to handle build conf file
  BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
  BaseTools: Add functions to get platform scope build options
  BaseTools: Decouple AutoGen Objects
  BaseTools: Enable Multiple Process AutoGen
  BaseTools: Add shared data for processes
  BaseTools: Add LogAgent to support multiple process Autogen
  BaseTools: Move BuildOption parser out of build.py
  BaseTools: Add the support for python 2
  BaseTools: Enable block queue log agent.

 BaseTools/Source/Python/AutoGen/AutoGen.py    | 4227 +----------------
 .../Source/Python/AutoGen/AutoGenWorker.py    |  259 +
 .../Source/Python/AutoGen/BuildEngine.py      |   22 +
 BaseTools/Source/Python/AutoGen/DataPipe.py   |  160 +
 BaseTools/Source/Python/AutoGen/GenC.py       |    6 +-
 .../Source/Python/AutoGen/ModuleAutoGen.py    | 1908 ++++++++
 .../Python/AutoGen/ModuleAutoGenHelper.py     |  619 +++
 .../Source/Python/AutoGen/PlatformAutoGen.py  | 1511 ++++++
 .../Source/Python/AutoGen/WorkspaceAutoGen.py |  906 ++++
 BaseTools/Source/Python/Common/EdkLogger.py   |  119 +-
 BaseTools/Source/Python/Common/Misc.py        |    1 -
 .../Python/Common/TargetTxtClassObject.py     |   28 +-
 .../Python/Common/ToolDefClassObject.py       |    6 +-
 BaseTools/Source/Python/GenFds/GenFds.py      |    4 +-
 .../Python/GenFds/GenFdsGlobalVariable.py     |   54 +-
 .../Python/PatchPcdValue/PatchPcdValue.py     |    1 -
 .../Source/Python/Workspace/DscBuildData.py   |   38 +-
 .../Source/Python/Workspace/InfBuildData.py   |   39 +
 .../Python/Workspace/WorkspaceCommon.py       |    4 +
 .../Python/Workspace/WorkspaceDatabase.py     |    3 +
 BaseTools/Source/Python/build/BuildReport.py  |    4 +-
 BaseTools/Source/Python/build/build.py        |  382 +-
 BaseTools/Source/Python/build/buildoptions.py |   92 +
 23 files changed, 5935 insertions(+), 4458 deletions(-)
 create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
 create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py
 create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
 create mode 100644 BaseTools/Source/Python/build/buildoptions.py

-- 
2.20.1.windows.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2019-08-06  7:15 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-08-06  7:15 [Patch 00/10 V7] Enable multiple process AutoGen Bob Feng
2019-08-06  7:15 ` [Patch 01/10 V7] BaseTools: Singleton the object to handle build conf file Bob Feng
2019-08-06  7:15 ` [Patch 02/10 V7] BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions Bob Feng
2019-08-06  7:15 ` [Patch 03/10 V7] BaseTools: Add functions to get platform scope build options Bob Feng
2019-08-06  7:15 ` [Patch 04/10 V7] BaseTools: Decouple AutoGen Objects Bob Feng
2019-08-06  7:15 ` [Patch 05/10 V7] BaseTools: Enable Multiple Process AutoGen Bob Feng
2019-08-06  7:15 ` [Patch 06/10 V7] BaseTools: Add shared data for processes Bob Feng
2019-08-06  7:15 ` [Patch 07/10 V7] BaseTools: Add LogAgent to support multiple process Autogen Bob Feng
2019-08-06  7:15 ` [Patch 08/10 V7] BaseTools: Move BuildOption parser out of build.py Bob Feng
2019-08-06  7:15 ` [Patch 09/10 V7] BaseTools: Add the support for python 2 Bob Feng
2019-08-06  7:15 ` [Patch 10/10 V7] BaseTools: Enable block queue log agent Bob Feng

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox