From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.93, mailfrom: bob.c.feng@intel.com) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by groups.io with SMTP; Tue, 30 Jul 2019 22:52:51 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Jul 2019 22:52:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,328,1559545200"; d="scan'208";a="323616634" Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 30 Jul 2019 22:52:49 -0700 From: "Bob Feng" To: devel@edk2.groups.io Subject: [Patch 00/10 V5] Enable multiple process AutoGen Date: Wed, 31 Jul 2019 13:52:34 +0800 Message-Id: <20190731055244.19872-1-bob.c.feng@intel.com> X-Mailer: git-send-email 2.20.1.windows.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875 In order to improve the build performance, we implemented multiple-processes AutoGen. This change will reduce 20% time for AutoGen phase. The design document can be got from: https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages under Edk2 repository and intel client and server platforms. V5: Restructure the patches of V4. Squash 11/11 patch of V4 into V5 4/10, 5/10 and 10/10. The details of the changes are as following: 1. Update Log queue maxsize as the value of thread number * 10. This change is implemented in patch 10/10 2. Fixed the regression issue that the PCDs from build command line are not passed into autogen sub-process correctly. 3. Fixed the regression issue that the exception raised inside autogen sub-process is not well handled, including code exception and Ctrl+C keyboard interrupt. The above 2 changes are implemented in patch 5/10 4. Fixed the regression issue that multiple builds of the same driver/application with FILE_GUID overrides is not handle correctly. 5. Fixed the regression issue that shared fixed PCD between lib and module is not handle correctly, which cause the autogen.c and autogen.h are different from orignal. The above 2 changes are implemented in patch 4/10 V4: Add one more patch 11/11 to enhance this feature. 1-10 are the same as V3 1. Set Log queue maxsize as thread number * 10 2. enhance ModuleUniqueBaseName function 3. fix bugs of build option pcd in sub Process 4. enhance error handling. Handle the exception of KeyboardInterrup and exceptions happen in subprocess. 5. fix the issue of shared fixed pcd between module and lib. 6. fix bug in the function of duplicate modules handling. V3: 1. Fixed incremental build issue. 2. Set AutoGen worker number to be align with "-n THREADNUMBER" 3. Enable block log queue. V2: 1. The first version missed autogen related commit from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit back. 2. Move CreateAsBuildInf into AutoGenWorker process 3. Save GlobalVar__.bin to build folder. 4. Regenerate patches based on master bb824f685d Feng, Bob C (10): BaseTools: Singleton the object to handle build conf file BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions BaseTools: Add functions to get platform scope build options BaseTools: Decouple AutoGen Objects BaseTools: Enable Multiple Process AutoGen BaseTools: Add shared data for processes BaseTools: Add LogAgent to support multiple process Autogen BaseTools: Move BuildOption parser out of build.py BaseTools: Add the support for python 2 BaseTools: Enable block queue log agent. BaseTools/Source/Python/AutoGen/AutoGen.py | 4227 +---------------- .../Source/Python/AutoGen/AutoGenWorker.py | 260 + .../Source/Python/AutoGen/BuildEngine.py | 22 + BaseTools/Source/Python/AutoGen/DataPipe.py | 160 + BaseTools/Source/Python/AutoGen/GenC.py | 6 +- .../Source/Python/AutoGen/ModuleAutoGen.py | 1903 ++++++++ .../Python/AutoGen/ModuleAutoGenHelper.py | 619 +++ .../Source/Python/AutoGen/PlatformAutoGen.py | 1511 ++++++ .../Source/Python/AutoGen/WorkspaceAutoGen.py | 907 ++++ BaseTools/Source/Python/Common/EdkLogger.py | 119 +- BaseTools/Source/Python/Common/Misc.py | 1 - .../Python/Common/TargetTxtClassObject.py | 28 +- .../Python/Common/ToolDefClassObject.py | 6 +- BaseTools/Source/Python/GenFds/GenFds.py | 4 +- .../Python/GenFds/GenFdsGlobalVariable.py | 54 +- .../Python/PatchPcdValue/PatchPcdValue.py | 1 - .../Source/Python/Workspace/DscBuildData.py | 38 +- .../Source/Python/Workspace/InfBuildData.py | 39 + .../Python/Workspace/WorkspaceCommon.py | 4 + .../Python/Workspace/WorkspaceDatabase.py | 3 + BaseTools/Source/Python/build/BuildReport.py | 4 +- BaseTools/Source/Python/build/build.py | 376 +- BaseTools/Source/Python/build/buildoptions.py | 92 + 23 files changed, 5927 insertions(+), 4457 deletions(-) create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py create mode 100644 BaseTools/Source/Python/build/buildoptions.py -- 2.20.1.windows.1