From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: redhat.com, ip: 209.132.183.28, mailfrom: lersek@redhat.com) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by groups.io with SMTP; Tue, 23 Jul 2019 04:30:59 -0700 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 10DB830C7128; Tue, 23 Jul 2019 11:30:59 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-117-52.ams2.redhat.com [10.36.117.52]) by smtp.corp.redhat.com (Postfix) with ESMTP id 64951608A5; Tue, 23 Jul 2019 11:30:58 +0000 (UTC) Subject: Re: [edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen To: devel@edk2.groups.io, bob.c.feng@intel.com References: <20190723035818.21112-1-bob.c.feng@intel.com> From: "Laszlo Ersek" Message-ID: <4ea3d3fa-2210-3642-2337-db525312d312@redhat.com> Date: Tue, 23 Jul 2019 13:30:57 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190723035818.21112-1-bob.c.feng@intel.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.46]); Tue, 23 Jul 2019 11:30:59 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 07/23/19 05:58, Bob Feng wrote: > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875 > > In order to improve the build performance, we implemented > multiple-processes AutoGen. This change will reduce 20% time > for AutoGen phase. > > The design document can be got from: > https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf > > This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages > under Edk2 repository and intel client and server platforms. > > V3: > 1. Fixed incremental build issue. > 2. Set AutoGen worker number to be align with "-n THREADNUMBER" > 3. Enable block log queue. > V2: > 1. The first version missed autogen related commit > from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit > back. > 2. Move CreateAsBuildInf into AutoGenWorker process > 3. Save GlobalVar__.bin to build folder. > 4. Regenerate patches based on master bb824f685d Tested this series on top of commit cf2d8d4978e8. (1) The GlobalVar*.bin files are now located under the Build directory, so that's OK. (2) Rebuilding OVMF (in a standalone edk2 clone, not as a QEMU submodule) with different -D flags works fine now. The fw binary hash matches a similar rebuild without the series applied. (3) The blocking log queue works fine with Ctrl-S / Ctrl-Q. Please see my note on the queue size (currently: constant 60) under patch#10. I think the factor 10 that I suggested makes sense as well, because I tested V3 with 4 threads (meaning room for 60/4=15 messages per thread), and I felt that the compilation threads got blocked just a bit too late after I pressed Ctrl-S. Therefore, a factor of 10 (rather than the effective 15 that I tested now) will make for a more direct user experience. (4) I tested "-n1" and "-n2"; they seem to work fine. (5) Restarting a build (incrementally) after Ctrl-C seems to work fine. (6) The build seems to notice changes to .c files. OK. (7) This version works fine as part of QEMU's edk2 submodule, with "make -j 4 -C roms efi". All six firmware binaries built there are functional. * For patches 01 through 09: Tested-by: Laszlo Ersek * For patch 10: I'd like to test v4, with the proportional log queue size, and give my T-b then. * If you change other patches in v4 (01-09), then please drop my Tested-by from those. * Note that I didn't test: - python3 - the "--hash" option - the "--binary-destination" / "--binary-source" options (I'm not sure what those options do in the first place) - the "--re-parse" option - the "--no-cache" option - the "--genfds-multi-thread" option Hopefully others can test those. Thanks Laszlo