From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.120; helo=mga04.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 48839211B6C1A for ; Wed, 23 Jan 2019 08:36:58 -0800 (PST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Jan 2019 08:36:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,512,1539673200"; d="scan'208";a="110452102" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga006.jf.intel.com with ESMTP; 23 Jan 2019 08:36:58 -0800 Received: from fmsmsx154.amr.corp.intel.com (10.18.116.70) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.408.0; Wed, 23 Jan 2019 08:36:57 -0800 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by FMSMSX154.amr.corp.intel.com (10.18.116.70) with Microsoft SMTP Server (TLS) id 14.3.408.0; Wed, 23 Jan 2019 08:36:57 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.159]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.147]) with mapi id 14.03.0415.000; Thu, 24 Jan 2019 00:36:55 +0800 From: "Gao, Liming" To: Laszlo Ersek CC: edk2-devel-01 Thread-Topic: [edk2] parallelism in the module-level, generated GNUmakefile's Thread-Index: AQHUslGo0w6T35ZOBk6Yj9LMWG2olaW9DFZg Date: Wed, 23 Jan 2019 16:36:54 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E3C0E74@SHSMSX104.ccr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYzExOWY3NWItMDJmYy00ZjQxLWJiYmEtODM5YmM5MmMxOWY1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoienFZWU0xUzc1bjhVMmRwR0NJVzRIMm9hQVd4VXRcLzN1cmdTWkIzR3F2YWU1VGsxaVpaVjJuZ1lwclV6OFhkb1AifQ== dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: parallelism in the module-level, generated GNUmakefile's X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2019 16:36:59 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Laszlo: By design, BaseTools generates GNUMakefile with the complete dependency. = So, the generated GNUMakefile should support parallelism run. But, I don't = verify this functionality. Have you found any limitation to forbid parallel= ism run in module GNUMakefile? Thanks Liming > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of La= szlo Ersek > Sent: Tuesday, January 22, 2019 8:55 PM > To: Gao, Liming > Cc: edk2-devel-01 > Subject: [edk2] parallelism in the module-level, generated GNUmakefile's >=20 > Hi Liming, >=20 > I'd like to ask a question about parallel ("multi threaded") make. >=20 > The use case is the following: >=20 > (1) The edk2 "build" BaseTools utility itself is invoked from a > *higher-level* Makefile that is outside of the edk2 project. >=20 > (2) The outer make process is invoked with "make -jN". >=20 > (3) The "build" utility is invoked in isolation. That is, at most *one* > instance of the "build" utility runs at any given time, despite the > outer make receiving "-jN". This guarantees that there is no corruption > in $WORKSPACE/Conf and so on. >=20 > (4) The "build" utility is invoked with the "-m" option, to build a > single module INF. >=20 > (5) The "-n" parameter of the "build" utility is not used (and it would > be useless anyway, since "-n" only parallelizes the build *between* INF > files, and not *within* INF files, and here we run "build" on a single > INF file, with "-m"). >=20 >=20 > As a result, when the "build" utility invokes the "make" program, on the > GNUmakefile that was generated by "GenMake.py", there is already a > higher-level make process that is the (indirect) ancestor of the new > make process. This is important because the higher level make process > sets some options in the MAKEFLAGS environment variable, and the inner > make process inherits those: >=20 > outer make --> sets MAKEFLAGS in the environment > | > | > build \ > -a ARCH \ > -p PLATFORM_DSC \ > -t TOOLCHAIN_TAG \ > -b TARGET \ > -m MODULE_INF > | > | > inner make --> inheirts MAKEFLAGS in the environment >=20 > Due to (2), the *inner* make inherits the following MAKEFLAGS: >=20 > --jobserver-fds=3D3,4 -j >=20 > The "-j" and "--jobserver-fds" options cause the inner make -- which is > started by "build" -- to communicate with the *outer* make process. The > goal of this communication is to ensure that no more than 4 jobs are > active at any given time. >=20 > The important part is that, if the "job server" (provided by the outer > make) *allows* the inner make to run two or more recipes in parallel, > from the generated GNUMakefile, then the inner make *will do that*. It > will launch multiple "nasm", "gcc", "iasl" etc commands in parallel. >=20 > In my testing, this happens to work fine -- the build completes okay. My > question is: >=20 > - Is this behavior *intentional*? >=20 > - In other words: does "GenMake.py" *intentionally* generate > "GNUmakefile" so that it is compatible with "make -j"? Or does it work > only by chance, on my end? >=20 >=20 > If the answer is "by chance only", then that is 100% fine with me; I am > not requesting that we add "make -j" compatibility to "GenMake.py". > Instead, I'd suggest that we expose this limitation *explicitly* in > "GenMake.py". For that, the ".NOTPARALLEL" target can be used, and it > really has to be placed into the generated GNUMakefile: >=20 > https://www.gnu.org/software/make/manual/make.html#index-_002eNOTPARALLEL >=20 > If we do that, then the inner make will safely ignore the inherited "-j" > option, for the targets that are listed as prerequisites of .NOTPARALLEL. >=20 > And then the problem becomes: how can we collect the full set of > GNUMakefile targest (so that we can list them all in .NOTPARALLEL)? > There are many types of build products, from iasl, nasm, VfrCompile, > etc, beyond the most common "object file from C source". >=20 > Thanks! > Laszlo > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel