From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=209.132.183.28; helo=mx1.redhat.com; envelope-from=lersek@redhat.com; receiver=edk2-devel@lists.01.org Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id BEB0421962301 for ; Tue, 22 Jan 2019 04:54:37 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D471699D29; Tue, 22 Jan 2019 12:54:36 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-61.rdu2.redhat.com [10.10.120.61]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0C505D6A9; Tue, 22 Jan 2019 12:54:32 +0000 (UTC) To: "Gao, Liming" Cc: edk2-devel-01 , =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= From: Laszlo Ersek Message-ID: Date: Tue, 22 Jan 2019 13:54:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 22 Jan 2019 12:54:36 +0000 (UTC) Subject: parallelism in the module-level, generated GNUmakefile's X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2019 12:54:39 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Hi Liming, I'd like to ask a question about parallel ("multi threaded") make. The use case is the following: (1) The edk2 "build" BaseTools utility itself is invoked from a *higher-level* Makefile that is outside of the edk2 project. (2) The outer make process is invoked with "make -jN". (3) The "build" utility is invoked in isolation. That is, at most *one* instance of the "build" utility runs at any given time, despite the outer make receiving "-jN". This guarantees that there is no corruption in $WORKSPACE/Conf and so on. (4) The "build" utility is invoked with the "-m" option, to build a single module INF. (5) The "-n" parameter of the "build" utility is not used (and it would be useless anyway, since "-n" only parallelizes the build *between* INF files, and not *within* INF files, and here we run "build" on a single INF file, with "-m"). As a result, when the "build" utility invokes the "make" program, on the GNUmakefile that was generated by "GenMake.py", there is already a higher-level make process that is the (indirect) ancestor of the new make process. This is important because the higher level make process sets some options in the MAKEFLAGS environment variable, and the inner make process inherits those: outer make --> sets MAKEFLAGS in the environment | | build \ -a ARCH \ -p PLATFORM_DSC \ -t TOOLCHAIN_TAG \ -b TARGET \ -m MODULE_INF | | inner make --> inheirts MAKEFLAGS in the environment Due to (2), the *inner* make inherits the following MAKEFLAGS: --jobserver-fds=3,4 -j The "-j" and "--jobserver-fds" options cause the inner make -- which is started by "build" -- to communicate with the *outer* make process. The goal of this communication is to ensure that no more than 4 jobs are active at any given time. The important part is that, if the "job server" (provided by the outer make) *allows* the inner make to run two or more recipes in parallel, from the generated GNUMakefile, then the inner make *will do that*. It will launch multiple "nasm", "gcc", "iasl" etc commands in parallel. In my testing, this happens to work fine -- the build completes okay. My question is: - Is this behavior *intentional*? - In other words: does "GenMake.py" *intentionally* generate "GNUmakefile" so that it is compatible with "make -j"? Or does it work only by chance, on my end? If the answer is "by chance only", then that is 100% fine with me; I am not requesting that we add "make -j" compatibility to "GenMake.py". Instead, I'd suggest that we expose this limitation *explicitly* in "GenMake.py". For that, the ".NOTPARALLEL" target can be used, and it really has to be placed into the generated GNUMakefile: https://www.gnu.org/software/make/manual/make.html#index-_002eNOTPARALLEL If we do that, then the inner make will safely ignore the inherited "-j" option, for the targets that are listed as prerequisites of .NOTPARALLEL. And then the problem becomes: how can we collect the full set of GNUMakefile targest (so that we can list them all in .NOTPARALLEL)? There are many types of build products, from iasl, nasm, VfrCompile, etc, beyond the most common "object file from C source". Thanks! Laszlo