Re: parallelism in the module-level, generated GNUmakefile's

["Gao, Liming" <liming.gao@intel.com>]

Laszlo:
  By design, BaseTools generates GNUMakefile with the complete dependency. So, the generated GNUMakefile should support parallelism run. But, I don't verify this functionality. Have you found any limitation to forbid parallelism run in module GNUMakefile?

Thanks
Liming
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Laszlo Ersek
> Sent: Tuesday, January 22, 2019 8:55 PM
> To: Gao, Liming <liming.gao@intel.com>
> Cc: edk2-devel-01 <edk2-devel@lists.01.org>
> Subject: [edk2] parallelism in the module-level, generated GNUmakefile's
> 
> Hi Liming,
> 
> I'd like to ask a question about parallel ("multi threaded") make.
> 
> The use case is the following:
> 
> (1) The edk2 "build" BaseTools utility itself is invoked from a
> *higher-level* Makefile that is outside of the edk2 project.
> 
> (2) The outer make process is invoked with "make -jN".
> 
> (3) The "build" utility is invoked in isolation. That is, at most *one*
> instance of the "build" utility runs at any given time, despite the
> outer make receiving "-jN". This guarantees that there is no corruption
> in $WORKSPACE/Conf and so on.
> 
> (4) The "build" utility is invoked with the "-m" option, to build a
> single module INF.
> 
> (5) The "-n" parameter of the "build" utility is not used (and it would
> be useless anyway, since "-n" only parallelizes the build *between* INF
> files, and not *within* INF files, and here we run "build" on a single
> INF file, with "-m").
> 
> 
> As a result, when the "build" utility invokes the "make" program, on the
> GNUmakefile that was generated by "GenMake.py", there is already a
> higher-level make process that is the (indirect) ancestor of the new
> make process. This is important because the higher level make process
> sets some options in the MAKEFLAGS environment variable, and the inner
> make process inherits those:
> 
>            outer make --> sets MAKEFLAGS in the environment
>                |
>                |
>            build \
>              -a ARCH \
>              -p PLATFORM_DSC \
>              -t TOOLCHAIN_TAG \
>              -b TARGET \
>              -m MODULE_INF
>                |
>                |
>            inner make --> inheirts MAKEFLAGS in the environment
> 
> Due to (2), the *inner* make inherits the following MAKEFLAGS:
> 
>   --jobserver-fds=3,4 -j
> 
> The "-j" and "--jobserver-fds" options cause the inner make -- which is
> started by "build" -- to communicate with the *outer* make process. The
> goal of this communication is to ensure that no more than 4 jobs are
> active at any given time.
> 
> The important part is that, if the "job server" (provided by the outer
> make) *allows* the inner make to run two or more recipes in parallel,
> from the generated GNUMakefile, then the inner make *will do that*. It
> will launch multiple "nasm", "gcc", "iasl" etc commands in parallel.
> 
> In my testing, this happens to work fine -- the build completes okay. My
> question is:
> 
> - Is this behavior *intentional*?
> 
> - In other words: does "GenMake.py" *intentionally* generate
> "GNUmakefile" so that it is compatible with "make -j"? Or does it work
> only by chance, on my end?
> 
> 
> If the answer is "by chance only", then that is 100% fine with me; I am
> not requesting that we add "make -j" compatibility to "GenMake.py".
> Instead, I'd suggest that we expose this limitation *explicitly* in
> "GenMake.py". For that, the ".NOTPARALLEL" target can be used, and it
> really has to be placed into the generated GNUMakefile:
> 
> https://www.gnu.org/software/make/manual/make.html#index-_002eNOTPARALLEL
> 
> If we do that, then the inner make will safely ignore the inherited "-j"
> option, for the targets that are listed as prerequisites of .NOTPARALLEL.
> 
> And then the problem becomes: how can we collect the full set of
> GNUMakefile targest (so that we can list them all in .NOTPARALLEL)?
> There are many types of build products, from iasl, nasm, VfrCompile,
> etc, beyond the most common "object file from C source".
> 
> Thanks!
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel