Re: [edk2-devel] [PATCH] OvmfPkg: enable multiprocessor builds when using build.sh

["Laszlo Ersek" <lersek@redhat.com>]

On 07/23/19 10:05, Jordan Justen wrote:
> On 2019-07-23 00:44:06, Laszlo Ersek wrote:
>> On 07/23/19 02:00, Rebecca Cran wrote:
>>> On 2019-07-22 17:14, Jordan Justen wrote:
>>>>
>>>> I was suggesting that if they didn't specify -n as a param to
>>>> build.sh, then build.sh should not send -n to the edk2 build command.
>>>> The effect would be for the edk2 build command to check
>>>> Conf/target.txt. By default, I think target.txt will not set
>>>> THREADNUMBER, so "0" would still be the result.
>>>>
>>>> Yet, it would give them the option to set it in Conf/target.txt.
>>>> Today, since we always use the -n param, target.txt is always ignored
>>>> for this parameter.
>>>
>>>
>>> On a related topic, I wonder if we should add a "-j" parameter if we
>>> build BaseTools for users (e.g. "make -j4 -C BaseTools")? I've found
>>> that it can be pretty slow without it: on my system adding -j4 reduces
>>> build time from 55 seconds to 15. Going higher doesn't seem to produce
>>> much more benefit: -j32 (on a ThreadRipper system) reduces it to 12 seconds.
>>>
>>>
>>
>> Passing
>>
>>   -j $(getconf _NPROCESSORS_ONLN)
>>
>> to "make" (for building BaseTools) makes sense, IMO.
> 
> I guess the concern might be that we'll be running a bunch of
> make invocations in parallel, each trying to spawn a compilation for
> each thread. O(n^2) compilations. :)
> 
> In the make man-page for -j: "When make invokes a sub-make, all
> instances of make will coordinate to run the specified number of jobs
> at a time;", but I'm not sure if that's how `build -n` is implemented.
> (With make...)
> 
> Since python writes the makefiles, it could be used instead of
> getconf, right?
> 
> What we need is someone to make the ninja-build backend for BaseTools.
> :)

Wait, there are two separate topics here.


(1) The "make" command we are discussing here is responsible solely for
building the host-native BaseTools binaries, from the C and CPP sources
under BaseTools:

  make -C $WORKSPACE/BaseTools

This "make" command is not an ancestor of the actual firmware build.


(2) Regarding the invocation of the "build" utility from an outer
"make", such that you end up with "make -j" --> build --> "make":

That's something we don't do in edk2 per se, but we *do* do it in edk2's
bundling in QEMU. It raised some challenges, but ultimately, the
innermost "make" processes will inherit such an environment from the
outermost "make" that will enable a successful coordination between
them, regardless of "-n 0" used with "build" in the middle.

Basically, even if "build -n 0" starts twenty inner "make" processes,
the job server in the outer "make" will ensure that no more than, say,
4, inner "make" processes are active at the same time. (Assuming 20
logical processors in the system, and "-j4" passed to the outer "make".)

A related TianoCore BZ is:

  https://bugzilla.tianocore.org/show_bug.cgi?id=1607

--*--

Anyway, for "OvmfPkg/build.sh", item (2) is irrelevant; so changing

  make -C $WORKSPACE/BaseTools

to

  make -C $WORKSPACE/BaseTools -j $(getconf _NPROCESSORS_ONLN)

would be useful and safe.

Thanks
Laszlo