From: Andrew Fish <afish@apple.com>
To: "Gao, Liming" <liming.gao@intel.com>
Cc: "Desimone, Nathaniel L" <nathaniel.l.desimone@intel.com>,
Aaron Durbin <adurbin@google.com>,
"edk2-devel@lists.01.org" <edk2-devel@lists.01.org>
Subject: Re: EDK2 build issues
Date: Mon, 27 Nov 2017 08:18:48 -0800 [thread overview]
Message-ID: <1BBD2474-00CD-4CCC-B30A-A3141EDE38CA@apple.com> (raw)
In-Reply-To: <4A89E2EF3DFEDB4C8BFDE51014F606A14E1822EE@SHSMSX104.ccr.corp.intel.com>
Liming,
We have a top level GNUmakefile vs. having script files to build targets. Thus building BaseTools, or using build are subtasks of this higher level makefile.
Looks like you have added some fixes to the BaseTools!
Thanks,
Andrew Fish
> On Nov 22, 2017, at 9:28 PM, Gao, Liming <liming.gao@intel.com> wrote:
>
> Andrew:
> To compile BaseTools C tools is a separate step. It happens before build. I review BaseTools C tool top GNUmakefile. It can be enhanced to support make -j N that has the better performance. I will provide my patch for code review.
>
> Thanks
> Liming
>> -----Original Message-----
>> From: afish@apple.com [mailto:afish@apple.com]
>> Sent: Thursday, November 23, 2017 8:02 AM
>> To: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>
>> Cc: Gao, Liming <liming.gao@intel.com>; Aaron Durbin
>> <adurbin@google.com>; edk2-devel@lists.01.org
>> Subject: Re: [edk2] EDK2 build issues
>>
>>
>>> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L
>> <nathaniel.l.desimone@intel.com> wrote:
>>>
>>> Hi Liming,
>>>
>>> I chatted with Aaron on the phone today. The VfrCompiler race condition
>> was discovered using "make -j <n>" (where n > 1). I have filed the bug in
>> Bugzilla:
>>>
>>
>> I think there are a few issues like that in BaseTools. We have a parallel
>> makefile and we had to turn off parallelism when calling the BaseTools.
>>
>> In general the edk2 build system fights against "make -j <n>" since the Python
>> build command is trying to do the threading and invoking make in parallel.
>> Given we ported EDK to parallel build, the Python parallel edk2 build is a lot
>> slower than the EDK make parallel build, even when adjusting for the extra
>> work the edk2 does.
>>
>> I even noticed a 1,000,000 calls to regex in Python during the ... phase of the
>> build.
>>
>> I think the correct short term fix may be to put .NOTPARALLEL: in the
>> BaseTools GNUmakefile given it is constructed in a way that it does not
>> support parallelism.
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>
>>
>>> https://bugzilla.tianocore.org/show_bug.cgi?id=786
>>>
>>> For the ARCH environment variable, we brainstormed two possible new
>> names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable
>> request in Bugzilla as well?
>>>
>>> Thanks,
>>>
>>> Nate
>>>
>>> -----Original Message-----
>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>> Gao, Liming
>>> Sent: Monday, November 13, 2017 5:46 PM
>>> To: Aaron Durbin <adurbin@google.com>
>>> Cc: edk2-devel@lists.01.org
>>> Subject: Re: [edk2] EDK2 build issues
>>>
>>> I add my comments.
>>>
>>>> -----Original Message-----
>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>>>> Aaron Durbin
>>>> Sent: Tuesday, November 14, 2017 1:38 AM
>>>> To: Gao, Liming <liming.gao@intel.com>
>>>> Cc: edk2-devel@lists.01.org
>>>> Subject: Re: [edk2] EDK2 build issues
>>>>
>>>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com>
>> wrote:
>>>>> Yes. BaseTools needs some environments. Before build BaseTools, we
>> need to call edkset.sh to setup them.
>>>>>
>>>>> 1. BaseTools Soure tools are compiled in serial instead of parallel.
>>>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be
>>>> built first. I agree that this way takes some time to compile
>>>> BaseTools source. I have one proposal to compile C source tools with
>> multiple thread. I can share my draft patch.
>>>>
>>>> If the depedencies are correctly met in the makefiles then why are
>>>> they built serially? It sounds like you understand the dependencies so
>>>> why aren't those explicitly noted so one can build in parallel?
>>>>
>>> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could
>> you submit this issue in bugzillar (https://bugzilla.tianocore.org/)?
>>> Besides, do you use make -j option to enable Parallel Execution? I want to
>> know how to verify it.
>>>
>>>>> 2. BaseTools C source are compiled to the executable files. They run
>>>>> in host machine. Here ARCH is for the executable files those can
>>>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects
>> ARCH based on 'uname -m' command.
>>>>
>>>> ARCH != host is my point. Why are those being conflated? Or are you
>>>> saying edk2 tools define ARCH to be what the host is?
>>>>
>>> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too
>> generic. In your environment, ARCH may be defined for the different purpose.
>>>
>>>>> 3. There are more GCC compiler distribution. We try to use the
>>>>> generic options in GCC tool chain. If you find the option doesn't
>>>>> work
>>>> on your GCC compiler, please report it. And, we also notice
>>>> tools_def.txt is too big to be hard to be maintain. We have one proposal to
>> simplify it. I will send this RFC to edk2 community.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
>>>>>> Of Aaron Durbin
>>>>>> Sent: Saturday, November 11, 2017 6:27 AM
>>>>>> To: edk2-devel@lists.01.org
>>>>>> Subject: [edk2] EDK2 build issues
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Here are some observations I've encountered trying to utilize edk2
>>>>>> for certain builds. Part of the problem seems to be with implicit
>>>>>> assumptions in how edk2 is used. I'm trying to build things using
>>>>>> edk2 from a clean enviroment on an automated builder. i.e. there
>>>>>> isn't a workspace that exists on one persons computer for the
>>>>>> lifetime of development.
>>>>>>
>>>>>> 1. BaseTools can't build in parallel. The tests are racey which
>>>>>> result in test failures. Because of this one has to build these in
>>>>>> serial instead of in parallel.
>>>>>>
>>>>>>
>> ===========================================================
>>>>>> ===========
>>>>>> FAIL: testHelp (TianoCompress.Tests)
>>>>>> --------------------------------------------------------------------
>>>>>> --
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>>> line 34, in testHelp
>>>>>> self.assertTrue(result == 0)
>>>>>> AssertionError: False is not true
>>>>>>
>>>>>>
>> ===========================================================
>>>>>> ===========
>>>>>> FAIL: testRandomDataCycles (TianoCompress.Tests)
>>>>>> --------------------------------------------------------------------
>>>>>> --
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>>> line 65, in testRandomDataCycles
>>>>>> self.compressionTestCycle(data)
>>>>>> File
>>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>>> line 44, in compressionTestCycle
>>>>>> self.assertTrue(result == 0)
>>>>>> AssertionError: False is not true
>>>>>>
>>>>>> In addition, it seems compilation even breaks trying to build a parser:
>>>>>>
>>>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M
>>>>>> ^
>>>>>> VfrSyntax.cpp: In constructor
>> 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)':
>>>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any
>>>>>> field named 'VfrLexer'
>>>>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M
>>>>>> ^
>>>>>>
>>>>>> This just slows down builds needing to things serially.
>>>>>>
>>>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm
>>>>>> not sure of the origins, but ARCH seems like a complete misnomer for
>>>>>> the *host* you are trying to build tools on -- not the target.
>>>>>> Trying to incorporate edk2 builds into a portage environment
>>>>>> effectively breaks because of this as ARCH refers to target
>>>>>> architecture -- not host builder's ARCH.
>>>>>>
>>>>>> 3. This more of an observation, but the tools definition seems to
>>>>>> make quite the leap on how consistent compilers are of a certain version.
>>>>>> e.g. GCC 4.9 can be built with all kinds of default options that
>>>>>> edk2 implicitly assumes are set based on some distribution's default
>> flags?
>>>>>> And in order to extend toolchain support one needs to create a
>>>>>> series of entries associated with a certain family. Barrier to entry
>>>>>> is pretty high in teasing out where to put things in the ~8000 line
>>>>>> tools_def.template.
>>>>>>
>>>>>> Thanks for the consideration.
>>>>>>
>>>>>> -Aaron
>>>>>> _______________________________________________
>>>>>> edk2-devel mailing list
>>>>>> edk2-devel@lists.01.org
>>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>>> _______________________________________________
>>>> edk2-devel mailing list
>>>> edk2-devel@lists.01.org
>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>
prev parent reply other threads:[~2017-11-27 16:14 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-11-10 22:26 EDK2 build issues Aaron Durbin
2017-11-13 2:15 ` Gao, Liming
2017-11-13 17:37 ` Aaron Durbin
2017-11-14 1:46 ` Gao, Liming
2017-11-16 21:37 ` Desimone, Nathaniel L
2017-11-17 8:45 ` Gao, Liming
2017-11-22 23:48 ` Desimone, Nathaniel L
2017-11-23 0:01 ` Andrew Fish
2017-11-23 5:28 ` Gao, Liming
2017-11-27 16:18 ` Andrew Fish [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1BBD2474-00CD-4CCC-B30A-A3141EDE38CA@apple.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox