Re: EDK2 build issues

["Gao, Liming" <liming.gao@intel.com>]

Andrew:
  To compile BaseTools C tools is a separate step. It happens before build. I review BaseTools C tool top GNUmakefile. It can be enhanced to support make -j N that has the better performance. I will provide my patch for code review. 

Thanks
Liming
>-----Original Message-----
>From: afish@apple.com [mailto:afish@apple.com]
>Sent: Thursday, November 23, 2017 8:02 AM
>To: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>
>Cc: Gao, Liming <liming.gao@intel.com>; Aaron Durbin
><adurbin@google.com>; edk2-devel@lists.01.org
>Subject: Re: [edk2] EDK2 build issues
>
>
>> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L
><nathaniel.l.desimone@intel.com> wrote:
>>
>> Hi Liming,
>>
>> I chatted with Aaron on the phone today. The VfrCompiler race condition
>was discovered using "make -j <n>" (where n > 1). I have filed the bug in
>Bugzilla:
>>
>
>I think there are a few issues like that in BaseTools. We have a parallel
>makefile and we had to turn off parallelism when calling the BaseTools.
>
>In general the edk2 build system fights against "make -j <n>" since the Python
>build command is trying to do the threading and invoking make in parallel.
>Given we ported EDK to parallel build, the Python parallel edk2 build is a lot
>slower than the EDK make parallel build, even when adjusting for the extra
>work the edk2 does.
>
>I even noticed a 1,000,000 calls to regex in Python during the ... phase of the
>build.
>
>I think the correct short term fix may be to put .NOTPARALLEL: in the
>BaseTools GNUmakefile given it is constructed in a way that it does not
>support parallelism.
>
>Thanks,
>
>Andrew Fish
>
>
>
>> https://bugzilla.tianocore.org/show_bug.cgi?id=786
>>
>> For the ARCH environment variable, we brainstormed two possible new
>names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable
>request in Bugzilla as well?
>>
>> Thanks,
>>
>> Nate
>>
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>Gao, Liming
>> Sent: Monday, November 13, 2017 5:46 PM
>> To: Aaron Durbin <adurbin@google.com>
>> Cc: edk2-devel@lists.01.org
>> Subject: Re: [edk2] EDK2 build issues
>>
>> I add my comments.
>>
>>> -----Original Message-----
>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>>> Aaron Durbin
>>> Sent: Tuesday, November 14, 2017 1:38 AM
>>> To: Gao, Liming <liming.gao@intel.com>
>>> Cc: edk2-devel@lists.01.org
>>> Subject: Re: [edk2] EDK2 build issues
>>>
>>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com>
>wrote:
>>>> Yes. BaseTools needs some environments. Before build BaseTools, we
>need to call edkset.sh to setup them.
>>>>
>>>> 1. BaseTools Soure tools are compiled in serial instead of parallel.
>>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be
>>> built first. I agree that this way takes some time to compile
>>> BaseTools source. I have one proposal to compile C source tools with
>multiple thread. I can share my draft patch.
>>>
>>> If the depedencies are correctly met in the makefiles then why are
>>> they built serially? It sounds like you understand the dependencies so
>>> why aren't those explicitly noted so one can build in parallel?
>>>
>> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could
>you submit this issue in bugzillar (https://bugzilla.tianocore.org/)?
>> Besides, do you use make -j option to enable Parallel Execution? I want to
>know how to verify it.
>>
>>>> 2. BaseTools C source are compiled to the executable files. They run
>>>> in host machine. Here ARCH is for the executable files those can
>>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects
>ARCH based on 'uname -m' command.
>>>
>>> ARCH != host is my point. Why are those being conflated? Or are you
>>> saying edk2 tools define ARCH to be what the host is?
>>>
>> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too
>generic. In your environment, ARCH may be defined for the different purpose.
>>
>>>> 3. There are more GCC compiler distribution. We try to use the
>>>> generic options in GCC tool chain. If you find the option doesn't
>>>> work
>>> on your GCC compiler, please report it. And, we also notice
>>> tools_def.txt is too big to be hard to be maintain. We have one proposal to
>simplify it. I will send this RFC to edk2 community.
>>>>
>>>>> -----Original Message-----
>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
>>>>> Of Aaron Durbin
>>>>> Sent: Saturday, November 11, 2017 6:27 AM
>>>>> To: edk2-devel@lists.01.org
>>>>> Subject: [edk2] EDK2 build issues
>>>>>
>>>>> Hello,
>>>>>
>>>>> Here are some observations I've encountered trying to utilize edk2
>>>>> for certain builds. Part of the problem seems to be with implicit
>>>>> assumptions in how edk2 is used. I'm trying to build things using
>>>>> edk2 from a clean enviroment on an automated builder. i.e. there
>>>>> isn't a workspace that exists on one persons computer for the
>>>>> lifetime of development.
>>>>>
>>>>> 1. BaseTools can't build in parallel. The tests are racey which
>>>>> result in test failures. Because of this one has to build these in
>>>>> serial instead of in parallel.
>>>>>
>>>>>
>===========================================================
>>>>> ===========
>>>>> FAIL: testHelp (TianoCompress.Tests)
>>>>> --------------------------------------------------------------------
>>>>> --
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>> line 34, in testHelp
>>>>>   self.assertTrue(result == 0)
>>>>> AssertionError: False is not true
>>>>>
>>>>>
>===========================================================
>>>>> ===========
>>>>> FAIL: testRandomDataCycles (TianoCompress.Tests)
>>>>> --------------------------------------------------------------------
>>>>> --
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>> line 65, in testRandomDataCycles
>>>>>   self.compressionTestCycle(data)
>>>>> File
>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>>> line 44, in compressionTestCycle
>>>>>   self.assertTrue(result == 0)
>>>>> AssertionError: False is not true
>>>>>
>>>>> In addition, it seems compilation even breaks trying to build a parser:
>>>>>
>>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token  {^M
>>>>> ^
>>>>> VfrSyntax.cpp: In constructor
>'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)':
>>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any
>>>>> field named 'VfrLexer'
>>>>>  CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M
>>>>>                                   ^
>>>>>
>>>>> This just slows down builds needing to things serially.
>>>>>
>>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm
>>>>> not sure of the origins, but ARCH seems like a complete misnomer for
>>>>> the *host* you are trying to build tools on -- not the target.
>>>>> Trying to incorporate edk2 builds into a portage environment
>>>>> effectively breaks because of this as ARCH refers to target
>>>>> architecture -- not host builder's ARCH.
>>>>>
>>>>> 3. This more of an observation, but the tools definition seems to
>>>>> make quite the leap on how consistent compilers are of a certain version.
>>>>> e.g. GCC 4.9 can be built with all kinds of default options that
>>>>> edk2 implicitly assumes are set based on some distribution's default
>flags?
>>>>> And in order to extend toolchain support one needs to create a
>>>>> series of entries associated with a certain family. Barrier to entry
>>>>> is pretty high in teasing out where to put things in the ~8000 line
>>>>> tools_def.template.
>>>>>
>>>>> Thanks for the consideration.
>>>>>
>>>>> -Aaron
>>>>> _______________________________________________
>>>>> edk2-devel mailing list
>>>>> edk2-devel@lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel