Re: EDK2 build issues

[Andrew Fish <afish@apple.com>]

> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L <nathaniel.l.desimone@intel.com> wrote:
> 
> Hi Liming,
> 
> I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j <n>" (where n > 1). I have filed the bug in Bugzilla:
> 

I think there are a few issues like that in BaseTools. We have a parallel makefile and we had to turn off parallelism when calling the BaseTools. 

In general the edk2 build system fights against "make -j <n>" since the Python build command is trying to do the threading and invoking make in parallel. Given we ported EDK to parallel build, the Python parallel edk2 build is a lot slower than the EDK make parallel build, even when adjusting for the extra work the edk2 does. 

I even noticed a 1,000,000 calls to regex in Python during the ... phase of the build. 

I think the correct short term fix may be to put .NOTPARALLEL: in the BaseTools GNUmakefile given it is constructed in a way that it does not support parallelism. 

Thanks,

Andrew Fish



> https://bugzilla.tianocore.org/show_bug.cgi?id=786
> 
> For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in Bugzilla as well?
> 
> Thanks,
> 
> Nate
> 
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming
> Sent: Monday, November 13, 2017 5:46 PM
> To: Aaron Durbin <adurbin@google.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] EDK2 build issues
> 
> I add my comments. 
> 
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of 
>> Aaron Durbin
>> Sent: Tuesday, November 14, 2017 1:38 AM
>> To: Gao, Liming <liming.gao@intel.com>
>> Cc: edk2-devel@lists.01.org
>> Subject: Re: [edk2] EDK2 build issues
>> 
>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote:
>>> Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them.
>>> 
>>> 1. BaseTools Soure tools are compiled in serial instead of parallel. 
>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be
>> built first. I agree that this way takes some time to compile 
>> BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch.
>> 
>> If the depedencies are correctly met in the makefiles then why are 
>> they built serially? It sounds like you understand the dependencies so 
>> why aren't those explicitly noted so one can build in parallel?
>> 
> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)?
> Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. 
> 
>>> 2. BaseTools C source are compiled to the executable files. They run 
>>> in host machine. Here ARCH is for the executable files those can
>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command.
>> 
>> ARCH != host is my point. Why are those being conflated? Or are you 
>> saying edk2 tools define ARCH to be what the host is?
>> 
> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the different purpose. 
> 
>>> 3. There are more GCC compiler distribution. We try to use the 
>>> generic options in GCC tool chain. If you find the option doesn't 
>>> work
>> on your GCC compiler, please report it. And, we also notice 
>> tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community.
>>> 
>>>> -----Original Message-----
>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf 
>>>> Of Aaron Durbin
>>>> Sent: Saturday, November 11, 2017 6:27 AM
>>>> To: edk2-devel@lists.01.org
>>>> Subject: [edk2] EDK2 build issues
>>>> 
>>>> Hello,
>>>> 
>>>> Here are some observations I've encountered trying to utilize edk2 
>>>> for certain builds. Part of the problem seems to be with implicit 
>>>> assumptions in how edk2 is used. I'm trying to build things using 
>>>> edk2 from a clean enviroment on an automated builder. i.e. there 
>>>> isn't a workspace that exists on one persons computer for the 
>>>> lifetime of development.
>>>> 
>>>> 1. BaseTools can't build in parallel. The tests are racey which 
>>>> result in test failures. Because of this one has to build these in 
>>>> serial instead of in parallel.
>>>> 
>>>> ===========================================================
>>>> ===========
>>>> FAIL: testHelp (TianoCompress.Tests)
>>>> --------------------------------------------------------------------
>>>> --
>>>> Traceback (most recent call last):
>>>> File 
>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>> line 34, in testHelp
>>>>   self.assertTrue(result == 0)
>>>> AssertionError: False is not true
>>>> 
>>>> ===========================================================
>>>> ===========
>>>> FAIL: testRandomDataCycles (TianoCompress.Tests)
>>>> --------------------------------------------------------------------
>>>> --
>>>> Traceback (most recent call last):
>>>> File 
>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>> line 65, in testRandomDataCycles
>>>>   self.compressionTestCycle(data)
>>>> File 
>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py",
>>>> line 44, in compressionTestCycle
>>>>   self.assertTrue(result == 0)
>>>> AssertionError: False is not true
>>>> 
>>>> In addition, it seems compilation even breaks trying to build a parser:
>>>> 
>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token  {^M  
>>>> ^
>>>> VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)':
>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any 
>>>> field named 'VfrLexer'
>>>>  CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M
>>>>                                   ^
>>>> 
>>>> This just slows down builds needing to things serially.
>>>> 
>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm 
>>>> not sure of the origins, but ARCH seems like a complete misnomer for 
>>>> the *host* you are trying to build tools on -- not the target. 
>>>> Trying to incorporate edk2 builds into a portage environment 
>>>> effectively breaks because of this as ARCH refers to target 
>>>> architecture -- not host builder's ARCH.
>>>> 
>>>> 3. This more of an observation, but the tools definition seems to 
>>>> make quite the leap on how consistent compilers are of a certain version.
>>>> e.g. GCC 4.9 can be built with all kinds of default options that 
>>>> edk2 implicitly assumes are set based on some distribution's default flags?
>>>> And in order to extend toolchain support one needs to create a 
>>>> series of entries associated with a certain family. Barrier to entry 
>>>> is pretty high in teasing out where to put things in the ~8000 line 
>>>> tools_def.template.
>>>> 
>>>> Thanks for the consideration.
>>>> 
>>>> -Aaron
>>>> _______________________________________________
>>>> edk2-devel mailing list
>>>> edk2-devel@lists.01.org
>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel