Re: EDK2 build issues

["Gao, Liming" <liming.gao@intel.com>]

Nate:
  OK. Please submit the tracker on ARCH request. 

Thanks
Liming
> -----Original Message-----
> From: Desimone, Nathaniel L
> Sent: Friday, November 17, 2017 5:38 AM
> To: Gao, Liming <liming.gao@intel.com>; Aaron Durbin <adurbin@google.com>
> Cc: edk2-devel@lists.01.org
> Subject: RE: [edk2] EDK2 build issues
> 
> Hi Liming,
> 
> I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j <n>" (where n > 1). I have filed
> the bug in Bugzilla:
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=786
> 
> For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the
> ARCH variable request in Bugzilla as well?
> 
> Thanks,
> 
> Nate
> 
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming
> Sent: Monday, November 13, 2017 5:46 PM
> To: Aaron Durbin <adurbin@google.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] EDK2 build issues
> 
> I add my comments.
> 
> > -----Original Message-----
> > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> > Aaron Durbin
> > Sent: Tuesday, November 14, 2017 1:38 AM
> > To: Gao, Liming <liming.gao@intel.com>
> > Cc: edk2-devel@lists.01.org
> > Subject: Re: [edk2] EDK2 build issues
> >
> > On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote:
> > > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them.
> > >
> > > 1. BaseTools Soure tools are compiled in serial instead of parallel.
> > > And, VfrCompiler depends on Anltr tool. So, Anltr is required to be
> > built first. I agree that this way takes some time to compile
> > BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch.
> >
> > If the depedencies are correctly met in the makefiles then why are
> > they built serially? It sounds like you understand the dependencies so
> > why aren't those explicitly noted so one can build in parallel?
> >
> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar
> (https://bugzilla.tianocore.org/)?
> Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it.
> 
> > > 2. BaseTools C source are compiled to the executable files. They run
> > > in host machine. Here ARCH is for the executable files those can
> > run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command.
> >
> > ARCH != host is my point. Why are those being conflated? Or are you
> > saying edk2 tools define ARCH to be what the host is?
> >
> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the
> different purpose.
> 
> > > 3. There are more GCC compiler distribution. We try to use the
> > > generic options in GCC tool chain. If you find the option doesn't
> > > work
> > on your GCC compiler, please report it. And, we also notice
> > tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community.
> > >
> > >>-----Original Message-----
> > >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf
> > >>Of Aaron Durbin
> > >>Sent: Saturday, November 11, 2017 6:27 AM
> > >>To: edk2-devel@lists.01.org
> > >>Subject: [edk2] EDK2 build issues
> > >>
> > >>Hello,
> > >>
> > >>Here are some observations I've encountered trying to utilize edk2
> > >>for certain builds. Part of the problem seems to be with implicit
> > >>assumptions in how edk2 is used. I'm trying to build things using
> > >>edk2 from a clean enviroment on an automated builder. i.e. there
> > >>isn't a workspace that exists on one persons computer for the
> > >>lifetime of development.
> > >>
> > >>1. BaseTools can't build in parallel. The tests are racey which
> > >>result in test failures. Because of this one has to build these in
> > >>serial instead of in parallel.
> > >>
> > >>===========================================================
> > >>===========
> > >>FAIL: testHelp (TianoCompress.Tests)
> > >>--------------------------------------------------------------------
> > >>--
> > >>Traceback (most recent call last):
> > >>  File
> > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
> > >>9999/Edk2/BaseTools/Tests/TianoCompress.py",
> > >>line 34, in testHelp
> > >>    self.assertTrue(result == 0)
> > >>AssertionError: False is not true
> > >>
> > >>===========================================================
> > >>===========
> > >>FAIL: testRandomDataCycles (TianoCompress.Tests)
> > >>--------------------------------------------------------------------
> > >>--
> > >>Traceback (most recent call last):
> > >>  File
> > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
> > >>9999/Edk2/BaseTools/Tests/TianoCompress.py",
> > >>line 65, in testRandomDataCycles
> > >>    self.compressionTestCycle(data)
> > >>  File
> > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-
> > >>9999/Edk2/BaseTools/Tests/TianoCompress.py",
> > >>line 44, in compressionTestCycle
> > >>    self.assertTrue(result == 0)
> > >>AssertionError: False is not true
> > >>
> > >>In addition, it seems compilation even breaks trying to build a parser:
> > >>
> > >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token  {^M
> > >>^
> > >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)':
> > >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any
> > >>field named 'VfrLexer'
> > >>   CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M
> > >>                                    ^
> > >>
> > >>This just slows down builds needing to things serially.
> > >>
> > >>2. It appears the BaseTools uses the ARCH environment variable. I'm
> > >>not sure of the origins, but ARCH seems like a complete misnomer for
> > >>the *host* you are trying to build tools on -- not the target.
> > >>Trying to incorporate edk2 builds into a portage environment
> > >>effectively breaks because of this as ARCH refers to target
> > >>architecture -- not host builder's ARCH.
> > >>
> > >>3. This more of an observation, but the tools definition seems to
> > >>make quite the leap on how consistent compilers are of a certain version.
> > >>e.g. GCC 4.9 can be built with all kinds of default options that
> > >>edk2 implicitly assumes are set based on some distribution's default flags?
> > >>And in order to extend toolchain support one needs to create a
> > >>series of entries associated with a certain family. Barrier to entry
> > >>is pretty high in teasing out where to put things in the ~8000 line
> > >>tools_def.template.
> > >>
> > >>Thanks for the consideration.
> > >>
> > >>-Aaron
> > >>_______________________________________________
> > >>edk2-devel mailing list
> > >>edk2-devel@lists.01.org
> > >>https://lists.01.org/mailman/listinfo/edk2-devel
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel