* EDK2 build issues
@ 2017-11-10 22:26 Aaron Durbin
2017-11-13 2:15 ` Gao, Liming
0 siblings, 1 reply; 10+ messages in thread
From: Aaron Durbin @ 2017-11-10 22:26 UTC (permalink / raw)
To: edk2-devel
Hello,
Here are some observations I've encountered trying to utilize edk2 for
certain builds. Part of the problem seems to be with implicit
assumptions in how edk2 is used. I'm trying to build things using edk2
from a clean enviroment on an automated builder. i.e. there isn't a
workspace that exists on one persons computer for the lifetime of
development.
1. BaseTools can't build in parallel. The tests are racey which result
in test failures. Because of this one has to build these in serial
instead of in parallel.
======================================================================
FAIL: testHelp (TianoCompress.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-9999/Edk2/BaseTools/Tests/TianoCompress.py",
line 34, in testHelp
self.assertTrue(result == 0)
AssertionError: False is not true
======================================================================
FAIL: testRandomDataCycles (TianoCompress.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-9999/Edk2/BaseTools/Tests/TianoCompress.py",
line 65, in testRandomDataCycles
self.compressionTestCycle(data)
File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl-9999/Edk2/BaseTools/Tests/TianoCompress.py",
line 44, in compressionTestCycle
self.assertTrue(result == 0)
AssertionError: False is not true
In addition, it seems compilation even breaks trying to build a parser:
VfrSyntax.cpp:53:1: error: expected class-name before '{' token
{^M
^
VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)':
VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any
field named 'VfrLexer'
CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M
^
This just slows down builds needing to things serially.
2. It appears the BaseTools uses the ARCH environment variable. I'm
not sure of the origins, but ARCH seems like a complete misnomer for
the *host* you are trying to build tools on -- not the target. Trying
to incorporate edk2 builds into a portage environment effectively
breaks because of this as ARCH refers to target architecture -- not
host builder's ARCH.
3. This more of an observation, but the tools definition seems to make
quite the leap on how consistent compilers are of a certain version.
e.g. GCC 4.9 can be built with all kinds of default options that edk2
implicitly assumes are set based on some distribution's default flags?
And in order to extend toolchain support one needs to create a series
of entries associated with a certain family. Barrier to entry is
pretty high in teasing out where to put things in the ~8000 line
tools_def.template.
Thanks for the consideration.
-Aaron
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: EDK2 build issues 2017-11-10 22:26 EDK2 build issues Aaron Durbin @ 2017-11-13 2:15 ` Gao, Liming 2017-11-13 17:37 ` Aaron Durbin 0 siblings, 1 reply; 10+ messages in thread From: Gao, Liming @ 2017-11-13 2:15 UTC (permalink / raw) To: Aaron Durbin, edk2-devel@lists.01.org Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. 1. BaseTools Soure tools are compiled in serial instead of parallel. And, VfrCompiler depends on Anltr tool. So, Anltr is required to be built first. I agree that this way takes some time to compile BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. 2. BaseTools C source are compiled to the executable files. They run in host machine. Here ARCH is for the executable files those can run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. 3. There are more GCC compiler distribution. We try to use the generic options in GCC tool chain. If you find the option doesn't work on your GCC compiler, please report it. And, we also notice tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. >-----Original Message----- >From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >Aaron Durbin >Sent: Saturday, November 11, 2017 6:27 AM >To: edk2-devel@lists.01.org >Subject: [edk2] EDK2 build issues > >Hello, > >Here are some observations I've encountered trying to utilize edk2 for >certain builds. Part of the problem seems to be with implicit >assumptions in how edk2 is used. I'm trying to build things using edk2 >from a clean enviroment on an automated builder. i.e. there isn't a >workspace that exists on one persons computer for the lifetime of >development. > >1. BaseTools can't build in parallel. The tests are racey which result >in test failures. Because of this one has to build these in serial >instead of in parallel. > >=========================================================== >=========== >FAIL: testHelp (TianoCompress.Tests) >---------------------------------------------------------------------- >Traceback (most recent call last): > File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >9999/Edk2/BaseTools/Tests/TianoCompress.py", >line 34, in testHelp > self.assertTrue(result == 0) >AssertionError: False is not true > >=========================================================== >=========== >FAIL: testRandomDataCycles (TianoCompress.Tests) >---------------------------------------------------------------------- >Traceback (most recent call last): > File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >9999/Edk2/BaseTools/Tests/TianoCompress.py", >line 65, in testRandomDataCycles > self.compressionTestCycle(data) > File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >9999/Edk2/BaseTools/Tests/TianoCompress.py", >line 44, in compressionTestCycle > self.assertTrue(result == 0) >AssertionError: False is not true > >In addition, it seems compilation even breaks trying to build a parser: > >VfrSyntax.cpp:53:1: error: expected class-name before '{' token > {^M > ^ >VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >field named 'VfrLexer' > CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M > ^ > >This just slows down builds needing to things serially. > >2. It appears the BaseTools uses the ARCH environment variable. I'm >not sure of the origins, but ARCH seems like a complete misnomer for >the *host* you are trying to build tools on -- not the target. Trying >to incorporate edk2 builds into a portage environment effectively >breaks because of this as ARCH refers to target architecture -- not >host builder's ARCH. > >3. This more of an observation, but the tools definition seems to make >quite the leap on how consistent compilers are of a certain version. >e.g. GCC 4.9 can be built with all kinds of default options that edk2 >implicitly assumes are set based on some distribution's default flags? >And in order to extend toolchain support one needs to create a series >of entries associated with a certain family. Barrier to entry is >pretty high in teasing out where to put things in the ~8000 line >tools_def.template. > >Thanks for the consideration. > >-Aaron >_______________________________________________ >edk2-devel mailing list >edk2-devel@lists.01.org >https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-13 2:15 ` Gao, Liming @ 2017-11-13 17:37 ` Aaron Durbin 2017-11-14 1:46 ` Gao, Liming 0 siblings, 1 reply; 10+ messages in thread From: Aaron Durbin @ 2017-11-13 17:37 UTC (permalink / raw) To: Gao, Liming; +Cc: edk2-devel@lists.01.org On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. > > 1. BaseTools Soure tools are compiled in serial instead of parallel. And, VfrCompiler depends on Anltr tool. So, Anltr is required to be built first. I agree that this way takes some time to compile BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. If the depedencies are correctly met in the makefiles then why are they built serially? It sounds like you understand the dependencies so why aren't those explicitly noted so one can build in parallel? > 2. BaseTools C source are compiled to the executable files. They run in host machine. Here ARCH is for the executable files those can run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. ARCH != host is my point. Why are those being conflated? Or are you saying edk2 tools define ARCH to be what the host is? > 3. There are more GCC compiler distribution. We try to use the generic options in GCC tool chain. If you find the option doesn't work on your GCC compiler, please report it. And, we also notice tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. > >>-----Original Message----- >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >>Aaron Durbin >>Sent: Saturday, November 11, 2017 6:27 AM >>To: edk2-devel@lists.01.org >>Subject: [edk2] EDK2 build issues >> >>Hello, >> >>Here are some observations I've encountered trying to utilize edk2 for >>certain builds. Part of the problem seems to be with implicit >>assumptions in how edk2 is used. I'm trying to build things using edk2 >>from a clean enviroment on an automated builder. i.e. there isn't a >>workspace that exists on one persons computer for the lifetime of >>development. >> >>1. BaseTools can't build in parallel. The tests are racey which result >>in test failures. Because of this one has to build these in serial >>instead of in parallel. >> >>=========================================================== >>=========== >>FAIL: testHelp (TianoCompress.Tests) >>---------------------------------------------------------------------- >>Traceback (most recent call last): >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>9999/Edk2/BaseTools/Tests/TianoCompress.py", >>line 34, in testHelp >> self.assertTrue(result == 0) >>AssertionError: False is not true >> >>=========================================================== >>=========== >>FAIL: testRandomDataCycles (TianoCompress.Tests) >>---------------------------------------------------------------------- >>Traceback (most recent call last): >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>9999/Edk2/BaseTools/Tests/TianoCompress.py", >>line 65, in testRandomDataCycles >> self.compressionTestCycle(data) >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>9999/Edk2/BaseTools/Tests/TianoCompress.py", >>line 44, in compressionTestCycle >> self.assertTrue(result == 0) >>AssertionError: False is not true >> >>In addition, it seems compilation even breaks trying to build a parser: >> >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token >> {^M >> ^ >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>field named 'VfrLexer' >> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >> ^ >> >>This just slows down builds needing to things serially. >> >>2. It appears the BaseTools uses the ARCH environment variable. I'm >>not sure of the origins, but ARCH seems like a complete misnomer for >>the *host* you are trying to build tools on -- not the target. Trying >>to incorporate edk2 builds into a portage environment effectively >>breaks because of this as ARCH refers to target architecture -- not >>host builder's ARCH. >> >>3. This more of an observation, but the tools definition seems to make >>quite the leap on how consistent compilers are of a certain version. >>e.g. GCC 4.9 can be built with all kinds of default options that edk2 >>implicitly assumes are set based on some distribution's default flags? >>And in order to extend toolchain support one needs to create a series >>of entries associated with a certain family. Barrier to entry is >>pretty high in teasing out where to put things in the ~8000 line >>tools_def.template. >> >>Thanks for the consideration. >> >>-Aaron >>_______________________________________________ >>edk2-devel mailing list >>edk2-devel@lists.01.org >>https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-13 17:37 ` Aaron Durbin @ 2017-11-14 1:46 ` Gao, Liming 2017-11-16 21:37 ` Desimone, Nathaniel L 0 siblings, 1 reply; 10+ messages in thread From: Gao, Liming @ 2017-11-14 1:46 UTC (permalink / raw) To: Aaron Durbin; +Cc: edk2-devel@lists.01.org I add my comments. > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Aaron Durbin > Sent: Tuesday, November 14, 2017 1:38 AM > To: Gao, Liming <liming.gao@intel.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: > > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. > > > > 1. BaseTools Soure tools are compiled in serial instead of parallel. And, VfrCompiler depends on Anltr tool. So, Anltr is required to be > built first. I agree that this way takes some time to compile BaseTools source. I have one proposal to compile C source tools with > multiple thread. I can share my draft patch. > > If the depedencies are correctly met in the makefiles then why are > they built serially? It sounds like you understand the dependencies so > why aren't those explicitly noted so one can build in parallel? > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > > 2. BaseTools C source are compiled to the executable files. They run in host machine. Here ARCH is for the executable files those can > run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. > > ARCH != host is my point. Why are those being conflated? Or are you > saying edk2 tools define ARCH to be what the host is? > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the different purpose. > > 3. There are more GCC compiler distribution. We try to use the generic options in GCC tool chain. If you find the option doesn't work > on your GCC compiler, please report it. And, we also notice tools_def.txt is too big to be hard to be maintain. We have one proposal to > simplify it. I will send this RFC to edk2 community. > > > >>-----Original Message----- > >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > >>Aaron Durbin > >>Sent: Saturday, November 11, 2017 6:27 AM > >>To: edk2-devel@lists.01.org > >>Subject: [edk2] EDK2 build issues > >> > >>Hello, > >> > >>Here are some observations I've encountered trying to utilize edk2 for > >>certain builds. Part of the problem seems to be with implicit > >>assumptions in how edk2 is used. I'm trying to build things using edk2 > >>from a clean enviroment on an automated builder. i.e. there isn't a > >>workspace that exists on one persons computer for the lifetime of > >>development. > >> > >>1. BaseTools can't build in parallel. The tests are racey which result > >>in test failures. Because of this one has to build these in serial > >>instead of in parallel. > >> > >>=========================================================== > >>=========== > >>FAIL: testHelp (TianoCompress.Tests) > >>---------------------------------------------------------------------- > >>Traceback (most recent call last): > >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 34, in testHelp > >> self.assertTrue(result == 0) > >>AssertionError: False is not true > >> > >>=========================================================== > >>=========== > >>FAIL: testRandomDataCycles (TianoCompress.Tests) > >>---------------------------------------------------------------------- > >>Traceback (most recent call last): > >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 65, in testRandomDataCycles > >> self.compressionTestCycle(data) > >> File "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 44, in compressionTestCycle > >> self.assertTrue(result == 0) > >>AssertionError: False is not true > >> > >>In addition, it seems compilation even breaks trying to build a parser: > >> > >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token > >> {^M > >> ^ > >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': > >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any > >>field named 'VfrLexer' > >> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M > >> ^ > >> > >>This just slows down builds needing to things serially. > >> > >>2. It appears the BaseTools uses the ARCH environment variable. I'm > >>not sure of the origins, but ARCH seems like a complete misnomer for > >>the *host* you are trying to build tools on -- not the target. Trying > >>to incorporate edk2 builds into a portage environment effectively > >>breaks because of this as ARCH refers to target architecture -- not > >>host builder's ARCH. > >> > >>3. This more of an observation, but the tools definition seems to make > >>quite the leap on how consistent compilers are of a certain version. > >>e.g. GCC 4.9 can be built with all kinds of default options that edk2 > >>implicitly assumes are set based on some distribution's default flags? > >>And in order to extend toolchain support one needs to create a series > >>of entries associated with a certain family. Barrier to entry is > >>pretty high in teasing out where to put things in the ~8000 line > >>tools_def.template. > >> > >>Thanks for the consideration. > >> > >>-Aaron > >>_______________________________________________ > >>edk2-devel mailing list > >>edk2-devel@lists.01.org > >>https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-14 1:46 ` Gao, Liming @ 2017-11-16 21:37 ` Desimone, Nathaniel L 2017-11-17 8:45 ` Gao, Liming 2017-11-23 0:01 ` Andrew Fish 0 siblings, 2 replies; 10+ messages in thread From: Desimone, Nathaniel L @ 2017-11-16 21:37 UTC (permalink / raw) To: Gao, Liming, Aaron Durbin; +Cc: edk2-devel@lists.01.org Hi Liming, I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j <n>" (where n > 1). I have filed the bug in Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=786 For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in Bugzilla as well? Thanks, Nate -----Original Message----- From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming Sent: Monday, November 13, 2017 5:46 PM To: Aaron Durbin <adurbin@google.com> Cc: edk2-devel@lists.01.org Subject: Re: [edk2] EDK2 build issues I add my comments. > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > Aaron Durbin > Sent: Tuesday, November 14, 2017 1:38 AM > To: Gao, Liming <liming.gao@intel.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: > > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. > > > > 1. BaseTools Soure tools are compiled in serial instead of parallel. > > And, VfrCompiler depends on Anltr tool. So, Anltr is required to be > built first. I agree that this way takes some time to compile > BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. > > If the depedencies are correctly met in the makefiles then why are > they built serially? It sounds like you understand the dependencies so > why aren't those explicitly noted so one can build in parallel? > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > > 2. BaseTools C source are compiled to the executable files. They run > > in host machine. Here ARCH is for the executable files those can > run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. > > ARCH != host is my point. Why are those being conflated? Or are you > saying edk2 tools define ARCH to be what the host is? > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the different purpose. > > 3. There are more GCC compiler distribution. We try to use the > > generic options in GCC tool chain. If you find the option doesn't > > work > on your GCC compiler, please report it. And, we also notice > tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. > > > >>-----Original Message----- > >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf > >>Of Aaron Durbin > >>Sent: Saturday, November 11, 2017 6:27 AM > >>To: edk2-devel@lists.01.org > >>Subject: [edk2] EDK2 build issues > >> > >>Hello, > >> > >>Here are some observations I've encountered trying to utilize edk2 > >>for certain builds. Part of the problem seems to be with implicit > >>assumptions in how edk2 is used. I'm trying to build things using > >>edk2 from a clean enviroment on an automated builder. i.e. there > >>isn't a workspace that exists on one persons computer for the > >>lifetime of development. > >> > >>1. BaseTools can't build in parallel. The tests are racey which > >>result in test failures. Because of this one has to build these in > >>serial instead of in parallel. > >> > >>=========================================================== > >>=========== > >>FAIL: testHelp (TianoCompress.Tests) > >>-------------------------------------------------------------------- > >>-- > >>Traceback (most recent call last): > >> File > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 34, in testHelp > >> self.assertTrue(result == 0) > >>AssertionError: False is not true > >> > >>=========================================================== > >>=========== > >>FAIL: testRandomDataCycles (TianoCompress.Tests) > >>-------------------------------------------------------------------- > >>-- > >>Traceback (most recent call last): > >> File > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 65, in testRandomDataCycles > >> self.compressionTestCycle(data) > >> File > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > >>line 44, in compressionTestCycle > >> self.assertTrue(result == 0) > >>AssertionError: False is not true > >> > >>In addition, it seems compilation even breaks trying to build a parser: > >> > >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M > >>^ > >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': > >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any > >>field named 'VfrLexer' > >> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M > >> ^ > >> > >>This just slows down builds needing to things serially. > >> > >>2. It appears the BaseTools uses the ARCH environment variable. I'm > >>not sure of the origins, but ARCH seems like a complete misnomer for > >>the *host* you are trying to build tools on -- not the target. > >>Trying to incorporate edk2 builds into a portage environment > >>effectively breaks because of this as ARCH refers to target > >>architecture -- not host builder's ARCH. > >> > >>3. This more of an observation, but the tools definition seems to > >>make quite the leap on how consistent compilers are of a certain version. > >>e.g. GCC 4.9 can be built with all kinds of default options that > >>edk2 implicitly assumes are set based on some distribution's default flags? > >>And in order to extend toolchain support one needs to create a > >>series of entries associated with a certain family. Barrier to entry > >>is pretty high in teasing out where to put things in the ~8000 line > >>tools_def.template. > >> > >>Thanks for the consideration. > >> > >>-Aaron > >>_______________________________________________ > >>edk2-devel mailing list > >>edk2-devel@lists.01.org > >>https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel _______________________________________________ edk2-devel mailing list edk2-devel@lists.01.org https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-16 21:37 ` Desimone, Nathaniel L @ 2017-11-17 8:45 ` Gao, Liming 2017-11-22 23:48 ` Desimone, Nathaniel L 2017-11-23 0:01 ` Andrew Fish 1 sibling, 1 reply; 10+ messages in thread From: Gao, Liming @ 2017-11-17 8:45 UTC (permalink / raw) To: Desimone, Nathaniel L, Aaron Durbin; +Cc: edk2-devel@lists.01.org Nate: OK. Please submit the tracker on ARCH request. Thanks Liming > -----Original Message----- > From: Desimone, Nathaniel L > Sent: Friday, November 17, 2017 5:38 AM > To: Gao, Liming <liming.gao@intel.com>; Aaron Durbin <adurbin@google.com> > Cc: edk2-devel@lists.01.org > Subject: RE: [edk2] EDK2 build issues > > Hi Liming, > > I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j <n>" (where n > 1). I have filed > the bug in Bugzilla: > > https://bugzilla.tianocore.org/show_bug.cgi?id=786 > > For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the > ARCH variable request in Bugzilla as well? > > Thanks, > > Nate > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming > Sent: Monday, November 13, 2017 5:46 PM > To: Aaron Durbin <adurbin@google.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > I add my comments. > > > -----Original Message----- > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > > Aaron Durbin > > Sent: Tuesday, November 14, 2017 1:38 AM > > To: Gao, Liming <liming.gao@intel.com> > > Cc: edk2-devel@lists.01.org > > Subject: Re: [edk2] EDK2 build issues > > > > On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: > > > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. > > > > > > 1. BaseTools Soure tools are compiled in serial instead of parallel. > > > And, VfrCompiler depends on Anltr tool. So, Anltr is required to be > > built first. I agree that this way takes some time to compile > > BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. > > > > If the depedencies are correctly met in the makefiles then why are > > they built serially? It sounds like you understand the dependencies so > > why aren't those explicitly noted so one can build in parallel? > > > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar > (https://bugzilla.tianocore.org/)? > Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > > > > 2. BaseTools C source are compiled to the executable files. They run > > > in host machine. Here ARCH is for the executable files those can > > run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. > > > > ARCH != host is my point. Why are those being conflated? Or are you > > saying edk2 tools define ARCH to be what the host is? > > > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the > different purpose. > > > > 3. There are more GCC compiler distribution. We try to use the > > > generic options in GCC tool chain. If you find the option doesn't > > > work > > on your GCC compiler, please report it. And, we also notice > > tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. > > > > > >>-----Original Message----- > > >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf > > >>Of Aaron Durbin > > >>Sent: Saturday, November 11, 2017 6:27 AM > > >>To: edk2-devel@lists.01.org > > >>Subject: [edk2] EDK2 build issues > > >> > > >>Hello, > > >> > > >>Here are some observations I've encountered trying to utilize edk2 > > >>for certain builds. Part of the problem seems to be with implicit > > >>assumptions in how edk2 is used. I'm trying to build things using > > >>edk2 from a clean enviroment on an automated builder. i.e. there > > >>isn't a workspace that exists on one persons computer for the > > >>lifetime of development. > > >> > > >>1. BaseTools can't build in parallel. The tests are racey which > > >>result in test failures. Because of this one has to build these in > > >>serial instead of in parallel. > > >> > > >>=========================================================== > > >>=========== > > >>FAIL: testHelp (TianoCompress.Tests) > > >>-------------------------------------------------------------------- > > >>-- > > >>Traceback (most recent call last): > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 34, in testHelp > > >> self.assertTrue(result == 0) > > >>AssertionError: False is not true > > >> > > >>=========================================================== > > >>=========== > > >>FAIL: testRandomDataCycles (TianoCompress.Tests) > > >>-------------------------------------------------------------------- > > >>-- > > >>Traceback (most recent call last): > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 65, in testRandomDataCycles > > >> self.compressionTestCycle(data) > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 44, in compressionTestCycle > > >> self.assertTrue(result == 0) > > >>AssertionError: False is not true > > >> > > >>In addition, it seems compilation even breaks trying to build a parser: > > >> > > >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M > > >>^ > > >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': > > >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any > > >>field named 'VfrLexer' > > >> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M > > >> ^ > > >> > > >>This just slows down builds needing to things serially. > > >> > > >>2. It appears the BaseTools uses the ARCH environment variable. I'm > > >>not sure of the origins, but ARCH seems like a complete misnomer for > > >>the *host* you are trying to build tools on -- not the target. > > >>Trying to incorporate edk2 builds into a portage environment > > >>effectively breaks because of this as ARCH refers to target > > >>architecture -- not host builder's ARCH. > > >> > > >>3. This more of an observation, but the tools definition seems to > > >>make quite the leap on how consistent compilers are of a certain version. > > >>e.g. GCC 4.9 can be built with all kinds of default options that > > >>edk2 implicitly assumes are set based on some distribution's default flags? > > >>And in order to extend toolchain support one needs to create a > > >>series of entries associated with a certain family. Barrier to entry > > >>is pretty high in teasing out where to put things in the ~8000 line > > >>tools_def.template. > > >> > > >>Thanks for the consideration. > > >> > > >>-Aaron > > >>_______________________________________________ > > >>edk2-devel mailing list > > >>edk2-devel@lists.01.org > > >>https://lists.01.org/mailman/listinfo/edk2-devel > > _______________________________________________ > > edk2-devel mailing list > > edk2-devel@lists.01.org > > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-17 8:45 ` Gao, Liming @ 2017-11-22 23:48 ` Desimone, Nathaniel L 0 siblings, 0 replies; 10+ messages in thread From: Desimone, Nathaniel L @ 2017-11-22 23:48 UTC (permalink / raw) To: Gao, Liming, Aaron Durbin; +Cc: edk2-devel@lists.01.org https://bugzilla.tianocore.org/show_bug.cgi?id=793 Thanks, Nate -----Original Message----- From: Gao, Liming Sent: Friday, November 17, 2017 12:46 AM To: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Aaron Durbin <adurbin@google.com> Cc: edk2-devel@lists.01.org Subject: RE: [edk2] EDK2 build issues Nate: OK. Please submit the tracker on ARCH request. Thanks Liming > -----Original Message----- > From: Desimone, Nathaniel L > Sent: Friday, November 17, 2017 5:38 AM > To: Gao, Liming <liming.gao@intel.com>; Aaron Durbin > <adurbin@google.com> > Cc: edk2-devel@lists.01.org > Subject: RE: [edk2] EDK2 build issues > > Hi Liming, > > I chatted with Aaron on the phone today. The VfrCompiler race > condition was discovered using "make -j <n>" (where n > 1). I have filed the bug in Bugzilla: > > https://bugzilla.tianocore.org/show_bug.cgi?id=786 > > For the ARCH environment variable, we brainstormed two possible new > names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in Bugzilla as well? > > Thanks, > > Nate > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of > Gao, Liming > Sent: Monday, November 13, 2017 5:46 PM > To: Aaron Durbin <adurbin@google.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > I add my comments. > > > -----Original Message----- > > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf > > Of Aaron Durbin > > Sent: Tuesday, November 14, 2017 1:38 AM > > To: Gao, Liming <liming.gao@intel.com> > > Cc: edk2-devel@lists.01.org > > Subject: Re: [edk2] EDK2 build issues > > > > On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: > > > Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. > > > > > > 1. BaseTools Soure tools are compiled in serial instead of parallel. > > > And, VfrCompiler depends on Anltr tool. So, Anltr is required to > > > be > > built first. I agree that this way takes some time to compile > > BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. > > > > If the depedencies are correctly met in the makefiles then why are > > they built serially? It sounds like you understand the dependencies > > so why aren't those explicitly noted so one can build in parallel? > > > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. > Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? > Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > > > > 2. BaseTools C source are compiled to the executable files. They > > > run in host machine. Here ARCH is for the executable files those > > > can > > run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. > > > > ARCH != host is my point. Why are those being conflated? Or are you > > saying edk2 tools define ARCH to be what the host is? > > > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name > is too generic. In your environment, ARCH may be defined for the different purpose. > > > > 3. There are more GCC compiler distribution. We try to use the > > > generic options in GCC tool chain. If you find the option doesn't > > > work > > on your GCC compiler, please report it. And, we also notice > > tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. > > > > > >>-----Original Message----- > > >>From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On > > >>Behalf Of Aaron Durbin > > >>Sent: Saturday, November 11, 2017 6:27 AM > > >>To: edk2-devel@lists.01.org > > >>Subject: [edk2] EDK2 build issues > > >> > > >>Hello, > > >> > > >>Here are some observations I've encountered trying to utilize edk2 > > >>for certain builds. Part of the problem seems to be with implicit > > >>assumptions in how edk2 is used. I'm trying to build things using > > >>edk2 from a clean enviroment on an automated builder. i.e. there > > >>isn't a workspace that exists on one persons computer for the > > >>lifetime of development. > > >> > > >>1. BaseTools can't build in parallel. The tests are racey which > > >>result in test failures. Because of this one has to build these in > > >>serial instead of in parallel. > > >> > > >>=========================================================== > > >>=========== > > >>FAIL: testHelp (TianoCompress.Tests) > > >>------------------------------------------------------------------ > > >>-- > > >>-- > > >>Traceback (most recent call last): > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 34, in testHelp > > >> self.assertTrue(result == 0) > > >>AssertionError: False is not true > > >> > > >>=========================================================== > > >>=========== > > >>FAIL: testRandomDataCycles (TianoCompress.Tests) > > >>------------------------------------------------------------------ > > >>-- > > >>-- > > >>Traceback (most recent call last): > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 65, in testRandomDataCycles > > >> self.compressionTestCycle(data) > > >> File > > >>"/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- > > >>9999/Edk2/BaseTools/Tests/TianoCompress.py", > > >>line 44, in compressionTestCycle > > >> self.assertTrue(result == 0) > > >>AssertionError: False is not true > > >> > > >>In addition, it seems compilation even breaks trying to build a parser: > > >> > > >>VfrSyntax.cpp:53:1: error: expected class-name before '{' token > > >>{^M ^ > > >>VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': > > >>VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any > > >>field named 'VfrLexer' > > >> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M > > >> ^ > > >> > > >>This just slows down builds needing to things serially. > > >> > > >>2. It appears the BaseTools uses the ARCH environment variable. > > >>I'm not sure of the origins, but ARCH seems like a complete > > >>misnomer for the *host* you are trying to build tools on -- not the target. > > >>Trying to incorporate edk2 builds into a portage environment > > >>effectively breaks because of this as ARCH refers to target > > >>architecture -- not host builder's ARCH. > > >> > > >>3. This more of an observation, but the tools definition seems to > > >>make quite the leap on how consistent compilers are of a certain version. > > >>e.g. GCC 4.9 can be built with all kinds of default options that > > >>edk2 implicitly assumes are set based on some distribution's default flags? > > >>And in order to extend toolchain support one needs to create a > > >>series of entries associated with a certain family. Barrier to > > >>entry is pretty high in teasing out where to put things in the > > >>~8000 line tools_def.template. > > >> > > >>Thanks for the consideration. > > >> > > >>-Aaron > > >>_______________________________________________ > > >>edk2-devel mailing list > > >>edk2-devel@lists.01.org > > >>https://lists.01.org/mailman/listinfo/edk2-devel > > _______________________________________________ > > edk2-devel mailing list > > edk2-devel@lists.01.org > > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-16 21:37 ` Desimone, Nathaniel L 2017-11-17 8:45 ` Gao, Liming @ 2017-11-23 0:01 ` Andrew Fish 2017-11-23 5:28 ` Gao, Liming 1 sibling, 1 reply; 10+ messages in thread From: Andrew Fish @ 2017-11-23 0:01 UTC (permalink / raw) To: Desimone, Nathaniel L; +Cc: Gao, Liming, Aaron Durbin, edk2-devel@lists.01.org > On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L <nathaniel.l.desimone@intel.com> wrote: > > Hi Liming, > > I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j <n>" (where n > 1). I have filed the bug in Bugzilla: > I think there are a few issues like that in BaseTools. We have a parallel makefile and we had to turn off parallelism when calling the BaseTools. In general the edk2 build system fights against "make -j <n>" since the Python build command is trying to do the threading and invoking make in parallel. Given we ported EDK to parallel build, the Python parallel edk2 build is a lot slower than the EDK make parallel build, even when adjusting for the extra work the edk2 does. I even noticed a 1,000,000 calls to regex in Python during the ... phase of the build. I think the correct short term fix may be to put .NOTPARALLEL: in the BaseTools GNUmakefile given it is constructed in a way that it does not support parallelism. Thanks, Andrew Fish > https://bugzilla.tianocore.org/show_bug.cgi?id=786 > > For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in Bugzilla as well? > > Thanks, > > Nate > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming > Sent: Monday, November 13, 2017 5:46 PM > To: Aaron Durbin <adurbin@google.com> > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > I add my comments. > >> -----Original Message----- >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >> Aaron Durbin >> Sent: Tuesday, November 14, 2017 1:38 AM >> To: Gao, Liming <liming.gao@intel.com> >> Cc: edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> wrote: >>> Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. >>> >>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >> built first. I agree that this way takes some time to compile >> BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. >> >> If the depedencies are correctly met in the makefiles then why are >> they built serially? It sounds like you understand the dependencies so >> why aren't those explicitly noted so one can build in parallel? >> > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? > Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > >>> 2. BaseTools C source are compiled to the executable files. They run >>> in host machine. Here ARCH is for the executable files those can >> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. >> >> ARCH != host is my point. Why are those being conflated? Or are you >> saying edk2 tools define ARCH to be what the host is? >> > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the different purpose. > >>> 3. There are more GCC compiler distribution. We try to use the >>> generic options in GCC tool chain. If you find the option doesn't >>> work >> on your GCC compiler, please report it. And, we also notice >> tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. >>> >>>> -----Original Message----- >>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>> Of Aaron Durbin >>>> Sent: Saturday, November 11, 2017 6:27 AM >>>> To: edk2-devel@lists.01.org >>>> Subject: [edk2] EDK2 build issues >>>> >>>> Hello, >>>> >>>> Here are some observations I've encountered trying to utilize edk2 >>>> for certain builds. Part of the problem seems to be with implicit >>>> assumptions in how edk2 is used. I'm trying to build things using >>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>> isn't a workspace that exists on one persons computer for the >>>> lifetime of development. >>>> >>>> 1. BaseTools can't build in parallel. The tests are racey which >>>> result in test failures. Because of this one has to build these in >>>> serial instead of in parallel. >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testHelp (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 34, in testHelp >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 65, in testRandomDataCycles >>>> self.compressionTestCycle(data) >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 44, in compressionTestCycle >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> In addition, it seems compilation even breaks trying to build a parser: >>>> >>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>> ^ >>>> VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>> field named 'VfrLexer' >>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>> ^ >>>> >>>> This just slows down builds needing to things serially. >>>> >>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>> the *host* you are trying to build tools on -- not the target. >>>> Trying to incorporate edk2 builds into a portage environment >>>> effectively breaks because of this as ARCH refers to target >>>> architecture -- not host builder's ARCH. >>>> >>>> 3. This more of an observation, but the tools definition seems to >>>> make quite the leap on how consistent compilers are of a certain version. >>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>> edk2 implicitly assumes are set based on some distribution's default flags? >>>> And in order to extend toolchain support one needs to create a >>>> series of entries associated with a certain family. Barrier to entry >>>> is pretty high in teasing out where to put things in the ~8000 line >>>> tools_def.template. >>>> >>>> Thanks for the consideration. >>>> >>>> -Aaron >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-23 0:01 ` Andrew Fish @ 2017-11-23 5:28 ` Gao, Liming 2017-11-27 16:18 ` Andrew Fish 0 siblings, 1 reply; 10+ messages in thread From: Gao, Liming @ 2017-11-23 5:28 UTC (permalink / raw) To: afish@apple.com, Desimone, Nathaniel L Cc: Aaron Durbin, edk2-devel@lists.01.org Andrew: To compile BaseTools C tools is a separate step. It happens before build. I review BaseTools C tool top GNUmakefile. It can be enhanced to support make -j N that has the better performance. I will provide my patch for code review. Thanks Liming >-----Original Message----- >From: afish@apple.com [mailto:afish@apple.com] >Sent: Thursday, November 23, 2017 8:02 AM >To: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com> >Cc: Gao, Liming <liming.gao@intel.com>; Aaron Durbin ><adurbin@google.com>; edk2-devel@lists.01.org >Subject: Re: [edk2] EDK2 build issues > > >> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L ><nathaniel.l.desimone@intel.com> wrote: >> >> Hi Liming, >> >> I chatted with Aaron on the phone today. The VfrCompiler race condition >was discovered using "make -j <n>" (where n > 1). I have filed the bug in >Bugzilla: >> > >I think there are a few issues like that in BaseTools. We have a parallel >makefile and we had to turn off parallelism when calling the BaseTools. > >In general the edk2 build system fights against "make -j <n>" since the Python >build command is trying to do the threading and invoking make in parallel. >Given we ported EDK to parallel build, the Python parallel edk2 build is a lot >slower than the EDK make parallel build, even when adjusting for the extra >work the edk2 does. > >I even noticed a 1,000,000 calls to regex in Python during the ... phase of the >build. > >I think the correct short term fix may be to put .NOTPARALLEL: in the >BaseTools GNUmakefile given it is constructed in a way that it does not >support parallelism. > >Thanks, > >Andrew Fish > > > >> https://bugzilla.tianocore.org/show_bug.cgi?id=786 >> >> For the ARCH environment variable, we brainstormed two possible new >names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable >request in Bugzilla as well? >> >> Thanks, >> >> Nate >> >> -----Original Message----- >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >Gao, Liming >> Sent: Monday, November 13, 2017 5:46 PM >> To: Aaron Durbin <adurbin@google.com> >> Cc: edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> I add my comments. >> >>> -----Original Message----- >>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >>> Aaron Durbin >>> Sent: Tuesday, November 14, 2017 1:38 AM >>> To: Gao, Liming <liming.gao@intel.com> >>> Cc: edk2-devel@lists.01.org >>> Subject: Re: [edk2] EDK2 build issues >>> >>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> >wrote: >>>> Yes. BaseTools needs some environments. Before build BaseTools, we >need to call edkset.sh to setup them. >>>> >>>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >>> built first. I agree that this way takes some time to compile >>> BaseTools source. I have one proposal to compile C source tools with >multiple thread. I can share my draft patch. >>> >>> If the depedencies are correctly met in the makefiles then why are >>> they built serially? It sounds like you understand the dependencies so >>> why aren't those explicitly noted so one can build in parallel? >>> >> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could >you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? >> Besides, do you use make -j option to enable Parallel Execution? I want to >know how to verify it. >> >>>> 2. BaseTools C source are compiled to the executable files. They run >>>> in host machine. Here ARCH is for the executable files those can >>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects >ARCH based on 'uname -m' command. >>> >>> ARCH != host is my point. Why are those being conflated? Or are you >>> saying edk2 tools define ARCH to be what the host is? >>> >> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too >generic. In your environment, ARCH may be defined for the different purpose. >> >>>> 3. There are more GCC compiler distribution. We try to use the >>>> generic options in GCC tool chain. If you find the option doesn't >>>> work >>> on your GCC compiler, please report it. And, we also notice >>> tools_def.txt is too big to be hard to be maintain. We have one proposal to >simplify it. I will send this RFC to edk2 community. >>>> >>>>> -----Original Message----- >>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>>> Of Aaron Durbin >>>>> Sent: Saturday, November 11, 2017 6:27 AM >>>>> To: edk2-devel@lists.01.org >>>>> Subject: [edk2] EDK2 build issues >>>>> >>>>> Hello, >>>>> >>>>> Here are some observations I've encountered trying to utilize edk2 >>>>> for certain builds. Part of the problem seems to be with implicit >>>>> assumptions in how edk2 is used. I'm trying to build things using >>>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>>> isn't a workspace that exists on one persons computer for the >>>>> lifetime of development. >>>>> >>>>> 1. BaseTools can't build in parallel. The tests are racey which >>>>> result in test failures. Because of this one has to build these in >>>>> serial instead of in parallel. >>>>> >>>>> >=========================================================== >>>>> =========== >>>>> FAIL: testHelp (TianoCompress.Tests) >>>>> -------------------------------------------------------------------- >>>>> -- >>>>> Traceback (most recent call last): >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 34, in testHelp >>>>> self.assertTrue(result == 0) >>>>> AssertionError: False is not true >>>>> >>>>> >=========================================================== >>>>> =========== >>>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>>> -------------------------------------------------------------------- >>>>> -- >>>>> Traceback (most recent call last): >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 65, in testRandomDataCycles >>>>> self.compressionTestCycle(data) >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 44, in compressionTestCycle >>>>> self.assertTrue(result == 0) >>>>> AssertionError: False is not true >>>>> >>>>> In addition, it seems compilation even breaks trying to build a parser: >>>>> >>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>>> ^ >>>>> VfrSyntax.cpp: In constructor >'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>>> field named 'VfrLexer' >>>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>>> ^ >>>>> >>>>> This just slows down builds needing to things serially. >>>>> >>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>>> the *host* you are trying to build tools on -- not the target. >>>>> Trying to incorporate edk2 builds into a portage environment >>>>> effectively breaks because of this as ARCH refers to target >>>>> architecture -- not host builder's ARCH. >>>>> >>>>> 3. This more of an observation, but the tools definition seems to >>>>> make quite the leap on how consistent compilers are of a certain version. >>>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>>> edk2 implicitly assumes are set based on some distribution's default >flags? >>>>> And in order to extend toolchain support one needs to create a >>>>> series of entries associated with a certain family. Barrier to entry >>>>> is pretty high in teasing out where to put things in the ~8000 line >>>>> tools_def.template. >>>>> >>>>> Thanks for the consideration. >>>>> >>>>> -Aaron >>>>> _______________________________________________ >>>>> edk2-devel mailing list >>>>> edk2-devel@lists.01.org >>>>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: EDK2 build issues 2017-11-23 5:28 ` Gao, Liming @ 2017-11-27 16:18 ` Andrew Fish 0 siblings, 0 replies; 10+ messages in thread From: Andrew Fish @ 2017-11-27 16:18 UTC (permalink / raw) To: Gao, Liming; +Cc: Desimone, Nathaniel L, Aaron Durbin, edk2-devel@lists.01.org Liming, We have a top level GNUmakefile vs. having script files to build targets. Thus building BaseTools, or using build are subtasks of this higher level makefile. Looks like you have added some fixes to the BaseTools! Thanks, Andrew Fish > On Nov 22, 2017, at 9:28 PM, Gao, Liming <liming.gao@intel.com> wrote: > > Andrew: > To compile BaseTools C tools is a separate step. It happens before build. I review BaseTools C tool top GNUmakefile. It can be enhanced to support make -j N that has the better performance. I will provide my patch for code review. > > Thanks > Liming >> -----Original Message----- >> From: afish@apple.com [mailto:afish@apple.com] >> Sent: Thursday, November 23, 2017 8:02 AM >> To: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com> >> Cc: Gao, Liming <liming.gao@intel.com>; Aaron Durbin >> <adurbin@google.com>; edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> >>> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L >> <nathaniel.l.desimone@intel.com> wrote: >>> >>> Hi Liming, >>> >>> I chatted with Aaron on the phone today. The VfrCompiler race condition >> was discovered using "make -j <n>" (where n > 1). I have filed the bug in >> Bugzilla: >>> >> >> I think there are a few issues like that in BaseTools. We have a parallel >> makefile and we had to turn off parallelism when calling the BaseTools. >> >> In general the edk2 build system fights against "make -j <n>" since the Python >> build command is trying to do the threading and invoking make in parallel. >> Given we ported EDK to parallel build, the Python parallel edk2 build is a lot >> slower than the EDK make parallel build, even when adjusting for the extra >> work the edk2 does. >> >> I even noticed a 1,000,000 calls to regex in Python during the ... phase of the >> build. >> >> I think the correct short term fix may be to put .NOTPARALLEL: in the >> BaseTools GNUmakefile given it is constructed in a way that it does not >> support parallelism. >> >> Thanks, >> >> Andrew Fish >> >> >> >>> https://bugzilla.tianocore.org/show_bug.cgi?id=786 >>> >>> For the ARCH environment variable, we brainstormed two possible new >> names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable >> request in Bugzilla as well? >>> >>> Thanks, >>> >>> Nate >>> >>> -----Original Message----- >>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >> Gao, Liming >>> Sent: Monday, November 13, 2017 5:46 PM >>> To: Aaron Durbin <adurbin@google.com> >>> Cc: edk2-devel@lists.01.org >>> Subject: Re: [edk2] EDK2 build issues >>> >>> I add my comments. >>> >>>> -----Original Message----- >>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >>>> Aaron Durbin >>>> Sent: Tuesday, November 14, 2017 1:38 AM >>>> To: Gao, Liming <liming.gao@intel.com> >>>> Cc: edk2-devel@lists.01.org >>>> Subject: Re: [edk2] EDK2 build issues >>>> >>>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming <liming.gao@intel.com> >> wrote: >>>>> Yes. BaseTools needs some environments. Before build BaseTools, we >> need to call edkset.sh to setup them. >>>>> >>>>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >>>> built first. I agree that this way takes some time to compile >>>> BaseTools source. I have one proposal to compile C source tools with >> multiple thread. I can share my draft patch. >>>> >>>> If the depedencies are correctly met in the makefiles then why are >>>> they built serially? It sounds like you understand the dependencies so >>>> why aren't those explicitly noted so one can build in parallel? >>>> >>> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could >> you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? >>> Besides, do you use make -j option to enable Parallel Execution? I want to >> know how to verify it. >>> >>>>> 2. BaseTools C source are compiled to the executable files. They run >>>>> in host machine. Here ARCH is for the executable files those can >>>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects >> ARCH based on 'uname -m' command. >>>> >>>> ARCH != host is my point. Why are those being conflated? Or are you >>>> saying edk2 tools define ARCH to be what the host is? >>>> >>> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too >> generic. In your environment, ARCH may be defined for the different purpose. >>> >>>>> 3. There are more GCC compiler distribution. We try to use the >>>>> generic options in GCC tool chain. If you find the option doesn't >>>>> work >>>> on your GCC compiler, please report it. And, we also notice >>>> tools_def.txt is too big to be hard to be maintain. We have one proposal to >> simplify it. I will send this RFC to edk2 community. >>>>> >>>>>> -----Original Message----- >>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>>>> Of Aaron Durbin >>>>>> Sent: Saturday, November 11, 2017 6:27 AM >>>>>> To: edk2-devel@lists.01.org >>>>>> Subject: [edk2] EDK2 build issues >>>>>> >>>>>> Hello, >>>>>> >>>>>> Here are some observations I've encountered trying to utilize edk2 >>>>>> for certain builds. Part of the problem seems to be with implicit >>>>>> assumptions in how edk2 is used. I'm trying to build things using >>>>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>>>> isn't a workspace that exists on one persons computer for the >>>>>> lifetime of development. >>>>>> >>>>>> 1. BaseTools can't build in parallel. The tests are racey which >>>>>> result in test failures. Because of this one has to build these in >>>>>> serial instead of in parallel. >>>>>> >>>>>> >> =========================================================== >>>>>> =========== >>>>>> FAIL: testHelp (TianoCompress.Tests) >>>>>> -------------------------------------------------------------------- >>>>>> -- >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 34, in testHelp >>>>>> self.assertTrue(result == 0) >>>>>> AssertionError: False is not true >>>>>> >>>>>> >> =========================================================== >>>>>> =========== >>>>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>>>> -------------------------------------------------------------------- >>>>>> -- >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 65, in testRandomDataCycles >>>>>> self.compressionTestCycle(data) >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 44, in compressionTestCycle >>>>>> self.assertTrue(result == 0) >>>>>> AssertionError: False is not true >>>>>> >>>>>> In addition, it seems compilation even breaks trying to build a parser: >>>>>> >>>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>>>> ^ >>>>>> VfrSyntax.cpp: In constructor >> 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>>>> field named 'VfrLexer' >>>>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>>>> ^ >>>>>> >>>>>> This just slows down builds needing to things serially. >>>>>> >>>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>>>> the *host* you are trying to build tools on -- not the target. >>>>>> Trying to incorporate edk2 builds into a portage environment >>>>>> effectively breaks because of this as ARCH refers to target >>>>>> architecture -- not host builder's ARCH. >>>>>> >>>>>> 3. This more of an observation, but the tools definition seems to >>>>>> make quite the leap on how consistent compilers are of a certain version. >>>>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>>>> edk2 implicitly assumes are set based on some distribution's default >> flags? >>>>>> And in order to extend toolchain support one needs to create a >>>>>> series of entries associated with a certain family. Barrier to entry >>>>>> is pretty high in teasing out where to put things in the ~8000 line >>>>>> tools_def.template. >>>>>> >>>>>> Thanks for the consideration. >>>>>> >>>>>> -Aaron >>>>>> _______________________________________________ >>>>>> edk2-devel mailing list >>>>>> edk2-devel@lists.01.org >>>>>> https://lists.01.org/mailman/listinfo/edk2-devel >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-11-27 16:14 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-11-10 22:26 EDK2 build issues Aaron Durbin 2017-11-13 2:15 ` Gao, Liming 2017-11-13 17:37 ` Aaron Durbin 2017-11-14 1:46 ` Gao, Liming 2017-11-16 21:37 ` Desimone, Nathaniel L 2017-11-17 8:45 ` Gao, Liming 2017-11-22 23:48 ` Desimone, Nathaniel L 2017-11-23 0:01 ` Andrew Fish 2017-11-23 5:28 ` Gao, Liming 2017-11-27 16:18 ` Andrew Fish
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox