Re: TianoCore Community Meeting Minutes

[Andrew Fish <afish@apple.com>]

> On Oct 11, 2018, at 10:43 AM, stephano <stephano.cetola@linux.intel.com> wrote:
> 
> Thank you all for a great set of meetings!
> 
> This is an overview of the topics discussed and the tasks that were assigned. Please feel free to send me any questions or comments.
> 

...

> Improvements to the Build Process
> --------------------------------
> We would like to gather a list of concrete specific proposals. Nate mentioned that there will always be some specialized tooling because of the nature of BIOS. Shawn mentioned that we need to keep in mind that making development easier for current developers should be a priority over making the processes easier for newcomers. I will send out emails asking for specific feedback on some of these topics:
> 
> -Which toolchains are being used, which are validated, and which are known to create reproducible builds.
> -When toolchains are known to be orphaned, should they be archived or simply removed.
> -Could we add some kconfig-like tool that allows introspection into what type of builds are available.
> -How can we better track the code quality of BaseTools and the current build system on the whole. Should we add a "requires documentation change" flag to BZ so that it will be easier to compile a list of required doc changes.
> 


Stephano,

I'll help start the conversation on the build. 

1) Making the process easier for newcomers. 

a) Getting it to build the 1st time
I think we could aspire to a more Unix like flow:
git clone
./configure
make ovmf
make ovmf.run

The ./config step would automate resolving dependencies. Things like figuring out what compiler is installed, what extra tools need to be installed etc. It could be interactive?

make <target> would build all the build tools, and do what ever local config is required to get the build working. Then build the requested target. 

make <target>.run is more of an optional step. It could do things like launch Ovmf with your new ROM you just constructed. 

b) Debugging failures. 
Better quality error messages. 
Making the build.log from the build command a default setting(or have an abstracted defined database that can be queried), and add documentation on how to use it. Maybe make the output easier to consume?

c) Making changes. 
That can be a phase 2.

2) Build performance, complexity, and maintainability. 
I'm about to complain about the build system, but I was around when it was first invented so I'm to blame as much as anyone....

We have way too much custom Python that is very complex, hard to maintain, and slow. One thing we did better on the EDK was break up the build tools into smaller chunks (maybe too small in some cases) and I always found I could figure things out, but then again all the code was C. Another interesting data point is we added parallel build, via makefiles, to the EDK. When we moved from the EDK to the edk2 we noticed the builds slowed down a lot. I did some profiling on the edk2 build and I noticed 1,000,000 calls to regex (part of that long  ..... sequence on a build of a complex platform). It turns out we have custom Python code that is generating the makefile dependencies so it was scanning all the C code. The other issue I see is build parallelism is controlled from the Python code. It also seems the edk2 build system is not a complete solution and a lot of full platform implementations have shell scripts or batch files, that wrap the calls to build. 

So I'd like to see a build system:
1) Start with a top level makefile in the root of the tree. 
2) Replaces custom batch or shell scripts with makefiles to handle pre and post build tasks. 
3) Use a tool to construct a parser for the ekd2 build files (INF, DSC, DEC). 
	Use this data to construct the makefiles and build a database for other build components to use.
	Chris Roberts has messed around and built an Earley parser using the EBNF in the build documents. There are some problematic issues in the grammar, but some grammar streamlining might make it possible to use tooling to construct a parser vs. writing one from scratch in Python. 
4) Move to using the compiler to generate dependencies. gcc/clang support this via -M*. I think it may be a little more complex for NMAKE but there is a /showIncludes compiler flag. 
5) Have the build tool terminate after parallel makefile generation
6) Top level makefile can invoke parallel build on generated makefiles. 
7) Invoke build tool to generate FVs and FDs from makefile. 
8) Invoke custom post processing steps in the makefile. 

Thanks,

Andrew Fish