Community Discussion: Improvements to the Build Process

Community Discussion: Improvements to the Build Process

[stephano]

Andrew got this conversation started on the meeting minutes email. I 
wanted to be sure it didn't get lost in the daily noise of the list. 
I'll be pulling out some other discussions as well to give folks a 
chance to comment before our next Community Meeting (happening next 
week, invite coming soon).

Some ideas to start the discussion:

-How should we track which toolchains are being used, which are 
validated, and which are known to create reproducible builds?
-When toolchains are known to be orphaned, should they be archived or 
simply removed?
-Could we add some kconfig-like tool that allows introspection into what 
type of builds are available?
-How can we better track the code quality of BaseTools and the current 
build system on the whole?
-Should we add a "requires documentation change" flag to BZ so that it 
will be easier to compile a list of required doc changes?

I've included Andrew's email for reference.

Cheers,
Stephano



 > I'll help start the conversation on the build.
 >
 > 1) Making the process easier for newcomers.
 >
 > a) Getting it to build the 1st time
 > I think we could aspire to a more Unix like flow:
 > git clone
 > ./configure
 > make ovmf
 > make ovmf.run
 >
 > The ./config step would automate resolving dependencies. Things like 
figuring out what compiler is installed, what extra tools need to be 
installed etc. It could be interactive?
 >
 > make <target> would build all the build tools, and do what ever local 
config is required to get the build working. Then build the requested 
target.
 >
 > make <target>.run is more of an optional step. It could do things 
like launch Ovmf with your new ROM you just constructed.
 >
 > b) Debugging failures.
 > Better quality error messages.
 > Making the build.log from the build command a default setting(or have 
an abstracted defined database that can be queried), and add 
documentation on how to use it. Maybe make the output easier to consume?
 >
 > c) Making changes.
 > That can be a phase 2.
 >
 > 2) Build performance, complexity, and maintainability.
 > I'm about to complain about the build system, but I was around when 
it was first invented so I'm to blame as much as anyone....
 >
 > We have way too much custom Python that is very complex, hard to 
maintain, and slow. One thing we did better on the EDK was break up the 
build tools into smaller chunks (maybe too small in some cases) and I 
always found I could figure things out, but then again all the code was 
C. Another interesting data point is we added parallel build, via 
makefiles, to the EDK. When we moved from the EDK to the edk2 we noticed 
the builds slowed down a lot. I did some profiling on the edk2 build and 
I noticed 1,000,000 calls to regex (part of that long  ..... sequence on 
a build of a complex platform). It turns out we have custom Python code 
that is generating the makefile dependencies so it was scanning all the 
C code. The other issue I see is build parallelism is controlled from 
the Python code. It also seems the edk2 build system is not a complete 
solution and a lot of full platform implementations have shell scripts 
or batch files, that wrap the calls to build.
 >
 > So I'd like to see a build system:
 > 1) Start with a top level makefile in the root of the tree.
 > 2) Replaces custom batch or shell scripts with makefiles to handle 
pre and post build tasks.
 > 3) Use a tool to construct a parser for the ekd2 build files (INF, 
DSC, DEC). Use this data to construct the makefiles and build a database 
for other build components to use. Chris Roberts has messed around and 
built an Earley parser using the EBNF in the build documents. There are 
some problematic issues in the grammar, but some grammar streamlining 
might make it possible to use tooling to construct a parser vs. writing 
one from scratch in Python.
 > 4) Move to using the compiler to generate dependencies. gcc/clang 
support this via -M*. I think it may be a little more complex for NMAKE 
but there is a /showIncludes compiler flag.
 > 5) Have the build tool terminate after parallel makefile generation
 > 6) Top level makefile can invoke parallel build on generated makefiles.
 > 7) Invoke build tool to generate FVs and FDs from makefile.
 > 8) Invoke custom post processing steps in the makefile.
 >
 > Thanks,
 > Andrew Fish