Re: [edk2-devel] TianoCore Community Design Meeting Minutes

["Brian J. Johnson" <brian.johnson@hpe.com>]

On 5/2/19 2:33 PM, sean.brogan via groups.io wrote:
> Brian,
> 
> I would really like to hear about the challenges your team faced and 
> issues that caused those solutions to be unworkable.  Project Mu has and 
> continues to invest a lot in testing capabilities, build automation, and 
> finding ways to improve quality that scale.
> 

Our products depend on a reference BIOS tree provided to us by a major 
processor vendor.  That tree includes portions of Edk2, plus numerous 
proprietary additions.  Each new platform starts with a new drop of 
vendor code.  They provide additional drops throughout the platform's 
life.  In the past these were distributed as zip files, but more 
recently they have transitioned to git.  We end up having to make 
extensive changes to their code to port it to our platform.  In 
addition, we maintain internally several packages of code used on all 
our platforms, designed to be platform-independent, plus a 
platform-dependent package which is intended to be modified for each 
platform.

When we first started using git, we looked for a way to share our 
all-platform code among platforms, and move our platform-dependent code 
easily to new platforms, while making it easy to integrate new drops 
from our vendor.  We considered using git submodules, but decided that 
would be too awkward.  Modifying code in a submodule involves committing 
in the submodule, then committing in the module containing it.  This 
seemed like too much trouble for our developers, who were all new to 
git.  Plus, it didn't interact well at all with our internal bug 
tracking system.  Basically, there was no good way to tie commits in 
various sub- and super-modules together in a straightforward, trackable way.

We tried a package called gitslave (http://gitslave.sourceforge.net/), 
which automates running git commands across a super-repo and various 
sub-repos, with some sugar for aggregating the results into a readable 
whole.  It's a bit more transparent than submodules.  But at the end of 
the day, you're still trying to coordinate multiple git repositories. 
We gave it a try for a month or two, but having to manage multiple 
repositories for day-to-day work, and the lack of a single commit 
history spanning the entire tree doomed that scheme.  Developers rebelled.

Ever since, we've used a single git repo per platform.  We keep the 
vendor code in a "base" branch, which we update as they provide drops, 
then merge into our master branch.  When we start a new platform, we use 
git filter-branch to extract our all-platform and platform-dependent 
code into a new branch, which we move to the new platform's repo and 
merge into master.  It's possible to re-extract the code if we need to 
pick up updates.  This doesn't provide total flexibility... for 
instance, backporting a fix in our all-platform code back to a previous 
platform involves manual cherrypicking.  But for day-to-day development, 
it lets us work in a single git tree, with a bisectable history, working 
git-blame, commit IDs which tie directly to our bug tracker, and no 
external tooling.  It's a bit of a pain to merge a new drop (shell 
scripts are our friends), but we're optimizing for ease of local 
development.  That seems like the best use of our resources.

So I'm leery of any scheme which involves multiple repos managed by an 
external tool.  It sounds like a difficult way to do day-to-day 
development.  If Edk2 does move to split repos, we could filter-branch 
and merge them all together into a single branch for internal use, I 
suppose.  But that does make it harder to push fixes upstream.  (Not 
that we end up doing a lot of that... we're not developing an 
open-source BIOS, just making use of open-source upstream components. 
So our use case is quite a bit different from Laszlo's.)  We're also 
generally focusing on one platform at a time, not trying to update 
shared code across many at once.  So our use case may be different from 
Sean's.

This got rather long... I hope it helps explain where we're coming from.
-- 
Brian J. Johnson
Enterprise X86 Lab
Hewlett Packard Enterprise
brian.johnson@hpe.com