From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: redhat.com, ip: 209.132.183.28, mailfrom: lersek@redhat.com) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by groups.io with SMTP; Mon, 06 May 2019 09:06:47 -0700 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F2FB220264; Mon, 6 May 2019 16:06:46 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-124-221.rdu2.redhat.com [10.10.124.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id D52DA18945; Mon, 6 May 2019 16:06:45 +0000 (UTC) Subject: Re: [edk2-devel] TianoCore Community Design Meeting Minutes To: devel@edk2.groups.io, brian.johnson@hpe.com, Sean References: <70d2f499-bc28-058a-8675-069beee5835e@hpe.com> <31264.1556825609503060272@groups.io> From: "Laszlo Ersek" Message-ID: Date: Mon, 6 May 2019 18:06:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 06 May 2019 16:06:47 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 05/03/19 23:41, Brian J. Johnson wrote: > On 5/2/19 2:33 PM, sean.brogan via groups.io wrote: >> Brian, >> >> I would really like to hear about the challenges your team faced and >> issues that caused those solutions to be unworkable.=C2=A0=C2=A0Projec= t Mu has >> and continues to invest a lot in testing capabilities, build >> automation, and finding ways to improve quality that scale. >> >=20 > Our products depend on a reference BIOS tree provided to us by a major > processor vendor.=C2=A0 That tree includes portions of Edk2, plus numer= ous > proprietary additions.=C2=A0 Each new platform starts with a new drop o= f > vendor code.=C2=A0 They provide additional drops throughout the platfor= m's > life.=C2=A0 In the past these were distributed as zip files, but more > recently they have transitioned to git.=C2=A0 We end up having to make > extensive changes to their code to port it to our platform.=C2=A0 In > addition, we maintain internally several packages of code used on all > our platforms, designed to be platform-independent, plus a > platform-dependent package which is intended to be modified for each > platform. >=20 > When we first started using git, we looked for a way to share our > all-platform code among platforms, and move our platform-dependent code > easily to new platforms, while making it easy to integrate new drops > from our vendor.=C2=A0 We considered using git submodules, but decided = that > would be too awkward.=C2=A0 Modifying code in a submodule involves comm= itting > in the submodule, then committing in the module containing it.=C2=A0 Th= is > seemed like too much trouble for our developers, who were all new to > git.=C2=A0 Plus, it didn't interact well at all with our internal bug > tracking system.=C2=A0 Basically, there was no good way to tie commits = in > various sub- and super-modules together in a straightforward, trackable > way. >=20 > We tried a package called gitslave (http://gitslave.sourceforge.net/), > which automates running git commands across a super-repo and various > sub-repos, with some sugar for aggregating the results into a readable > whole.=C2=A0 It's a bit more transparent than submodules.=C2=A0 But at = the end of > the day, you're still trying to coordinate multiple git repositories. W= e > gave it a try for a month or two, but having to manage multiple > repositories for day-to-day work, and the lack of a single commit > history spanning the entire tree doomed that scheme.=C2=A0 Developers r= ebelled. >=20 > Ever since, we've used a single git repo per platform.=C2=A0 We keep th= e > vendor code in a "base" branch, which we update as they provide drops, > then merge into our master branch.=C2=A0 When we start a new platform, = we use > git filter-branch to extract our all-platform and platform-dependent > code into a new branch, which we move to the new platform's repo and > merge into master.=C2=A0 It's possible to re-extract the code if we nee= d to > pick up updates.=C2=A0 This doesn't provide total flexibility... for > instance, backporting a fix in our all-platform code back to a previous > platform involves manual cherrypicking. Good point -- and cherry-picking is a first class citizen in the git toolset. Upstream projects use it all the time, between their master and stable branches. And we (RH) happen to use it all the time too. "git cherry-pick -s -x" (possibly "-e" too) is the main tool for backporting upstream patches to downstream branches. > But for day-to-day development, > it lets us work in a single git tree, with a bisectable history, workin= g > git-blame, commit IDs which tie directly to our bug tracker, and no > external tooling.=C2=A0 It's a bit of a pain to merge a new drop (shell > scripts are our friends), but we're optimizing for ease of local > development.=C2=A0 That seems like the best use of our resources. >=20 > So I'm leery of any scheme which involves multiple repos managed by an > external tool.=C2=A0 It sounds like a difficult way to do day-to-day > development.=C2=A0 If Edk2 does move to split repos, we could filter-br= anch > and merge them all together into a single branch for internal use, I > suppose.=C2=A0 But that does make it harder to push fixes upstream. Even if that re-merging worked in practica, and even if two consumers of edk2 followed the exact same procedure for re-unifying the repo, they would still end up with different commit hashes -- and that would make it more difficult to reference the same commits in upstream discussion. > (Not that we end up doing a lot of that... we're not developing an > open-source BIOS, just making use of open-source upstream components. S= o > our use case is quite a bit different from Laszlo's.)=C2=A0 We're also > generally focusing on one platform at a time, not trying to update > shared code across many at once.=C2=A0 So our use case may be different= from > Sean's. >=20 > This got rather long... I hope it helps explain where we're coming from= . It's very educational to me -- I don't have to deal with "ZIP drops" from vendors, and I'm impressed by the "commit vendor drop on side branch, merge into master separately" workflow. How difficult have your git-merges been? (You mention shell scripts.) Have you found a correlation between merge difficulty and vendor drop frequency? (I'd expect the less frequently new code is dropped, the harder the merge is.) At RH, we generally rebase our product branches on new upstream fork-off points (typically stable releases), instead of merging. (And, this strategy applies to more projects than just edk2.) Downstream, we don't create merge commits -- the downstream branches (consisting of a handful of downstream-only commits, and a large number of upstream backports, as time passes) have a linear history. The "web-like" git history is inherited from upstream up to the new fork-off point (=3D an upstream stable tag). The linear nature of the downstream branches is very suitable for "RPM", where you have a base tarball (a flat source tree generated at the upstream tag), plus a list of downstream patches that can be applied in strict (linear) sequence, for binary package building. Thanks! Laszlo