From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: hpe.com, ip: 148.163.147.86, mailfrom: brian.johnson@hpe.com) Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) by groups.io with SMTP; Fri, 03 May 2019 14:41:57 -0700 Received: from pps.filterd (m0134420.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x43LaoKL028140; Fri, 3 May 2019 21:41:57 GMT Received: from g4t3426.houston.hpe.com (g4t3426.houston.hpe.com [15.241.140.75]) by mx0b-002e3701.pphosted.com with ESMTP id 2s8vj98e92-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 03 May 2019 21:41:56 +0000 Received: from g4t3433.houston.hpecorp.net (g4t3433.houston.hpecorp.net [16.208.49.245]) by g4t3426.houston.hpe.com (Postfix) with ESMTP id F320E6A; Fri, 3 May 2019 21:41:55 +0000 (UTC) Received: from [10.33.152.19] (bjj-laptop2.americas.hpqcorp.net [10.33.152.19]) by g4t3433.houston.hpecorp.net (Postfix) with ESMTP id 91F3346; Fri, 3 May 2019 21:41:55 +0000 (UTC) Subject: Re: [edk2-devel] TianoCore Community Design Meeting Minutes To: Sean , devel@edk2.groups.io References: <70d2f499-bc28-058a-8675-069beee5835e@hpe.com> <31264.1556825609503060272@groups.io> From: "Brian J. Johnson" Message-ID: Date: Fri, 3 May 2019 16:41:55 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <31264.1556825609503060272@groups.io> X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-03_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905030139 X-MIME-Autoconverted: from 8bit to quoted-printable by mx0b-002e3701.pphosted.com id x43LaoKL028140 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 5/2/19 2:33 PM, sean.brogan via groups.io wrote: > Brian, >=20 > I would really like to hear about the challenges your team faced and=20 > issues that caused those solutions to be unworkable.=C2=A0=C2=A0Project= Mu has and=20 > continues to invest a lot in testing capabilities, build automation, an= d=20 > finding ways to improve quality that scale. >=20 Our products depend on a reference BIOS tree provided to us by a major=20 processor vendor. That tree includes portions of Edk2, plus numerous=20 proprietary additions. Each new platform starts with a new drop of=20 vendor code. They provide additional drops throughout the platform's=20 life. In the past these were distributed as zip files, but more=20 recently they have transitioned to git. We end up having to make=20 extensive changes to their code to port it to our platform. In=20 addition, we maintain internally several packages of code used on all=20 our platforms, designed to be platform-independent, plus a=20 platform-dependent package which is intended to be modified for each=20 platform. When we first started using git, we looked for a way to share our=20 all-platform code among platforms, and move our platform-dependent code=20 easily to new platforms, while making it easy to integrate new drops=20 from our vendor. We considered using git submodules, but decided that=20 would be too awkward. Modifying code in a submodule involves committing=20 in the submodule, then committing in the module containing it. This=20 seemed like too much trouble for our developers, who were all new to=20 git. Plus, it didn't interact well at all with our internal bug=20 tracking system. Basically, there was no good way to tie commits in=20 various sub- and super-modules together in a straightforward, trackable w= ay. We tried a package called gitslave (http://gitslave.sourceforge.net/),=20 which automates running git commands across a super-repo and various=20 sub-repos, with some sugar for aggregating the results into a readable=20 whole. It's a bit more transparent than submodules. But at the end of=20 the day, you're still trying to coordinate multiple git repositories.=20 We gave it a try for a month or two, but having to manage multiple=20 repositories for day-to-day work, and the lack of a single commit=20 history spanning the entire tree doomed that scheme. Developers rebelled. Ever since, we've used a single git repo per platform. We keep the=20 vendor code in a "base" branch, which we update as they provide drops,=20 then merge into our master branch. When we start a new platform, we use=20 git filter-branch to extract our all-platform and platform-dependent=20 code into a new branch, which we move to the new platform's repo and=20 merge into master. It's possible to re-extract the code if we need to=20 pick up updates. This doesn't provide total flexibility... for=20 instance, backporting a fix in our all-platform code back to a previous=20 platform involves manual cherrypicking. But for day-to-day development,=20 it lets us work in a single git tree, with a bisectable history, working=20 git-blame, commit IDs which tie directly to our bug tracker, and no=20 external tooling. It's a bit of a pain to merge a new drop (shell=20 scripts are our friends), but we're optimizing for ease of local=20 development. That seems like the best use of our resources. So I'm leery of any scheme which involves multiple repos managed by an=20 external tool. It sounds like a difficult way to do day-to-day=20 development. If Edk2 does move to split repos, we could filter-branch=20 and merge them all together into a single branch for internal use, I=20 suppose. But that does make it harder to push fixes upstream. (Not=20 that we end up doing a lot of that... we're not developing an=20 open-source BIOS, just making use of open-source upstream components.=20 So our use case is quite a bit different from Laszlo's.) We're also=20 generally focusing on one platform at a time, not trying to update=20 shared code across many at once. So our use case may be different from=20 Sean's. This got rather long... I hope it helps explain where we're coming from. --=20 Brian J. Johnson Enterprise X86 Lab Hewlett Packard Enterprise brian.johnson@hpe.com