From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (NAM12-DM6-obe.outbound.protection.outlook.com [40.107.243.99]) by mx.groups.io with SMTP id smtpd.web12.74.1587406992456754008 for ; Mon, 20 Apr 2020 11:23:12 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=Ci4tifpQ; spf=pass (domain: microsoft.com, ip: 40.107.243.99, mailfrom: sean.brogan@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QRuVU01KtUULj6JHjzov2FWcoi7lev43nXryr6Y4Y8FG+X2M2JZHrx9lH8RrI5JpZ7KxOIf+mXiT70g1EtnpHIGgH6DYChNuohpk5S/OdDbXAkeHnYBaXvUs8nZ2Wj50rs25ebdZKdRHgPD4V3W8KuiqfyjmUit9KN0EyFQffscdW5OX0jK+40Ed2KgA0nPgssIhQ7CjuWkymTIsngt1y+v1vhoWWrNq//sZQvglfnBipGFVIUVCSCqVVKcAq1ez5MSrRbs3K2UO4AqAAJNLLSYNi+MFbPE7dm07A/XOZjUZm9RAuKgnFvSNYocAp4S/aPptNCB10nSYWkZPe1zkHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E5Z70jH8tUvo9N1LMN0FbVY0wzf4PKcqcQktDoalwT8=; b=i34yUqAF8QthIUaajT/tVe1uWWzp0Kg56F063d0CHiSO/gzflGl5ymY6mIZvvM1gUhPbwUwjWsCs07W/FzJgGrK6QM3AM3SOqA5XdzCHeXX5n7wbZwPi63rD3H2GCbNAnpceC6R2AhFP8puW/ORwqXwZ9Ef986ZVA8xsGKGKcfy4dMX+0OI5ASSjQxudmy7H9Y2u4QJ52IzK04iDrpK9HnM777DBkhPZxrcrds5M+VwDTyWUYHYtB4VOSSuGPissUUy6fgyXmBDik3mZDy2tF5luobMVOfPl1DJELG4q2N59Bkkkupi1Vpe0mSmzZThiwSqunrqSe2xN3WJayVq2tg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E5Z70jH8tUvo9N1LMN0FbVY0wzf4PKcqcQktDoalwT8=; b=Ci4tifpQWBtwzNif7/kydEFBIew9BaM/JxLiTIIy69LJQksrRaDKWk7AG531F2lCPMadfYcwA6ez0HujkFUiH5d38f9qcYH36d/KOxVWHp7ZTrgQSfjCTfMzT3eINn/M0loYnNBdK3+aGOaaNQ3kjlqbv0HxHQO1DlL7SUvr2/c= Received: from MW2PR2101MB0924.namprd21.prod.outlook.com (2603:10b6:302:10::32) by MW2PR2101MB0891.namprd21.prod.outlook.com (2603:10b6:302:10::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.3; Mon, 20 Apr 2020 18:23:10 +0000 Received: from MW2PR2101MB0924.namprd21.prod.outlook.com ([fe80::1d19:6132:a8a9:4d2e]) by MW2PR2101MB0924.namprd21.prod.outlook.com ([fe80::1d19:6132:a8a9:4d2e%9]) with mapi id 15.20.2958.001; Mon, 20 Apr 2020 18:23:10 +0000 From: "Sean" To: "devel@edk2.groups.io" , "lersek@redhat.com" , Matthew Carlson CC: "ray.ni@intel.com" , "Ard Biesheuvel (ARM address)" Subject: Re: [edk2-devel] FW: Discussion: Basetools a separate repo Thread-Topic: [edk2-devel] FW: Discussion: Basetools a separate repo Thread-Index: AdYXQL4AXSmutfHBQpWLO8aXC9ghqQ== Date: Mon, 20 Apr 2020 18:23:10 +0000 Message-ID: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=sebrogan@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-04-20T18:23:08.9491547Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=0249bb20-4c56-44a4-870e-b899d38fd68c; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic authentication-results: spf=none (sender IP is ) smtp.mailfrom=sean.brogan@microsoft.com; x-originating-ip: [50.35.74.15] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 609b3819-8751-4ebd-bad4-08d7e557e1a3 x-ms-traffictypediagnostic: MW2PR2101MB0891:|MW2PR2101MB0891: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 03793408BA x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW2PR2101MB0924.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(4636009)(396003)(136003)(376002)(366004)(346002)(39860400002)(71200400001)(8936002)(2906002)(81156014)(8676002)(30864003)(5660300002)(7696005)(26005)(316002)(6636002)(54906003)(110136005)(53546011)(6506007)(66556008)(66446008)(44832011)(4326008)(86362001)(8990500004)(966005)(83080400001)(10290500003)(76116006)(33656002)(52536014)(186003)(66946007)(82950400001)(55016002)(9686003)(82960400001)(66476007)(478600001)(64756008);DIR:OUT;SFP:1102; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Pcl5egKzeiNC0UZwMXwEti0O/G/BhOQtmCojQeob1MfLDqkvQghwAU49qVw9KhB+s47aB9Dw+SG0djuqXK/VALGiD5fdD16FKUTalj/e5W0nSbsz989lq/76JS7e9c3Z1obwr9o1Pzc9c4Za5NiOVsQHeb3HYrmjBakPgKur0Nht9dQpoY0OI/xYqCwpm2/NDBrEGsOIIqgqY1MTAtvs6Ev8qr/MKWHRq/3TlfrVsGeR4z/3KERqKjDPdMcAFkzkvpCHVF/67QJpU/lAogyZ0DeF/N82DJuM29A65TcYTMrbMVlwt2FxGGSs0PsKebJZC/fO/xvgc+UWsjdqpprN3BAcJQmVMUmxdUIFFUe3r97sieQHgHZQuMbgC+6r/1Ndpq2vyRNBzCWc6OEhsDFl4ogl97/6pO8vWgPr1lCP58PZCZX4qCDnQVRkw2fyLdAXtE/HG8lMXRQZnUkAOb5uhMHHTcP+ubOQ28teuQXlbzON8RKMcY3h7Ot0ZdzLzhsTFhrR9AQuUjbPiExp/kRbJQ== x-ms-exchange-antispam-messagedata: j+NQ+rKwTmzrwNspPxV1r0WpTBB3faBgGMe47LFqcecIIgi3ciJzjGfe9GLcT7Pwl4RYfbrNcPLJh2UKvb0CvbMIQg5JiG/nofdb2YltBX0Yl/3/axGri26kmho7HaaHrY+79ohZxVXIUYRY9v05XvV41GYpGuZ42ryb3Q4XwjkZLUIaDEylJrsPt+ZFW6jTTMfYWRpnV60VozqH85R4SWJI/jXIkmV4h/38xSCwN46uE+tiXk9JsYSeXuePfP05EdB8NgU+sDN9/9O+RvNa2upGQnoyD8RlXiwVdGMgOTYwoGlgm3qRjPPgHbIlCvBFsXnZDWY8PzhTUnqO2idngnhgN02OKWFkd6sREn8l+8vYpgVZIIyuXuS++vSYRBtVZ2vQUNWOLPoKwEe7oNMSoktlOi2Nbgg4f05WxuI7FiHhWyPsNDBaUNj8OabSrKa7GLhOlHdJlRMZ9VCU4tZlsgGt0Bz28lOADGmZh+UweJz61BxGW6yv0tbRhWoumPqNDNw2MvC79bkE6bVBSj/QwV8Ffzu5445X9n6y1NuROFUFgkqr5NdtNbljHy+E0YEkJuOr3BdYx8i1nvavBu0coI9Jld0ie/IQ4yZoHeScUId0WV6Labw9/QIIXiLKAr4WZ7WgqccZHnHDCG55RSsfHUKk0ETAACjeLetZ6rFBfm/WLmw3t2Mqq7CIFeyjCocpwUlGR9hRTb4rYgqXvxqIMpDR4P4+wL+oevEoKAjDwzSCkvfneCuMx8Tg5eUoUytFfWnu6EHb56GW7JeN+pkt8l2xiHaRoApf59F8+1bYLS0= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 609b3819-8751-4ebd-bad4-08d7e557e1a3 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Apr 2020 18:23:10.4534 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: KpLbfCCMLLAhZV7upgOooGewqRLL37O2hVn3HPVKelB9ABtJk1hJhe0MgPZx57qLghkGUofNjDkjiYQqbNDBWQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR2101MB0891 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Laszlo, PIP vs Submodules: The issue with submodule and not using python packages (pip is really just= a super convenient helper to install that consistently) is that it require= s the paradigm of file system python path management. Think of this as edk= 1 vs edk2 public includes (if you remember back that far), basically you ju= st figure out the path from yourself to the file you want and add that. It= gives you no ability to apply consistent ways to access modules and in the= end means you just have a folder with python scripts/modules in it. It ca= uses major pain for downstream consumers when/if there is ever refactoring = (and we know there needs to be refactoring). It leads to others not being = able to extend or leverage these modules without introducing significant fr= agility. What Pip does is gives you a way to consistently install and access the py= thon modules. You can pip install from a local copy of the git repo or you= can pip install a "release" using something like pypi. Either way it does= n't change how consuming code accesses the python modules. This is the val= ue of pip in this case. =20 If there is a strong desire from the developer community to have a workflo= w that avoids pip I believe a little documentation could handle that. Pip = is just a helper. Python packages can be built from the source and install= ed. I see no reason to recommend this as it requires numerous more command= s but if avoiding pip is your goal, it is possible. More value for python package management (PIP): As we start looking at refactoring and potentially moving tools from C to = python we can take advantage of package management to lower our maintenance= burden. For example Brotli has a pypi release and thus we wouldn't need t= o carry that code. https://pypi.org/project/Brotli/ VERSION / DEPENDENCY:=20 To minimize the dependency challenges and "bisectability" I would suggest = we leverage the versioning capabilities within pip and repo tagging. With = versioning you have lots of options as you can lock to a specific version w= hich requires an update each time or you can use some sort of floating vers= ion within the tuple of version (xx.yy.zz). It also needs to be discussed = if a release would be generated for every change in the python or only on a= n important change. These two tools can make this pretty flexible. In your scenario of DEC or INF syntax change. My first pass on workflow w= ould be.=20 1. Create the issue for basetools 2. Update basetools python 3. Write the unit test that shows it works as expected 4. Check in and make a release 5. Update edk2 pip-requirements.txt to require at least this new version. = This gives you the tracking necessary to align with the tools. 6. Use this new feature in the edk2 fw code.=20 TOOLS DEVELOPER WORKFLOWS: You mentioned you wanted to make sure a developer could modify source loca= lly and test against local code. This is covered by workflow C. It is ver= y easy to manage and is one of the reasons we are proposing this change. Thanks Sean -----Original Message----- From: devel@edk2.groups.io On Behalf Of Laszlo Erse= k via groups.io Sent: Monday, April 20, 2020 5:49 AM To: Matthew Carlson Cc: devel@edk2.groups.io; ray.ni@intel.com; Ard Biesheuvel (ARM address) <= ard.biesheuvel@arm.com> Subject: [EXTERNAL] Re: [edk2-devel] FW: Discussion: Basetools a separate = repo On 04/17/20 03:40, Ni, Ray wrote: > From: Matthew Carlson=20 > > > Sent: Wednesday, April 15, 2020 4:42 AM > To: Ni, Ray >; Sean Brogan=20 > > > Subject: Discussion: Basetools a separate repo >=20 > Hello Ray, >=20 > I sent this to the discuss list on tianocore, but I think I'm moderated. >=20 > ---- >=20 > I'm looking to discuss the movement of the basetools folder in edk2 to a= separate repo and treated as a separate python project. BaseTools used to live in a separate repo (with periodic syncs into edk2),= and it was an endless source of misery for edk2. Introducing new basetools features (or fixing critical bugs in existing fe= atures), and putting those features to actual use in edk2 content, should b= e interleaved in a single shared git history. Note: this applies at the module (INF) and package (DEC) level too, not ju= st at the platform (DSC / FDF) level. For example, whenever new syntax is added to DEC or INF, universal edk2 mo= dules in MdePkg or MdeModulePkg have to delay their utilization of that syn= tax until BaseTools actually implements the syntax (and important bugs rela= ted to parsing the syntax are fixed as well). Right now this happens natura= lly, due to the shared history. If we wanted to separate BaseTools out again, the bare minimum would be a = git submodule (not pip). Even that way, bisectability would take a big hit.= And that problem -- lack of bisectability -- used to be the most painful i= ssue with the original out-of-tree BaseTools too. (You'd get a multi-thousa= nd line code drop with each sync, things would break, and you'd have no way= of programmatically / mechanically narrowing down the issue.) I understand that strict CI may help prevent such issues, and that's good.= But (a) it's still no substitute for git-bisect, and (b) a user should *re= ally* not have to install pip whatever in order to locally build edk2 prope= r. I can see myself somehow stomaching this change if it is proposed as a git= submodule. (I'm not trying to prevent "pip"; I'm trying to prevent "pip" f= rom being the primary, or exclusive, interface to consuming basetools.) With a git submodule: - only git tools are needed for a local source tree setup, for an upstream= user; - at least *some* historical accuracy is preserved between the two project= s (as the superproject, i.e. edk2, will still record *some* basetools state= s via the submodule reference); - an upstream user will still be given the chance to hack on both basetool= s and specific INF/DEC files at the same time (e.g. for bug reproduction or= feature/bugfix testing). With BaseTools being consumed via a git submodule in edk2, I'd insist on v= ery frequent submodule advances in edk2 superproject -- practically every t= ime "master" were moved in the basetools subproject (after every BaseTools = merge), the submodule ref would have to be bumped in edk2. That's the only way we'd get a halfway functional substitute for git-bisec= t. Thanks Laszlo > Why a separate repo? > The recent efforts in expanding the role of CI in the platform and core = code of EDK2 will pay big dividends in the future, leading to higher qualit= y code and easier integrations for everyone. Having basetools as it's own r= epo would simplify adding a similar CI/linting process and unit-tests to th= e basetools python code, leading to higher quality code. >=20 > A second major benefit is it would allow others that write tools for UEF= I and Edk2 to leverage this vast resource of python code using standard pyt= hon package inclusion. It would allow those tools to be decoupled from edk= 2 source and provide a consistent and managed user experience. The python = project would be published as a Pip module for those that want to leverage = the basetools modules the same way they leverage the existing python ecosys= tem. Packing basetools as a pip module, would reach the most developers and= provide the most flexibility and versatility. There are numerous way this = could be used; Pip is just one method suggested here. Other ways to levera= ge this are described below. >=20 > Why a pip module? > The investment into basetools is sizable and it has some amazing functio= nality that's difficult to reproduce. For example, the DSC, FDF, INF, and D= EC parsers handle an incredible amount of edge cases. If I wanted to write = a tool that could do a CI validation check or build a UEFI capsule, current= ly I would need to clone all of EDK2 to get basetools. If it was in a separ= ate repo and available as a wheel, as a developer, I could include it via p= ip and have that dependency managed with virtual environment or just in the= global cache. In addition, other tools that currently are difficult to bui= ld would become possible with access to the Basetools functionality. >=20 > However, there have been some concerns expressed about having a global b= asetools and the impact this has on developers with multiple workspaces (po= tentially at different versions). There are several tools and strategies t= o consider for managing this dependency. Some outlined below. >=20 > How will this change your workflow? > If this moved there would have to be a change for all platforms using ed= k2 and we have been evaluating options. These could become requirements a = developer must do before building edk2 or with minimal effort could be adde= d to the edksetup or build scripts. These can also be more easily isolated= if python virtual environments are used. >=20 > For those just consuming released versions of basetools python code: > Option A: leverage Python package management and install a released vers= ion from Pypi into a per project virtual environment. > Option B: leverage pip to install from the git repo at known=20 > tag/release (pip_requirements) >=20 > For those wanting to do active development of basetools within an edk2 p= roject: > Option C: Clone the python package source and install the package locall= y (pip install -e ./). All changes made in the local package are reflected= into your python site packages. >=20 >=20 > We have a demo of what this would look like:=20 > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith > ub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&data=3D02%7C01%7Csean > .brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf8 > 6f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927&sdata=3DGV6hzSt > b2dCypWlIhKCEiwfiVlni3Qu06Te3BnVvB90%3D&reserved=3D0 afelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fmatthe > wfcarlson%2Fedk2-pytool-base%2F&data=3D02%7C01%7Csean.brogan%40micro > soft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf86f141af91ab2d7c > d011db47%7C1%7C0%7C637229837751621927&sdata=3DGV6hzStb2dCypWlIhKCEiw > fiVlni3Qu06Te3BnVvB90%3D&reserved=3D0> > And the EDK2 that leverages it=20 > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith > ub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&d > ata=3D02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e5 > 29465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927 > &sdata=3DkQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&reserv > ed=3D0 Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools& > amp;data=3D02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe910 > 8d7e529465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372298377516 > 21927&sdata=3DkQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&r > eserved=3D0> >=20 > What happens next? > Right now, we're gathering feedback and seeing if anyone has an concerns= or platforms that this would not work for. We'd love to hear what you have= to say. Baring any serious concerns, we'd move forward with: >=20 > 1. Create new GitHub repo on tianocore for the basetools project > 2. Develop the testing, PR, and release process > 3. Release the initial version to pypi > 4. Delete the source folder in edk2 repo and replace with readme and = method to get pip version installed > 5. Continually improve basetools and add more testing >=20 > What's the long-term plan? > The current tentative long term plan is to merge some or all of basetool= s in with the existing edk2-pytool-library repo. This is still an active co= nversation, and we'd like to hear your thoughts. >=20 > Matthew Carlson > Core UEFI > Microsoft >=20 >=20 >=20 >=20 >=20 >=20