From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web11.3186.1587444794988202358 for ; Mon, 20 Apr 2020 21:53:15 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=EW+Oum9q; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: kondal.r.purma@intel.com) IronPort-SDR: geEvRIJ4/h7hystkuNGqXPcuXDqkIdfYv4xqpWOXGuVrp9FHqUZCky3N15hDT8wz04TMssQVf+ i2RZynBCJNNg== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2020 21:53:10 -0700 IronPort-SDR: wIgQHJlKX+DR9vBlq8ESf6aRVUHqYMOh1j6CzP//aAIKNBBrkU+XzC4HwTho7zOpYPr7RAuUam 1DjkP8AbhDrQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,409,1580803200"; d="scan'208";a="279493548" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by fmsmga004.fm.intel.com with ESMTP; 20 Apr 2020 21:53:10 -0700 Received: from fmsmsx604.amr.corp.intel.com (10.18.126.84) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 20 Apr 2020 21:53:10 -0700 Received: from fmsmsx604.amr.corp.intel.com (10.18.126.84) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 20 Apr 2020 21:53:10 -0700 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 20 Apr 2020 21:53:10 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 20 Apr 2020 21:53:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AMR1P4tJImZ3xb8K9dW2lVyboIyF3ElEWym17t+Gvn8oVvYVEoml52c+toS6Fw17oLpokxqL0tLJfkFzOM/1fLDFVDdBjR61g/MeuJxpW5qvKec73rl8DYwZsGVhEqbyJxlbshEY+7Aysf45hK+Rh14vKCHtMyGZRoWzDjxN2t5T5069HkPr+urSsqVNEj5+nWXtuA1T03H4zB0Woghk/UE3dQvBoy9Uyms2QBk4yTjaJqF6KUoLoX8g9pr4XDpVyKNVrU9SXMga1DSOpoAj1lEZxv7mvMwGoqbUqFKrrENRmQmMOELCsTJyLB4F5ZdHg6PW1GXG75aQXiOOlMHvGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=25UJHkRteB65qj8H18hjBrYX/t8IKg3Us0lf0Syiyds=; b=TZNh+evxQQ3WLHw3PRUFrBpQQYTgaGzJ1ln9cOyVCaQ9D4nHw+qL83uYIQac1/HMKVIkA3R+WZB1A0L07UeVwB+A9DxcdcuVEF2oZyAaU8jHhWjsVkS0TmWTEr9bYbTttsK1HUauPN9nrc5QBeVJlKi5YEsDWvmv/HonkjWEqvqfdIr/EELicv87teXHYDMAG1trS3CWbxdBky5HGtVRYTSfIglsI9gQdxjLk2A7X7YZqaiYmHL3hPw+A0J7UAp3hpg7O2OEQsX4CdkLYnWiqa/x026XIVaT2jTgO+GQAr0M6DPmLMdqzizrM0tZ49p9loP6FaP8SGpzcsSZd/RMyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=25UJHkRteB65qj8H18hjBrYX/t8IKg3Us0lf0Syiyds=; b=EW+Oum9qIJzV58vMaON95ScN9t/CH8ONVvvigdI0CVoswGsVr4MzL+g41mo5i7RPA4LdkjWitxlVknvp5CDpH3VJ+18G/q15ZkM+wKj2agWmoTsX2IPz30wZBcfCTZF53OaabT/y+jdnqEkTgYj8vMjtTrLZuTGO2//G23XCYr4= Received: from BY5PR11MB3878.namprd11.prod.outlook.com (2603:10b6:a03:182::31) by BY5PR11MB3992.namprd11.prod.outlook.com (2603:10b6:a03:188::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2921.27; Tue, 21 Apr 2020 04:53:07 +0000 Received: from BY5PR11MB3878.namprd11.prod.outlook.com ([fe80::11d1:1f87:a9b0:d2d5]) by BY5PR11MB3878.namprd11.prod.outlook.com ([fe80::11d1:1f87:a9b0:d2d5%6]) with mapi id 15.20.2921.030; Tue, 21 Apr 2020 04:53:07 +0000 From: "Purma, Kondal R" To: "devel@edk2.groups.io" , "sean.brogan@microsoft.com" , "lersek@redhat.com" , "Matthew Carlson" CC: "Ni, Ray" , "Ard Biesheuvel (ARM address)" Subject: Re: [edk2-devel] FW: Discussion: Basetools a separate repo Thread-Topic: [edk2-devel] FW: Discussion: Basetools a separate repo Thread-Index: AdYXQL4AXSmutfHBQpWLO8aXC9ghqQAU18eA Date: Tue, 21 Apr 2020 04:53:06 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=sebrogan@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-04-20T18:23:08.9491547Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=0249bb20-4c56-44a4-870e-b899d38fd68c; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 authentication-results: spf=none (sender IP is ) smtp.mailfrom=kondal.r.purma@intel.com; x-originating-ip: [192.55.52.222] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 63bcb355-0ae1-41cd-1a8b-08d7e5afe216 x-ms-traffictypediagnostic: BY5PR11MB3992: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 038002787A x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3878.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(10019020)(346002)(366004)(376002)(396003)(39860400002)(136003)(110136005)(53546011)(54906003)(6506007)(9686003)(26005)(2906002)(316002)(30864003)(83080400001)(55016002)(86362001)(5660300002)(7696005)(76116006)(8936002)(186003)(8676002)(81156014)(52536014)(71200400001)(4326008)(33656002)(66446008)(64756008)(66476007)(66946007)(45080400002)(66556008)(478600001)(966005);DIR:OUT;SFP:1102; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 9Md3Ja8fLCh/7vj4GQAEsogo4sDizIL7TpRMEOSwA4UTFkThAAwpHzjRw4L/+ksv27y1S22sxEWA24VtlDnnDXbXzljJvWAL0rDnYoT4fpdGtt6E7ux7TiAh5qVfu+dCdlrjfQ5dntKwqNGU5UmZ5z00ePJIWXINAXo4HQjDtk+8nJY2EpjLmqnQaKt8OuJIVifm3NJpovA8PXMk3r7x5UDOJhW5rqv3Gw+w8bJ/DTZ2kEb9DUqSVK89YLXwyWMJG73jIuP85YVy/vLVzw7tPxJJPVQ7Lrjwr4wKBiUavzDsQxSghCr71A98tSJ+HFVA3MqH/ThavTepW0zMRhBxifPAISI/j81tDcg9Y2M9OAWu1yIruXJeBSympLoJyf0B6ODYbZgbNxmEuMAWvUTO6mdBq/Trprl11ncrxrTRuPmnPFLfHYexctJdCD6W7BayCQvsLaWOeqOYSThzNXu0JimkFJllXcFdZvv2VguoanelBY4kKjaL60V0OzCpBW07WmIKrkCPhodHUPBYGoPBGA== x-ms-exchange-antispam-messagedata: Q/4JELfCX7B2jJTh0wXdKby/Ia8aEhYkeaFJkx+xJl3oGgW+x7GXWFE/mZf7bByTOHkJMqTvcwu4qaSCLdJJQaXqwqzTtmcCcMTvdetrJFQkqo4zfKBsWmuDJHJy4shg8CYxJBTld9MWgMb3tpGMIg== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 63bcb355-0ae1-41cd-1a8b-08d7e5afe216 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Apr 2020 04:53:07.0092 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: h5+nVsIOx4erL2KLPugOy6kbtzVksTAjHHwkFg8y37Bfr7CxBmADYav8CSs6iUj7Jc4PD4YMrULFd+RxcngCR+eoEWwYR9YMmJMm1yJNIVQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB3992 Return-Path: kondal.r.purma@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Even though submodule approach solves the problem after detaching BaseTool= s from edk2, I think maintaining BaseTools with PIP is best possible approa= ch. Using PIP and versioning helps to easy environment setup for users over s= ubmodule with TAG approach. Moving all non-python based tools to python allows us to write unit tes= t for every module and helps continuous development. My understanding over all in this is 1. Detach from edk2 and start new repo. 2. Identify major high level functionalities and make them as top level mo= dules 3.Add test module and add tests for each modules up to individual methods/= functions using pytest and may be plugins for enhanced features. 4.Convert all c or other tools to python based modules=20 5.version base tools and install any mentioned version of BaseTools using = PIP 6.Virtual environment support for contributors who can setup environment = for given version of tools to fix bugs or contribute. 7. Create tight CI to trigger unittest, flake8 and other possible validati= on steps to verify patches. 8.We should also think about adding supporting plugin architecture. -----Original Message----- From: devel@edk2.groups.io On Behalf Of Sean via gr= oups.io Sent: Monday, April 20, 2020 11:23 AM To: devel@edk2.groups.io; lersek@redhat.com; Matthew Carlson Cc: Ni, Ray ; Ard Biesheuvel (ARM address) Subject: Re: [edk2-devel] FW: Discussion: Basetools a separate repo Laszlo, PIP vs Submodules: The issue with submodule and not using python packages (pip is really just= a super convenient helper to install that consistently) is that it require= s the paradigm of file system python path management. Think of this as edk= 1 vs edk2 public includes (if you remember back that far), basically you ju= st figure out the path from yourself to the file you want and add that. It= gives you no ability to apply consistent ways to access modules and in the= end means you just have a folder with python scripts/modules in it. It ca= uses major pain for downstream consumers when/if there is ever refactoring = (and we know there needs to be refactoring). It leads to others not being = able to extend or leverage these modules without introducing significant fr= agility. What Pip does is gives you a way to consistently install and access the py= thon modules. You can pip install from a local copy of the git repo or you= can pip install a "release" using something like pypi. Either way it does= n't change how consuming code accesses the python modules. This is the val= ue of pip in this case. =20 If there is a strong desire from the developer community to have a workflo= w that avoids pip I believe a little documentation could handle that. Pip = is just a helper. Python packages can be built from the source and install= ed. I see no reason to recommend this as it requires numerous more command= s but if avoiding pip is your goal, it is possible. More value for python package management (PIP): As we start looking at refactoring and potentially moving tools from C to = python we can take advantage of package management to lower our maintenance= burden. For example Brotli has a pypi release and thus we wouldn't need t= o carry that code. https://pypi.org/project/Brotli/ VERSION / DEPENDENCY:=20 To minimize the dependency challenges and "bisectability" I would suggest = we leverage the versioning capabilities within pip and repo tagging. With = versioning you have lots of options as you can lock to a specific version w= hich requires an update each time or you can use some sort of floating vers= ion within the tuple of version (xx.yy.zz). It also needs to be discussed = if a release would be generated for every change in the python or only on a= n important change. These two tools can make this pretty flexible. In your scenario of DEC or INF syntax change. My first pass on workflow w= ould be.=20 1. Create the issue for basetools 2. Update basetools python 3. Write the unit test that shows it works as expected 4. Check in and mak= e a release 5. Update edk2 pip-requirements.txt to require at least this ne= w version. This gives you the tracking necessary to align with the tools. 6. Use this new feature in the edk2 fw code.=20 TOOLS DEVELOPER WORKFLOWS: You mentioned you wanted to make sure a developer could modify source loca= lly and test against local code. This is covered by workflow C. It is ver= y easy to manage and is one of the reasons we are proposing this change. Thanks Sean -----Original Message----- From: devel@edk2.groups.io On Behalf Of Laszlo Erse= k via groups.io Sent: Monday, April 20, 2020 5:49 AM To: Matthew Carlson Cc: devel@edk2.groups.io; ray.ni@intel.com; Ard Biesheuvel (ARM address) <= ard.biesheuvel@arm.com> Subject: [EXTERNAL] Re: [edk2-devel] FW: Discussion: Basetools a separate = repo On 04/17/20 03:40, Ni, Ray wrote: > From: Matthew Carlson > > > Sent: Wednesday, April 15, 2020 4:42 AM > To: Ni, Ray >; Sean Brogan=20 > > > Subject: Discussion: Basetools a separate repo >=20 > Hello Ray, >=20 > I sent this to the discuss list on tianocore, but I think I'm moderated. >=20 > ---- >=20 > I'm looking to discuss the movement of the basetools folder in edk2 to a= separate repo and treated as a separate python project. BaseTools used to live in a separate repo (with periodic syncs into edk2),= and it was an endless source of misery for edk2. Introducing new basetools features (or fixing critical bugs in existing fe= atures), and putting those features to actual use in edk2 content, should b= e interleaved in a single shared git history. Note: this applies at the module (INF) and package (DEC) level too, not ju= st at the platform (DSC / FDF) level. For example, whenever new syntax is added to DEC or INF, universal edk2 mo= dules in MdePkg or MdeModulePkg have to delay their utilization of that syn= tax until BaseTools actually implements the syntax (and important bugs rela= ted to parsing the syntax are fixed as well). Right now this happens natura= lly, due to the shared history. If we wanted to separate BaseTools out again, the bare minimum would be a = git submodule (not pip). Even that way, bisectability would take a big hit.= And that problem -- lack of bisectability -- used to be the most painful i= ssue with the original out-of-tree BaseTools too. (You'd get a multi-thousa= nd line code drop with each sync, things would break, and you'd have no way= of programmatically / mechanically narrowing down the issue.) I understand that strict CI may help prevent such issues, and that's good.= But (a) it's still no substitute for git-bisect, and (b) a user should *re= ally* not have to install pip whatever in order to locally build edk2 prope= r. I can see myself somehow stomaching this change if it is proposed as a git= submodule. (I'm not trying to prevent "pip"; I'm trying to prevent "pip" f= rom being the primary, or exclusive, interface to consuming basetools.) With a git submodule: - only git tools are needed for a local source tree setup, for an upstream= user; - at least *some* historical accuracy is preserved between the two project= s (as the superproject, i.e. edk2, will still record *some* basetools state= s via the submodule reference); - an upstream user will still be given the chance to hack on both basetool= s and specific INF/DEC files at the same time (e.g. for bug reproduction or= feature/bugfix testing). With BaseTools being consumed via a git submodule in edk2, I'd insist on v= ery frequent submodule advances in edk2 superproject -- practically every t= ime "master" were moved in the basetools subproject (after every BaseTools = merge), the submodule ref would have to be bumped in edk2. That's the only way we'd get a halfway functional substitute for git-bisec= t. Thanks Laszlo > Why a separate repo? > The recent efforts in expanding the role of CI in the platform and core = code of EDK2 will pay big dividends in the future, leading to higher qualit= y code and easier integrations for everyone. Having basetools as it's own r= epo would simplify adding a similar CI/linting process and unit-tests to th= e basetools python code, leading to higher quality code. >=20 > A second major benefit is it would allow others that write tools for UEF= I and Edk2 to leverage this vast resource of python code using standard pyt= hon package inclusion. It would allow those tools to be decoupled from edk= 2 source and provide a consistent and managed user experience. The python = project would be published as a Pip module for those that want to leverage = the basetools modules the same way they leverage the existing python ecosys= tem. Packing basetools as a pip module, would reach the most developers and= provide the most flexibility and versatility. There are numerous way this = could be used; Pip is just one method suggested here. Other ways to levera= ge this are described below. >=20 > Why a pip module? > The investment into basetools is sizable and it has some amazing functio= nality that's difficult to reproduce. For example, the DSC, FDF, INF, and D= EC parsers handle an incredible amount of edge cases. If I wanted to write = a tool that could do a CI validation check or build a UEFI capsule, current= ly I would need to clone all of EDK2 to get basetools. If it was in a separ= ate repo and available as a wheel, as a developer, I could include it via p= ip and have that dependency managed with virtual environment or just in the= global cache. In addition, other tools that currently are difficult to bui= ld would become possible with access to the Basetools functionality. >=20 > However, there have been some concerns expressed about having a global b= asetools and the impact this has on developers with multiple workspaces (po= tentially at different versions). There are several tools and strategies t= o consider for managing this dependency. Some outlined below. >=20 > How will this change your workflow? > If this moved there would have to be a change for all platforms using ed= k2 and we have been evaluating options. These could become requirements a = developer must do before building edk2 or with minimal effort could be adde= d to the edksetup or build scripts. These can also be more easily isolated= if python virtual environments are used. >=20 > For those just consuming released versions of basetools python code: > Option A: leverage Python package management and install a released vers= ion from Pypi into a per project virtual environment. > Option B: leverage pip to install from the git repo at known=20 > tag/release (pip_requirements) >=20 > For those wanting to do active development of basetools within an edk2 p= roject: > Option C: Clone the python package source and install the package locall= y (pip install -e ./). All changes made in the local package are reflected= into your python site packages. >=20 >=20 > We have a demo of what this would look like:=20 > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith > ub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&data=3D02%7C01%7Csean > .brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf8 > 6f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927&sdata=3DGV6hzSt > b2dCypWlIhKCEiwfiVlni3Qu06Te3BnVvB90%3D&reserved=3D0 afelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub.com%2Fmatthe > wfcarlson%2Fedk2-pytool-base%2F&data=3D02%7C01%7Csean.brogan%40micro > soft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf86f141af91ab2d7c > d011db47%7C1%7C0%7C637229837751621927&sdata=3DGV6hzStb2dCypWlIhKCEiw > fiVlni3Qu06Te3BnVvB90%3D&reserved=3D0> > And the EDK2 that leverages it > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith > ub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&d > ata=3D02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e5 > 29465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927 > &sdata=3DkQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&reserv > ed=3D0 Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools& > amp;data=3D02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe910 > 8d7e529465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372298377516 > 21927&sdata=3DkQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&r > eserved=3D0> >=20 > What happens next? > Right now, we're gathering feedback and seeing if anyone has an concerns= or platforms that this would not work for. We'd love to hear what you have= to say. Baring any serious concerns, we'd move forward with: >=20 > 1. Create new GitHub repo on tianocore for the basetools project > 2. Develop the testing, PR, and release process > 3. Release the initial version to pypi > 4. Delete the source folder in edk2 repo and replace with readme and = method to get pip version installed > 5. Continually improve basetools and add more testing >=20 > What's the long-term plan? > The current tentative long term plan is to merge some or all of basetool= s in with the existing edk2-pytool-library repo. This is still an active co= nversation, and we'd like to hear your thoughts. >=20 > Matthew Carlson > Core UEFI > Microsoft >=20 >=20 >=20 >=20 >=20 >=20