FW: Discussion: Basetools a separate repo

FW: Discussion: Basetools a separate repo

[Ni, Ray]

[-- Attachment #1: Type: text/plain, Size: 7200 bytes --]



From: Ni, Ray
Sent: Thursday, April 16, 2020 1:41 PM
To: 'Sean Brogan' <sean.brogan@microsoft.com>; Matthew Carlson <macarl@microsoft.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.org
Subject: Discussion: Basetools a separate repo

Sean,
Matthew needs to join the edk2.groups.io and sends message from the registered mail address in order to send out message to everyone in the forum. I just checked and didn't find any blocking messages from Matthew in the groups.io website.

I copied edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org> in this mail so everyone in the forum can get notified before the meeting.

Thanks,
Ray

From: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
Sent: Thursday, April 16, 2020 1:31 PM
To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Matthew Carlson <macarl@microsoft.com<mailto:macarl@microsoft.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: RE: Discussion: Basetools a separate repo

Ray,

Can you check if the message is blocked/moderated?  I think it would be good to get the message out to the list prior to the design meeting.

Thanks
Sean


From: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>
Sent: Tuesday, April 14, 2020 8:23 PM
To: Matthew Carlson <macarl@microsoft.com<mailto:macarl@microsoft.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Subject: [EXTERNAL] RE: Discussion: Basetools a separate repo

Matthew,
Meeting agenda was updated.

Thanks,
Ray

From: Matthew Carlson <macarl@microsoft.com<mailto:macarl@microsoft.com>>
Sent: Wednesday, April 15, 2020 4:42 AM
To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
Subject: Discussion: Basetools a separate repo

Hello Ray,

I sent this to the discuss list on tianocore, but I think I'm moderated.

----

I'm looking to discuss the movement of the basetools folder in edk2 to a separate repo and treated as a separate python project.

I'd like to get on the agenda for the Thursday (April 16th) design meeting with this topic. I'm planning on talking through the reasoning and current thinking for 5 minutes and having a discussion. I expect at least 20-30 minutes of discussion.

Here's a basic overview of the what and the why behind this proposal:

Why a separate repo?
The recent efforts in expanding the role of CI in the platform and core code of EDK2 will pay big dividends in the future, leading to higher quality code and easier integrations for everyone. Having basetools as it's own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python code, leading to higher quality code.

A second major benefit is it would allow others that write tools for UEFI and Edk2 to leverage this vast resource of python code using standard python package inclusion.  It would allow those tools to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would be published as a Pip module for those that want to leverage the basetools modules the same way they leverage the existing python ecosystem. Packing basetools as a pip module, would reach the most developers and provide the most flexibility and versatility. There are numerous way this could be used; Pip is just one method suggested here.  Other ways to leverage this are described below.

Why a pip module?
The investment into basetools is sizable and it has some amazing functionality that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge cases. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need to clone all of EDK2 to get basetools. If it was in a separate repo and available as a wheel, as a developer, I could include it via pip and have that dependency managed with virtual environment or just in the global cache. In addition, other tools that currently are difficult to build would become possible with access to the Basetools functionality.

However, there have been some concerns expressed about having a global basetools and the impact this has on developers with multiple workspaces (potentially at different versions).  There are several tools and strategies to consider for managing this dependency.  Some outlined below.

How will this change your workflow?
If this moved there would have to be a change for all platforms using edk2 and we have been evaluating options.  These could become requirements a developer must do before building edk2 or with minimal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python virtual environments are used.

For those just consuming released versions of basetools python code:
Option A: leverage Python package management and install a released version from Pypi into a per project virtual environment.
Option B: leverage pip to install from the git repo at known tag/release (pip_requirements)

For those wanting to do active development of basetools within an edk2 project:
Option C: Clone the python package source and install the package locally (pip install -e ./).  All changes made in the local package are reflected into your python site packages.


We have a demo of what this would look like: https://github.com/matthewfcarlson/edk2-pytool-base/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772777671&sdata=yk559NU%2FbBbwSBEWo481w50gCObUy4h8G9BJEVIPO4I%3D&reserved=0>
And the EDK2 that leverages it https://github.com/matthewfcarlson/edk2/tree/feature/pip-basetools<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772787672&sdata=w3BGez4Tp%2BpcQ%2Fk%2F7QZ30yKvCcyHGiqzYbAFgskPjkA%3D&reserved=0>

What happens next?
Right now, we're gathering feedback and seeing if anyone has an concerns or platforms that this would not work for. We'd love to hear what you have to say. Baring any serious concerns, we'd move forward with:

  1.  Create new GitHub repo on tianocore for the basetools project
  2.  Develop the testing, PR, and release process
  3.  Release the initial version to pypi
  4.  Delete the source folder in edk2 repo and replace with readme and method to get pip version installed
  5.  Continually improve basetools and add more testing

What's the long-term plan?
The current tentative long term plan is to merge some or all of basetools in with the existing edk2-pytool-library repo. This is still an active conversation, and we'd like to hear your thoughts.

Matthew Carlson
Core UEFI
Microsoft



[-- Attachment #2: Type: text/html, Size: 15993 bytes --]

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Laszlo Ersek]

On 04/17/20 03:40, Ni, Ray wrote:

> From: Matthew Carlson <macarl@microsoft.com<mailto:macarl@microsoft.com>>
> Sent: Wednesday, April 15, 2020 4:42 AM
> To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Subject: Discussion: Basetools a separate repo
> 
> Hello Ray,
> 
> I sent this to the discuss list on tianocore, but I think I'm moderated.
> 
> ----
> 
> I'm looking to discuss the movement of the basetools folder in edk2 to a separate repo and treated as a separate python project.

BaseTools used to live in a separate repo (with periodic syncs into
edk2), and it was an endless source of misery for edk2.

Introducing new basetools features (or fixing critical bugs in existing
features), and putting those features to actual use in edk2 content,
should be interleaved in a single shared git history.

Note: this applies at the module (INF) and package (DEC) level too, not
just at the platform (DSC / FDF) level.

For example, whenever new syntax is added to DEC or INF, universal edk2
modules in MdePkg or MdeModulePkg have to delay their utilization of
that syntax until BaseTools actually implements the syntax (and
important bugs related to parsing the syntax are fixed as well). Right
now this happens naturally, due to the shared history.

If we wanted to separate BaseTools out again, the bare minimum would be
a git submodule (not pip). Even that way, bisectability would take a big
hit. And that problem -- lack of bisectability -- used to be the most
painful issue with the original out-of-tree BaseTools too. (You'd get a
multi-thousand line code drop with each sync, things would break, and
you'd have no way of programmatically / mechanically narrowing down the
issue.)

I understand that strict CI may help prevent such issues, and that's
good. But (a) it's still no substitute for git-bisect, and (b) a user
should *really* not have to install pip whatever in order to locally
build edk2 proper.

I can see myself somehow stomaching this change if it is proposed as a
git submodule. (I'm not trying to prevent "pip"; I'm trying to prevent
"pip" from being the primary, or exclusive, interface to consuming
basetools.) With a git submodule:

- only git tools are needed for a local source tree setup, for an
upstream user;

- at least *some* historical accuracy is preserved between the two
projects (as the superproject, i.e. edk2, will still record *some*
basetools states via the submodule reference);

- an upstream user will still be given the chance to hack on both
basetools and specific INF/DEC files at the same time (e.g. for bug
reproduction or feature/bugfix testing).

With BaseTools being consumed via a git submodule in edk2, I'd insist on
very frequent submodule advances in edk2 superproject -- practically
every time "master" were moved in the basetools subproject (after every
BaseTools merge), the submodule ref would have to be bumped in edk2.
That's the only way we'd get a halfway functional substitute for git-bisect.

Thanks
Laszlo



> Why a separate repo?
> The recent efforts in expanding the role of CI in the platform and core code of EDK2 will pay big dividends in the future, leading to higher quality code and easier integrations for everyone. Having basetools as it's own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python code, leading to higher quality code.
> 
> A second major benefit is it would allow others that write tools for UEFI and Edk2 to leverage this vast resource of python code using standard python package inclusion.  It would allow those tools to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would be published as a Pip module for those that want to leverage the basetools modules the same way they leverage the existing python ecosystem. Packing basetools as a pip module, would reach the most developers and provide the most flexibility and versatility. There are numerous way this could be used; Pip is just one method suggested here.  Other ways to leverage this are described below.
> 
> Why a pip module?
> The investment into basetools is sizable and it has some amazing functionality that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge cases. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need to clone all of EDK2 to get basetools. If it was in a separate repo and available as a wheel, as a developer, I could include it via pip and have that dependency managed with virtual environment or just in the global cache. In addition, other tools that currently are difficult to build would become possible with access to the Basetools functionality.
> 
> However, there have been some concerns expressed about having a global basetools and the impact this has on developers with multiple workspaces (potentially at different versions).  There are several tools and strategies to consider for managing this dependency.  Some outlined below.
> 
> How will this change your workflow?
> If this moved there would have to be a change for all platforms using edk2 and we have been evaluating options.  These could become requirements a developer must do before building edk2 or with minimal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python virtual environments are used.
> 
> For those just consuming released versions of basetools python code:
> Option A: leverage Python package management and install a released version from Pypi into a per project virtual environment.
> Option B: leverage pip to install from the git repo at known tag/release (pip_requirements)
> 
> For those wanting to do active development of basetools within an edk2 project:
> Option C: Clone the python package source and install the package locally (pip install -e ./).  All changes made in the local package are reflected into your python site packages.
> 
> 
> We have a demo of what this would look like: https://github.com/matthewfcarlson/edk2-pytool-base/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772777671&sdata=yk559NU%2FbBbwSBEWo481w50gCObUy4h8G9BJEVIPO4I%3D&reserved=0>
> And the EDK2 that leverages it https://github.com/matthewfcarlson/edk2/tree/feature/pip-basetools<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772787672&sdata=w3BGez4Tp%2BpcQ%2Fk%2F7QZ30yKvCcyHGiqzYbAFgskPjkA%3D&reserved=0>
> 
> What happens next?
> Right now, we're gathering feedback and seeing if anyone has an concerns or platforms that this would not work for. We'd love to hear what you have to say. Baring any serious concerns, we'd move forward with:
> 
>   1.  Create new GitHub repo on tianocore for the basetools project
>   2.  Develop the testing, PR, and release process
>   3.  Release the initial version to pypi
>   4.  Delete the source folder in edk2 repo and replace with readme and method to get pip version installed
>   5.  Continually improve basetools and add more testing
> 
> What's the long-term plan?
> The current tentative long term plan is to merge some or all of basetools in with the existing edk2-pytool-library repo. This is still an active conversation, and we'd like to hear your thoughts.
> 
> Matthew Carlson
> Core UEFI
> Microsoft
> 
> 
> 
> 
> 
>