Re: [edk2-devel] FW: Discussion: Basetools a separate repo

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Laszlo Ersek]

On 04/17/20 03:40, Ni, Ray wrote:

> From: Matthew Carlson <macarl@microsoft.com<mailto:macarl@microsoft.com>>
> Sent: Wednesday, April 15, 2020 4:42 AM
> To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Subject: Discussion: Basetools a separate repo
> 
> Hello Ray,
> 
> I sent this to the discuss list on tianocore, but I think I'm moderated.
> 
> ----
> 
> I'm looking to discuss the movement of the basetools folder in edk2 to a separate repo and treated as a separate python project.

BaseTools used to live in a separate repo (with periodic syncs into
edk2), and it was an endless source of misery for edk2.

Introducing new basetools features (or fixing critical bugs in existing
features), and putting those features to actual use in edk2 content,
should be interleaved in a single shared git history.

Note: this applies at the module (INF) and package (DEC) level too, not
just at the platform (DSC / FDF) level.

For example, whenever new syntax is added to DEC or INF, universal edk2
modules in MdePkg or MdeModulePkg have to delay their utilization of
that syntax until BaseTools actually implements the syntax (and
important bugs related to parsing the syntax are fixed as well). Right
now this happens naturally, due to the shared history.

If we wanted to separate BaseTools out again, the bare minimum would be
a git submodule (not pip). Even that way, bisectability would take a big
hit. And that problem -- lack of bisectability -- used to be the most
painful issue with the original out-of-tree BaseTools too. (You'd get a
multi-thousand line code drop with each sync, things would break, and
you'd have no way of programmatically / mechanically narrowing down the
issue.)

I understand that strict CI may help prevent such issues, and that's
good. But (a) it's still no substitute for git-bisect, and (b) a user
should *really* not have to install pip whatever in order to locally
build edk2 proper.

I can see myself somehow stomaching this change if it is proposed as a
git submodule. (I'm not trying to prevent "pip"; I'm trying to prevent
"pip" from being the primary, or exclusive, interface to consuming
basetools.) With a git submodule:

- only git tools are needed for a local source tree setup, for an
upstream user;

- at least *some* historical accuracy is preserved between the two
projects (as the superproject, i.e. edk2, will still record *some*
basetools states via the submodule reference);

- an upstream user will still be given the chance to hack on both
basetools and specific INF/DEC files at the same time (e.g. for bug
reproduction or feature/bugfix testing).

With BaseTools being consumed via a git submodule in edk2, I'd insist on
very frequent submodule advances in edk2 superproject -- practically
every time "master" were moved in the basetools subproject (after every
BaseTools merge), the submodule ref would have to be bumped in edk2.
That's the only way we'd get a halfway functional substitute for git-bisect.

Thanks
Laszlo



> Why a separate repo?
> The recent efforts in expanding the role of CI in the platform and core code of EDK2 will pay big dividends in the future, leading to higher quality code and easier integrations for everyone. Having basetools as it's own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python code, leading to higher quality code.
> 
> A second major benefit is it would allow others that write tools for UEFI and Edk2 to leverage this vast resource of python code using standard python package inclusion.  It would allow those tools to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would be published as a Pip module for those that want to leverage the basetools modules the same way they leverage the existing python ecosystem. Packing basetools as a pip module, would reach the most developers and provide the most flexibility and versatility. There are numerous way this could be used; Pip is just one method suggested here.  Other ways to leverage this are described below.
> 
> Why a pip module?
> The investment into basetools is sizable and it has some amazing functionality that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge cases. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need to clone all of EDK2 to get basetools. If it was in a separate repo and available as a wheel, as a developer, I could include it via pip and have that dependency managed with virtual environment or just in the global cache. In addition, other tools that currently are difficult to build would become possible with access to the Basetools functionality.
> 
> However, there have been some concerns expressed about having a global basetools and the impact this has on developers with multiple workspaces (potentially at different versions).  There are several tools and strategies to consider for managing this dependency.  Some outlined below.
> 
> How will this change your workflow?
> If this moved there would have to be a change for all platforms using edk2 and we have been evaluating options.  These could become requirements a developer must do before building edk2 or with minimal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python virtual environments are used.
> 
> For those just consuming released versions of basetools python code:
> Option A: leverage Python package management and install a released version from Pypi into a per project virtual environment.
> Option B: leverage pip to install from the git repo at known tag/release (pip_requirements)
> 
> For those wanting to do active development of basetools within an edk2 project:
> Option C: Clone the python package source and install the package locally (pip install -e ./).  All changes made in the local package are reflected into your python site packages.
> 
> 
> We have a demo of what this would look like: https://github.com/matthewfcarlson/edk2-pytool-base/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772777671&sdata=yk559NU%2FbBbwSBEWo481w50gCObUy4h8G9BJEVIPO4I%3D&reserved=0>
> And the EDK2 that leverages it https://github.com/matthewfcarlson/edk2/tree/feature/pip-basetools<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&data=02%7C01%7Csean.brogan%40microsoft.com%7C4717479fbabf487c7e5208d7e0ec4a1a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637225177772787672&sdata=w3BGez4Tp%2BpcQ%2Fk%2F7QZ30yKvCcyHGiqzYbAFgskPjkA%3D&reserved=0>
> 
> What happens next?
> Right now, we're gathering feedback and seeing if anyone has an concerns or platforms that this would not work for. We'd love to hear what you have to say. Baring any serious concerns, we'd move forward with:
> 
>   1.  Create new GitHub repo on tianocore for the basetools project
>   2.  Develop the testing, PR, and release process
>   3.  Release the initial version to pypi
>   4.  Delete the source folder in edk2 repo and replace with readme and method to get pip version installed
>   5.  Continually improve basetools and add more testing
> 
> What's the long-term plan?
> The current tentative long term plan is to merge some or all of basetools in with the existing edk2-pytool-library repo. This is still an active conversation, and we'd like to hear your thoughts.
> 
> Matthew Carlson
> Core UEFI
> Microsoft
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Sean]

Laszlo,

PIP vs Submodules:
The issue with submodule and not using python packages (pip is really just a super convenient helper to install that consistently) is that it requires the paradigm of file system python path management.  Think of this as edk1 vs edk2 public includes (if you remember back that far), basically you just figure out the path from yourself to the file you want and add that.  It gives you no ability to apply consistent ways to access modules and in the end means you just have a folder with python scripts/modules in it.  It causes major pain for downstream consumers when/if there is ever refactoring (and we know there needs to be refactoring).  It leads to others not being able to extend or leverage these modules without introducing significant fragility.

What Pip does is gives you a way to consistently install and access the python modules.  You can pip install from a local copy of the git repo or you can pip install a "release" using something like pypi.  Either way it doesn't change how consuming code accesses the python modules.  This is the value of pip in this case.   

If there is a strong desire from the developer community to have a workflow that avoids pip I believe a little documentation could handle that.  Pip is just a helper.  Python packages can be built from the source and installed.  I see no reason to recommend this as it requires numerous more commands but if avoiding pip is your goal, it is possible.

More value for python package management (PIP):
As we start looking at refactoring and potentially moving tools from C to python we can take advantage of package management to lower our maintenance burden.  For example Brotli has a pypi release and thus we wouldn't need to carry that code.  https://pypi.org/project/Brotli/


VERSION / DEPENDENCY: 
To minimize the dependency challenges and "bisectability" I would suggest we leverage the versioning capabilities within pip and repo tagging.  With versioning you have lots of options as you can lock to a specific version which requires an update each time or you can use some sort of floating version within the tuple of version (xx.yy.zz).  It also needs to be discussed if a release would be generated for every change in the python or only on an important change.  These two tools can make this pretty flexible.

In your scenario of DEC or INF syntax change.  My first pass on workflow would be. 
1. Create the issue for basetools
2. Update basetools python
3. Write the unit test that shows it works as expected
4. Check in and make a release
5. Update edk2 pip-requirements.txt to require at least this new version.  This gives you the tracking necessary to align with the tools.
6. Use this new feature in the edk2 fw code. 


TOOLS DEVELOPER WORKFLOWS:
You mentioned you wanted to make sure a developer could modify source locally and test against local code.  This is covered by workflow C.  It is very easy to manage and is one of the reasons we are proposing this change.


Thanks
Sean


-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek via groups.io
Sent: Monday, April 20, 2020 5:49 AM
To: Matthew Carlson <macarl@microsoft.com>
Cc: devel@edk2.groups.io; ray.ni@intel.com; Ard Biesheuvel (ARM address) <ard.biesheuvel@arm.com>
Subject: [EXTERNAL] Re: [edk2-devel] FW: Discussion: Basetools a separate repo

On 04/17/20 03:40, Ni, Ray wrote:

> From: Matthew Carlson 
> <macarl@microsoft.com<mailto:macarl@microsoft.com>>
> Sent: Wednesday, April 15, 2020 4:42 AM
> To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sean Brogan 
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Subject: Discussion: Basetools a separate repo
> 
> Hello Ray,
> 
> I sent this to the discuss list on tianocore, but I think I'm moderated.
> 
> ----
> 
> I'm looking to discuss the movement of the basetools folder in edk2 to a separate repo and treated as a separate python project.

BaseTools used to live in a separate repo (with periodic syncs into edk2), and it was an endless source of misery for edk2.

Introducing new basetools features (or fixing critical bugs in existing features), and putting those features to actual use in edk2 content, should be interleaved in a single shared git history.

Note: this applies at the module (INF) and package (DEC) level too, not just at the platform (DSC / FDF) level.

For example, whenever new syntax is added to DEC or INF, universal edk2 modules in MdePkg or MdeModulePkg have to delay their utilization of that syntax until BaseTools actually implements the syntax (and important bugs related to parsing the syntax are fixed as well). Right now this happens naturally, due to the shared history.

If we wanted to separate BaseTools out again, the bare minimum would be a git submodule (not pip). Even that way, bisectability would take a big hit. And that problem -- lack of bisectability -- used to be the most painful issue with the original out-of-tree BaseTools too. (You'd get a multi-thousand line code drop with each sync, things would break, and you'd have no way of programmatically / mechanically narrowing down the
issue.)

I understand that strict CI may help prevent such issues, and that's good. But (a) it's still no substitute for git-bisect, and (b) a user should *really* not have to install pip whatever in order to locally build edk2 proper.

I can see myself somehow stomaching this change if it is proposed as a git submodule. (I'm not trying to prevent "pip"; I'm trying to prevent "pip" from being the primary, or exclusive, interface to consuming
basetools.) With a git submodule:

- only git tools are needed for a local source tree setup, for an upstream user;

- at least *some* historical accuracy is preserved between the two projects (as the superproject, i.e. edk2, will still record *some* basetools states via the submodule reference);

- an upstream user will still be given the chance to hack on both basetools and specific INF/DEC files at the same time (e.g. for bug reproduction or feature/bugfix testing).

With BaseTools being consumed via a git submodule in edk2, I'd insist on very frequent submodule advances in edk2 superproject -- practically every time "master" were moved in the basetools subproject (after every BaseTools merge), the submodule ref would have to be bumped in edk2.
That's the only way we'd get a halfway functional substitute for git-bisect.

Thanks
Laszlo



> Why a separate repo?
> The recent efforts in expanding the role of CI in the platform and core code of EDK2 will pay big dividends in the future, leading to higher quality code and easier integrations for everyone. Having basetools as it's own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python code, leading to higher quality code.
> 
> A second major benefit is it would allow others that write tools for UEFI and Edk2 to leverage this vast resource of python code using standard python package inclusion.  It would allow those tools to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would be published as a Pip module for those that want to leverage the basetools modules the same way they leverage the existing python ecosystem. Packing basetools as a pip module, would reach the most developers and provide the most flexibility and versatility. There are numerous way this could be used; Pip is just one method suggested here.  Other ways to leverage this are described below.
> 
> Why a pip module?
> The investment into basetools is sizable and it has some amazing functionality that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge cases. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need to clone all of EDK2 to get basetools. If it was in a separate repo and available as a wheel, as a developer, I could include it via pip and have that dependency managed with virtual environment or just in the global cache. In addition, other tools that currently are difficult to build would become possible with access to the Basetools functionality.
> 
> However, there have been some concerns expressed about having a global basetools and the impact this has on developers with multiple workspaces (potentially at different versions).  There are several tools and strategies to consider for managing this dependency.  Some outlined below.
> 
> How will this change your workflow?
> If this moved there would have to be a change for all platforms using edk2 and we have been evaluating options.  These could become requirements a developer must do before building edk2 or with minimal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python virtual environments are used.
> 
> For those just consuming released versions of basetools python code:
> Option A: leverage Python package management and install a released version from Pypi into a per project virtual environment.
> Option B: leverage pip to install from the git repo at known 
> tag/release (pip_requirements)
> 
> For those wanting to do active development of basetools within an edk2 project:
> Option C: Clone the python package source and install the package locally (pip install -e ./).  All changes made in the local package are reflected into your python site packages.
> 
> 
> We have a demo of what this would look like: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&amp;data=02%7C01%7Csean
> .brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927&amp;sdata=GV6hzSt
> b2dCypWlIhKCEiwfiVlni3Qu06Te3BnVvB90%3D&amp;reserved=0<https://nam06.s
> afelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthe
> wfcarlson%2Fedk2-pytool-base%2F&amp;data=02%7C01%7Csean.brogan%40micro
> soft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637229837751621927&amp;sdata=GV6hzStb2dCypWlIhKCEiw
> fiVlni3Qu06Te3BnVvB90%3D&amp;reserved=0>
> And the EDK2 that leverages it 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&amp;d
> ata=02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e5
> 29465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927
> &amp;sdata=kQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&amp;reserv
> ed=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2
> Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&
> amp;data=02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe910
> 8d7e529465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372298377516
> 21927&amp;sdata=kQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&amp;r
> eserved=0>
> 
> What happens next?
> Right now, we're gathering feedback and seeing if anyone has an concerns or platforms that this would not work for. We'd love to hear what you have to say. Baring any serious concerns, we'd move forward with:
> 
>   1.  Create new GitHub repo on tianocore for the basetools project
>   2.  Develop the testing, PR, and release process
>   3.  Release the initial version to pypi
>   4.  Delete the source folder in edk2 repo and replace with readme and method to get pip version installed
>   5.  Continually improve basetools and add more testing
> 
> What's the long-term plan?
> The current tentative long term plan is to merge some or all of basetools in with the existing edk2-pytool-library repo. This is still an active conversation, and we'd like to hear your thoughts.
> 
> Matthew Carlson
> Core UEFI
> Microsoft
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Purma, Kondal R]

Even though submodule approach solves the problem after  detaching BaseTools from edk2, I think maintaining BaseTools with PIP is best possible approach.

Using PIP and versioning helps to easy environment setup for users  over submodule with TAG approach.

 Moving all non-python based tools to python allows  us to write  unit test for every module and helps continuous development.

My understanding over all in this  is

1. Detach from edk2 and start new repo.
2. Identify major high level functionalities and make them as top level modules
3.Add test module and add tests for each modules up to individual methods/functions using pytest and may be plugins for enhanced features.
4.Convert all c or other tools to python based modules 
5.version base tools and install any mentioned version of BaseTools using PIP
6.Virtual environment support  for contributors who can setup environment for given version of tools to  fix bugs or contribute.
7. Create tight CI to trigger unittest, flake8 and other possible validation steps to verify patches.
8.We should also think about adding supporting plugin architecture.

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean via groups.io
Sent: Monday, April 20, 2020 11:23 AM
To: devel@edk2.groups.io; lersek@redhat.com; Matthew Carlson <macarl@microsoft.com>
Cc: Ni, Ray <ray.ni@intel.com>; Ard Biesheuvel (ARM address) <ard.biesheuvel@arm.com>
Subject: Re: [edk2-devel] FW: Discussion: Basetools a separate repo

Laszlo,

PIP vs Submodules:
The issue with submodule and not using python packages (pip is really just a super convenient helper to install that consistently) is that it requires the paradigm of file system python path management.  Think of this as edk1 vs edk2 public includes (if you remember back that far), basically you just figure out the path from yourself to the file you want and add that.  It gives you no ability to apply consistent ways to access modules and in the end means you just have a folder with python scripts/modules in it.  It causes major pain for downstream consumers when/if there is ever refactoring (and we know there needs to be refactoring).  It leads to others not being able to extend or leverage these modules without introducing significant fragility.

What Pip does is gives you a way to consistently install and access the python modules.  You can pip install from a local copy of the git repo or you can pip install a "release" using something like pypi.  Either way it doesn't change how consuming code accesses the python modules.  This is the value of pip in this case.   

If there is a strong desire from the developer community to have a workflow that avoids pip I believe a little documentation could handle that.  Pip is just a helper.  Python packages can be built from the source and installed.  I see no reason to recommend this as it requires numerous more commands but if avoiding pip is your goal, it is possible.

More value for python package management (PIP):
As we start looking at refactoring and potentially moving tools from C to python we can take advantage of package management to lower our maintenance burden.  For example Brotli has a pypi release and thus we wouldn't need to carry that code.  https://pypi.org/project/Brotli/


VERSION / DEPENDENCY: 
To minimize the dependency challenges and "bisectability" I would suggest we leverage the versioning capabilities within pip and repo tagging.  With versioning you have lots of options as you can lock to a specific version which requires an update each time or you can use some sort of floating version within the tuple of version (xx.yy.zz).  It also needs to be discussed if a release would be generated for every change in the python or only on an important change.  These two tools can make this pretty flexible.

In your scenario of DEC or INF syntax change.  My first pass on workflow would be. 
1. Create the issue for basetools
2. Update basetools python
3. Write the unit test that shows it works as expected 4. Check in and make a release 5. Update edk2 pip-requirements.txt to require at least this new version.  This gives you the tracking necessary to align with the tools.
6. Use this new feature in the edk2 fw code. 


TOOLS DEVELOPER WORKFLOWS:
You mentioned you wanted to make sure a developer could modify source locally and test against local code.  This is covered by workflow C.  It is very easy to manage and is one of the reasons we are proposing this change.


Thanks
Sean


-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek via groups.io
Sent: Monday, April 20, 2020 5:49 AM
To: Matthew Carlson <macarl@microsoft.com>
Cc: devel@edk2.groups.io; ray.ni@intel.com; Ard Biesheuvel (ARM address) <ard.biesheuvel@arm.com>
Subject: [EXTERNAL] Re: [edk2-devel] FW: Discussion: Basetools a separate repo

On 04/17/20 03:40, Ni, Ray wrote:

> From: Matthew Carlson
> <macarl@microsoft.com<mailto:macarl@microsoft.com>>
> Sent: Wednesday, April 15, 2020 4:42 AM
> To: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Sean Brogan 
> <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>
> Subject: Discussion: Basetools a separate repo
> 
> Hello Ray,
> 
> I sent this to the discuss list on tianocore, but I think I'm moderated.
> 
> ----
> 
> I'm looking to discuss the movement of the basetools folder in edk2 to a separate repo and treated as a separate python project.

BaseTools used to live in a separate repo (with periodic syncs into edk2), and it was an endless source of misery for edk2.

Introducing new basetools features (or fixing critical bugs in existing features), and putting those features to actual use in edk2 content, should be interleaved in a single shared git history.

Note: this applies at the module (INF) and package (DEC) level too, not just at the platform (DSC / FDF) level.

For example, whenever new syntax is added to DEC or INF, universal edk2 modules in MdePkg or MdeModulePkg have to delay their utilization of that syntax until BaseTools actually implements the syntax (and important bugs related to parsing the syntax are fixed as well). Right now this happens naturally, due to the shared history.

If we wanted to separate BaseTools out again, the bare minimum would be a git submodule (not pip). Even that way, bisectability would take a big hit. And that problem -- lack of bisectability -- used to be the most painful issue with the original out-of-tree BaseTools too. (You'd get a multi-thousand line code drop with each sync, things would break, and you'd have no way of programmatically / mechanically narrowing down the
issue.)

I understand that strict CI may help prevent such issues, and that's good. But (a) it's still no substitute for git-bisect, and (b) a user should *really* not have to install pip whatever in order to locally build edk2 proper.

I can see myself somehow stomaching this change if it is proposed as a git submodule. (I'm not trying to prevent "pip"; I'm trying to prevent "pip" from being the primary, or exclusive, interface to consuming
basetools.) With a git submodule:

- only git tools are needed for a local source tree setup, for an upstream user;

- at least *some* historical accuracy is preserved between the two projects (as the superproject, i.e. edk2, will still record *some* basetools states via the submodule reference);

- an upstream user will still be given the chance to hack on both basetools and specific INF/DEC files at the same time (e.g. for bug reproduction or feature/bugfix testing).

With BaseTools being consumed via a git submodule in edk2, I'd insist on very frequent submodule advances in edk2 superproject -- practically every time "master" were moved in the basetools subproject (after every BaseTools merge), the submodule ref would have to be bumped in edk2.
That's the only way we'd get a halfway functional substitute for git-bisect.

Thanks
Laszlo



> Why a separate repo?
> The recent efforts in expanding the role of CI in the platform and core code of EDK2 will pay big dividends in the future, leading to higher quality code and easier integrations for everyone. Having basetools as it's own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python code, leading to higher quality code.
> 
> A second major benefit is it would allow others that write tools for UEFI and Edk2 to leverage this vast resource of python code using standard python package inclusion.  It would allow those tools to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would be published as a Pip module for those that want to leverage the basetools modules the same way they leverage the existing python ecosystem. Packing basetools as a pip module, would reach the most developers and provide the most flexibility and versatility. There are numerous way this could be used; Pip is just one method suggested here.  Other ways to leverage this are described below.
> 
> Why a pip module?
> The investment into basetools is sizable and it has some amazing functionality that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge cases. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need to clone all of EDK2 to get basetools. If it was in a separate repo and available as a wheel, as a developer, I could include it via pip and have that dependency managed with virtual environment or just in the global cache. In addition, other tools that currently are difficult to build would become possible with access to the Basetools functionality.
> 
> However, there have been some concerns expressed about having a global basetools and the impact this has on developers with multiple workspaces (potentially at different versions).  There are several tools and strategies to consider for managing this dependency.  Some outlined below.
> 
> How will this change your workflow?
> If this moved there would have to be a change for all platforms using edk2 and we have been evaluating options.  These could become requirements a developer must do before building edk2 or with minimal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python virtual environments are used.
> 
> For those just consuming released versions of basetools python code:
> Option A: leverage Python package management and install a released version from Pypi into a per project virtual environment.
> Option B: leverage pip to install from the git repo at known 
> tag/release (pip_requirements)
> 
> For those wanting to do active development of basetools within an edk2 project:
> Option C: Clone the python package source and install the package locally (pip install -e ./).  All changes made in the local package are reflected into your python site packages.
> 
> 
> We have a demo of what this would look like: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fmatthewfcarlson%2Fedk2-pytool-base%2F&amp;data=02%7C01%7Csean
> .brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927&amp;sdata=GV6hzSt
> b2dCypWlIhKCEiwfiVlni3Qu06Te3BnVvB90%3D&amp;reserved=0<https://nam06.s
> afelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmatthe
> wfcarlson%2Fedk2-pytool-base%2F&amp;data=02%7C01%7Csean.brogan%40micro
> soft.com%7Cd02dc272480b4a2cbe9108d7e529465f%7C72f988bf86f141af91ab2d7c
> d011db47%7C1%7C0%7C637229837751621927&amp;sdata=GV6hzStb2dCypWlIhKCEiw
> fiVlni3Qu06Te3BnVvB90%3D&amp;reserved=0>
> And the EDK2 that leverages it
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&amp;d
> ata=02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe9108d7e5
> 29465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637229837751621927
> &amp;sdata=kQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&amp;reserv
> ed=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2
> Fgithub.com%2Fmatthewfcarlson%2Fedk2%2Ftree%2Ffeature%2Fpip-basetools&
> amp;data=02%7C01%7Csean.brogan%40microsoft.com%7Cd02dc272480b4a2cbe910
> 8d7e529465f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6372298377516
> 21927&amp;sdata=kQloOPT46%2F3ryp7AgYpkiYqSjPm7Yl40oOHPkjNtm9E%3D&amp;r
> eserved=0>
> 
> What happens next?
> Right now, we're gathering feedback and seeing if anyone has an concerns or platforms that this would not work for. We'd love to hear what you have to say. Baring any serious concerns, we'd move forward with:
> 
>   1.  Create new GitHub repo on tianocore for the basetools project
>   2.  Develop the testing, PR, and release process
>   3.  Release the initial version to pypi
>   4.  Delete the source folder in edk2 repo and replace with readme and method to get pip version installed
>   5.  Continually improve basetools and add more testing
> 
> What's the long-term plan?
> The current tentative long term plan is to merge some or all of basetools in with the existing edk2-pytool-library repo. This is still an active conversation, and we'd like to hear your thoughts.
> 
> Matthew Carlson
> Core UEFI
> Microsoft
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Laszlo Ersek]

On 04/20/20 20:23, Sean Brogan wrote:
> Laszlo,
> 
> PIP vs Submodules:
> The issue with submodule and not using python packages (pip is really just a super convenient helper to install that consistently) is that it requires the paradigm of file system python path management.  Think of this as edk1 vs edk2 public includes (if you remember back that far), basically you just figure out the path from yourself to the file you want and add that.  It gives you no ability to apply consistent ways to access modules and in the end means you just have a folder with python scripts/modules in it.  It causes major pain for downstream consumers when/if there is ever refactoring (and we know there needs to be refactoring).  It leads to others not being able to extend or leverage these modules without introducing significant fragility.  
> 
> What Pip does is gives you a way to consistently install and access the python modules.  You can pip install from a local copy of the git repo or you can pip install a "release" using something like pypi.  Either way it doesn't change how consuming code accesses the python modules.  This is the value of pip in this case.   
> 
> If there is a strong desire from the developer community to have a workflow that avoids pip I believe a little documentation could handle that.  Pip is just a helper.  Python packages can be built from the source and installed.  I see no reason to recommend this as it requires numerous more commands but if avoiding pip is your goal, it is possible.  
> 
> More value for python package management (PIP):
> As we start looking at refactoring and potentially moving tools from C to python we can take advantage of package management to lower our maintenance burden.  For example Brotli has a pypi release and thus we wouldn't need to carry that code.  https://pypi.org/project/Brotli/
> 
> 
> VERSION / DEPENDENCY: 
> To minimize the dependency challenges and "bisectability" I would suggest we leverage the versioning capabilities within pip and repo tagging.  With versioning you have lots of options as you can lock to a specific version which requires an update each time or you can use some sort of floating version within the tuple of version (xx.yy.zz).  It also needs to be discussed if a release would be generated for every change in the python or only on an important change.  These two tools can make this pretty flexible.  
> 
> In your scenario of DEC or INF syntax change.  My first pass on workflow would be. 
> 1. Create the issue for basetools
> 2. Update basetools python
> 3. Write the unit test that shows it works as expected
> 4. Check in and make a release
> 5. Update edk2 pip-requirements.txt to require at least this new version.  This gives you the tracking necessary to align with the tools.
> 6. Use this new feature in the edk2 fw code. 
> 
> 
> TOOLS DEVELOPER WORKFLOWS:
> You mentioned you wanted to make sure a developer could modify source locally and test against local code.  This is covered by workflow C.  It is very easy to manage and is one of the reasons we are proposing this change.  

Please provide a step by step guide for the following use case:

- Edk2 at commit C1 uses a particular new BaseTools feature. However,
there is a regression in BaseTools, and an independent MdeModulePkg
module (unrelated to the new BaseTools feature) no longer builds.

- Edk2 at commit C0 works fine. At this point in time (i.e. when C0 was
made), the BaseTools feature / regression didn't exist yet.

- The edk2 clone is checked out at C1, and the breakage is witnessed.
The user wants to return to state C0, and get a functional build. The
user wants to revert BaseTools too, to the same version that was
"BaseTools master" at the time edk2 commit C0 was made.

- The user is not allowed to run commands as root, and they also don't
want to confuse other python programs they may have installed under
their $HOME.

- How *exactly* does the user determine the exact BaseTools version, in
retrospect, for when commit C0 was the HEAD of edk2 master? For example,
is it

  git show C0:pip-requirements.txt

?

- What are the *precise* commands the user needs to run, for moving from
state C1 to C0, covering both edk2 and BaseTools?

I know I can run "git checkout C0" in the edk2 tree, to return to that
point in the past. What are the exact commands I need to run for
synching BaseTools to that state?


If the C1->C0 transition requires (re)installing BaseTools to some
private directory under $HOME, using a "virtual environment" or some
such with pip, that's entirely fine. I'm happy to do that. I've just
never done it before, and I want to make it absolutely sure that "moving
(back) through time" like this is doable without interfering with either
system Python state, or the user's general ($HOME) Python state.

For reference, with a git submodule, the BaseTools "sync" command would be

  git submodule update --init --force

Thanks
Laszlo

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Matthew Carlson]

[-- Attachment #1: Type: text/plain, Size: 3249 bytes --]

Hey Lazlo,

This is a great point of discussion. Just to make sure we're on the same page, let me paraphrase your scenario. Two features at the same time or a improperly tested new feature were committed in BaseTools and a new release was spun out. EDK2 was updated to have it's pip-requirements.txt to point to the new version committed. In an ideal world, CI should catch the build break for MdeModulePkg but let's say it didn't catch it. Your scenario is, how do you go back to the previous version of basetools that EDK2 used or perhaps go the the version of BaseTools that last worked.

There are several great ways to do this. The release cadence of a pip based BaseTools hasn't really been discussed but one option was every feature in BaseTools would be a new release (this would be easy to do via pipelines). The exact versioning would need to be discussed but there would likely be some notion of breaking changes with major and minor version. This means that you would be able to easily pip install the basetools 1:1 with the feature commit history. In the scenario you mentioned, it would be easy to roll back to C0, just "pip install -r the previous requirements file". If a virtual environment is setup, this pip install would be entirely localized to the workspace the user is operating in. The git show command you mentioned would show the pip-requirements file. We would likely tag the pip_requirements file to a specific release of BaseTools and there might be another file like pip_suggestions that would allow for a looser version such as 0.3.x were any minor version in 0.3 is fine. That said, determining what exact basetools was used in a given build would likely be best accomplished through build artifacts. We have a BuildToolsReport generator that specifically does this ( https://github.com/tianocore/edk2/tree/master/BaseTools/Plugin/BuildToolsReport ) and it would generate a report that would detail the exact pip version used in that build. Without build artifacts, it grows a little bit more fuzzy, but you can have reasonable confidence since it would need to at least conform to pip suggestions and likely was on pip_requirements. To summarize, moving back in time is simple and even trivial with minimal delay to a developer (pip installs are usually sub-second procedures). There might even be a part of edksetup that might do the pip install for you if that's what the community wants.

Going to the next step, if you wanted to debug the BaseTools at C1, you might want to generate a diff of the basetools to try and see where the error and compare the diff. This I think better addresses your question of bisectability (not sure if that's a word). You could clone the basetools repo and do a pip install -e {path to basetools}. You would checkout the version in C1 pip_requirements and generate the diff between it and C0. Then you could apply changes or revert changes and test them locally as the basetools used in your edk2 repo will be the local repo you cloned. Once you're done, simply do a pip install of your pip_requirements file and the symlinks will be overwritten and things will go back to the way they were before.

Hopefully that addresses your concerns.

Thanks,
- Matthew Carlson

[-- Attachment #2: Type: text/html, Size: 3432 bytes --]

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Laszlo Ersek]

Hello Matthew,

On 04/22/20 02:05, macarl via [] wrote:
> Hey Lazlo,
> 
> This is a great point of discussion. Just to make sure we're on the same page, let me paraphrase your scenario. Two features at the same time or a improperly tested new feature were committed in BaseTools and a new release was spun out. EDK2 was updated to have it's pip-requirements.txt to point to the new version committed. In an ideal world, CI should catch the build break for MdeModulePkg but let's say it didn't catch it. Your scenario is, how do you go back to the previous version of basetools that EDK2 used or perhaps go the the version of BaseTools that last worked.
> 
> There are several great ways to do this. The release cadence of a pip based BaseTools hasn't really been discussed but one option was every feature in BaseTools would be a new release (this would be easy to do via pipelines). The exact versioning would need to be discussed but there would likely be some notion of breaking changes with major and minor version. This means that you would be able to easily pip install the basetools 1:1 with the feature commit history. In the scenario you mentioned, it would be easy to roll back to C0, just "pip install -r the previous requirements file". If a virtual environment is setup, this pip install would be entirely localized to the workspace the user is operating in. The git show command you mentioned would show the pip-requirements file. We would likely tag the pip_requirements file to a specific release of BaseTools and there might be another file like pip_suggestions that would allow for a looser version such as 0.3.x were any minor version in 0.3 is fine. That said, determining what exact basetools was used in a given build would likely be best accomplished through build artifacts. We have a BuildToolsReport generator that specifically does this ( https://github.com/tianocore/edk2/tree/master/BaseTools/Plugin/BuildToolsReport ) and it would generate a report that would detail the exact pip version used in that build. Without build artifacts, it grows a little bit more fuzzy, but you can have reasonable confidence since it would need to at least conform to pip suggestions and likely was on pip_requirements. To summarize, moving back in time is simple and even trivial with minimal delay to a developer (pip installs are usually sub-second procedures). There might even be a part of edksetup that might do the pip install for you if that's what the community wants.
> 
> Going to the next step, if you wanted to debug the BaseTools at C1, you might want to generate a diff of the basetools to try and see where the error and compare the diff. This I think better addresses your question of bisectability (not sure if that's a word). You could clone the basetools repo and do a pip install -e {path to basetools}. You would checkout the version in C1 pip_requirements and generate the diff between it and C0. Then you could apply changes or revert changes and test them locally as the basetools used in your edk2 repo will be the local repo you cloned. Once you're done, simply do a pip install of your pip_requirements file and the symlinks will be overwritten and things will go back to the way they were before.
> 
> Hopefully that addresses your concerns.

Thanks for the explanation. So here's how I understand:

* When I move around the edk2 git history, the "pip-requirements.txt"
file is tracked as an integral part of that history. So, at whichever
edk2 commit I've checked out the edk2 tree, I can just run "pip install
-r pip-requirements.txt", and then the *matching* basetools state will
be installed in my virtual environment. Is that correct?

If it is, then it sounds good. It is not very different from running a
git submodule update command.


* Where would "pip install -r pip-requirements.txt" *fetch* the required
basetools version from? Is it always a network download, or can it be
performed locally?


* Can you please explain the effects of the "pip install -e
<basetools-git-checkout-dir>" command in more detail? I've tried
googling it, not very successfully.

If I understand correctly, it installs the current checkout of basetools
from *that* directory, into the virtual environment. Is that right?

Is this something that works "automatically", or would basetools have to
be developed (in its own git repo) explicitly such that it be compatible
with "pip install -e"?

I'm asking because I would like to avoid a situation where a particular
BaseTools checkout is not installable with "pip install -e"; that is,
without "packaging up" that BaseTools state into some package /
distribution format. It would be great if "pip install -e" would
automatically work from any random BaseTools checkout.


* Regarding the "release cadence of a pip based BaseTools", I'd like
edk2's "pip-requirements.txt" to track the "BaseTools master branch"
very-very closely. I'd like edk2 to consume BaseTools changes
immediately, and with a fine granularity. Problems in big BaseTools
releases are difficult to recover from.

Thanks!
Laszlo

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Matthew Carlson]

[-- Attachment #1: Type: text/plain, Size: 1800 bytes --]

I think you've got it. The version of basetools will be carried via a pip-requirements file.

Where would "pip install -r pip-requirements.txt" *fetch* the required basetools version from?
I believe that it has a cache internal to pip. But if you want to fetch a basetools that hasn't been fetched before, yes, it would require a network download. Alternatively, if you're without internet access, you can check it out locally and use the pip install -e.

Can you please explain the effects of the "pip install -e in more detail?
So I'm a little murky on how it works but I believe that it creates symlinks or some other mechanism to repoint the global python module (global meaning your pip install or virtualenv). This is something that works automatically. The setup.py in the root of the repo takes care of this since it's used also to package the pip module. You can do pip install -e., and it just works (tm). Any basetools commit you check out (once it is in it's own repo, going back a given basetools commit two years ago is unfortunately not as easy) will work with pip install -e with not setup or configuration on your part as a developer. I use this whenever I work on the pytools and it works quite well. It will also show up in the BuildToolsReport and pip freeze as being locally installed and it will tell you the git commit it is currently at.

I agree- I think we're trending towards very very frequent releases for basetools to provide high granularity. But that is pending community feedback and the decision of the tools maintainer. It is certainly trivial to automate the release pipeline in such a way that a release with every commit that passes CI is easy and mostly automatic for the maintainers.

Thank you for the excellent questions!

--
- Matthew Carlson

[-- Attachment #2: Type: text/html, Size: 2024 bytes --]

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Laszlo Ersek]

On 04/22/20 21:30, macarl via [] wrote:
> I think you've got it. The version of basetools will be carried via a pip-requirements file.
> 
> Where would "pip install -r pip-requirements.txt" *fetch* the required basetools version from?
> I believe that it has a cache internal to pip. But if you want to fetch a basetools that hasn't been fetched before, yes, it would require a network download. Alternatively, if you're without internet access, you can check it out locally and use the pip install -e.
> 
> Can you please explain the effects of the "pip install -e in more detail?
> So I'm a little murky on how it works but I believe that it creates symlinks or some other mechanism to repoint the global python module (global meaning your pip install or virtualenv). This is something that works automatically. The setup.py in the root of the repo takes care of this since it's used also to package the pip module. You can do pip install -e., and it just works (tm). Any basetools commit you check out (once it is in it's own repo, going back a given basetools commit two years ago is unfortunately not as easy) will work with pip install -e with not setup or configuration on your part as a developer. I use this whenever I work on the pytools and it works quite well. It will also show up in the BuildToolsReport and pip freeze as being locally installed and it will tell you the git commit it is currently at.
> 
> I agree- I think we're trending towards very very frequent releases for basetools to provide high granularity. But that is pending community feedback and the decision of the tools maintainer. It is certainly trivial to automate the release pipeline in such a way that a release with every commit that passes CI is easy and mostly automatic for the maintainers.

Thank you. All of the above sounds nice.

I'm carefully optimistic about these changes, and am not opposing them
at this point. I hope I can find the time to test these features once
they are being proposed in code form.

Regarding the frequent (automatic) release tagging for the separate
basetools repo -- that too sounds great; I just think it will need
manual work on the edk2 side, i.e. to keep "pip-requirements.txt"
bumped. While that work sounds quite "menial", I much hope that it can
be integrated into the basetools development requirements -- "whenever
you get something into basetools, and the system tags a new release
automatically, be sure to post a patch to edk2 for advancing it to the
new basetools release".

Thanks!
Laszlo

Re: [edk2-devel] FW: Discussion: Basetools a separate repo

[Purma, Kondal R]

> Can you please explain the effects of the "pip install -e in more detail?

"Pip install -e" basically editable mode feature, means it installs setuptools in development mode. We can install any prjects in editable mode and also supports local or  from git or svn supported mode.

Pip also supports pinned version numbers (which can protect bugs or incompatibilities) and Hash-checking mode. 

I think for long term we are moving right direction .

Thanks,
Kondal.
-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
Sent: Friday, April 24, 2020 2:28 AM
To: Matthew Carlson <macarl@microsoft.com>; devel@edk2.groups.io
Subject: Re: [edk2-devel] FW: Discussion: Basetools a separate repo

On 04/22/20 21:30, macarl via [] wrote:
> I think you've got it. The version of basetools will be carried via a pip-requirements file.
> 
> Where would "pip install -r pip-requirements.txt" *fetch* the required basetools version from?
> I believe that it has a cache internal to pip. But if you want to fetch a basetools that hasn't been fetched before, yes, it would require a network download. Alternatively, if you're without internet access, you can check it out locally and use the pip install -e.
> 
> Can you please explain the effects of the "pip install -e in more detail?
> So I'm a little murky on how it works but I believe that it creates symlinks or some other mechanism to repoint the global python module (global meaning your pip install or virtualenv). This is something that works automatically. The setup.py in the root of the repo takes care of this since it's used also to package the pip module. You can do pip install -e., and it just works (tm). Any basetools commit you check out (once it is in it's own repo, going back a given basetools commit two years ago is unfortunately not as easy) will work with pip install -e with not setup or configuration on your part as a developer. I use this whenever I work on the pytools and it works quite well. It will also show up in the BuildToolsReport and pip freeze as being locally installed and it will tell you the git commit it is currently at.
> 
> I agree- I think we're trending towards very very frequent releases for basetools to provide high granularity. But that is pending community feedback and the decision of the tools maintainer. It is certainly trivial to automate the release pipeline in such a way that a release with every commit that passes CI is easy and mostly automatic for the maintainers.

Thank you. All of the above sounds nice.

I'm carefully optimistic about these changes, and am not opposing them at this point. I hope I can find the time to test these features once they are being proposed in code form.

Regarding the frequent (automatic) release tagging for the separate basetools repo -- that too sounds great; I just think it will need manual work on the edk2 side, i.e. to keep "pip-requirements.txt"
bumped. While that work sounds quite "menial", I much hope that it can be integrated into the basetools development requirements -- "whenever you get something into basetools, and the system tags a new release automatically, be sure to post a patch to edk2 for advancing it to the new basetools release".

Thanks!
Laszlo