Hey Andrew,

Your comments were h= elpful and valid &#= 128522;

Part of the appeal of making it into a pip module is that = it doesn=E2=80=99t have a specific file system path. Python itself take car= es of it and it can be addressed by the module name rather than adding it t= o the path. So you hit the nail on the head, EDK_TOOL_PATH doesn=E2=80=99t = really make sense in an all pip module world. It will probably be deprecate= d. You can see here: https:= //github.com/matthewfcarlson/edk2/blob/feature/pip-basetools/BaseTools/BinW= rappers/WindowsLike/build.bat that we call directly into the module wit= hout having to setup the python path. The file here doesn=E2=80=99t use the= BaseTools inside EDK2, that still need to built.

You have some great que= stions and I=E2=80=99ll do my best to answer with what has been discussed o= n the original RFC.

if I run build from th= e command line how did my path get set?

The pip system adds it to the python patch= for you where ever you have it installed (locally, user, system, etc). In = addition, pip module offers console scripts, which are small hooks that bas= ically do what the build.bat script does: start the python interpreter and = call the build module. So in the current pip module there are a few new com= mands, one of them being edk2_build which is available anywhere (within the= scope of your pip environment). Alternatively, you could use the BinWrappe= rs and let the edksetup do the heavy lifting for you. It should all just wo= rk out of the box (fingers crossed).

How do I confi= gure different versions of BaseTools?

So Lazlo pointed out that you could use virtu= al environments, one for each edk2 clone on your system. It has the added b= enefit of making your system state much simpler and more reproducible since= the pip module are for that repo rather than your whole system. Another op= tion is just to install a local copy of the pip module using pip install -e= . which will stay in sync with whatever you have checked out. This does me= an an extra step of making sure you have the right commit checked out and p= robably isn=E2=80=99t advisable since it=E2=80=99s easy to forget. But sinc= e I only am ever have one clone open at a time, it=E2=80=99s not bad for me= personally.

I guess my repo could have a yml file= that points to the version of the tools that I use, but that seems like a = per user, not per repo config

There=E2=80=99s a pip requirement file that keeps tra= ck of the exact version of the pip module you need (EDK2 already has one fo= r CI). For a given commit in EDK2, you=E2=80=99d have the exact version of = basetools that you know works.

It seems to me we co= uld have developers that want to contribute to edk2 and work on their own c= ode base and that could rehire (require?) two different Basetool versions i= nstalled on the system, and I think we need a story for that?
1. Yeah- I agree that st= ory isn=E2=80=99t as fleshed out. In some way it=E2=80=99s better than EDK2= =E2=80=99s current story as the basetools can be consumed separately from E= DK2 itself. You could setup your own codebase to leverage the basetools as = is or you could point it at your own fork of the BaseTools. But I can see f= olks who have forked EDK2 and have tweaks in BaseTools having to push those= changes back into BaseTools. I believe that moving it into it=E2=80=99s ow= n project will make it easier for fixes and other projects to be built off = the BaseTools as it makes it easy to include without needing the rest of ED= K2 with it. For example, easy to use and easy to install capsule generation= tools for FMP devices.
For the macOS Xcode compile= r you can install as many versions as you want and there is a command line = tool to let you set the current version of Xcode, and to show you the curre= ntly select versions. Basically the tools in the magic location in your pat= h are just redirectors to the currently selected tools.
1. I think that=E2=80=99= s possible inside pip to have sort of the functionality you describe but ge= nerally it would be through virtual environments and perhaps installing it = in different places (system vs user for example). With pip, you can also do= a pip freeze to see the state of your pip system in the given environment.= It=E2=80=99s likely not as robust/fully-featured as the XCode system, thou= gh it=E2=80=99s been a few years since I was a serious XCode user and my re= collection of exactly the experience is a little fuzzy.
Thanks= again for your feedback and hopefully that addresses your questions.<= /o:p>

From: Andrew Fish
Sent: Wednesday, September 2, 2020 7:14 P= M
To: Matthew Carlso= n
Cc: Laszlo Ersek; = edk2-devel-groups-io; rfc@edk2.groups.io
Subject: Re= : [edk2-devel] Basetools as a pip module

Matthew,

I did not= meant to imply we should optimize for the current installed base, I just t= hink it is useful to think about them. I think Lazlo is pointing out what i= s best for the project and that should have more weight than the installed = base, but it is always good to think how things impact different groups.&nb= sp;

In terms of the UI I was thinking of pointing t= o the pip install location vs. your git repo not so much a boolean. I actua= lly don=E2=80=99t quite understand what EDK_TOOL_PATH actually means if Bas= eTools is a pip module. If that is an obsolete concept then we should remov= e it, and replace with some kind of statement that the pip installed BaseTo= ols are being used. Another question if I run build from the command line h= ow did my path get set? For example my user account has 5 different version= s of edk2 in it how do I configure different versions of BaseTools? How do = I pip multiple versions on to my system? I guess my repo could have a yml f= ile that points to the version of the tools that I use, but that seems like= a per user, not per repo config? It seems to me we could have developers t= hat want to contribute to edk2 and work on their own code base and that cou= ld rehire two different Basetool versions installed on the system, and I th= ink we need a story for that.

For the macOS X= code compiler you can install as many versions as you want and there is a c= ommand line tool to let you set the current version of Xcode, and to show y= ou the currently select versions. Basically the tools in the magic location= in your path are just redirectors to the currently selected tools.

Thanks,

Andrew Fish

On Sep 2, = 2020, at 12:06 PM, Matthew Carlson <matthewfcarlson@gmail.com> wrote:

An= drew:

I think leveraging the existing edksetup is a= great idea. Using the existing EDK_TOOL_PATH variable could work but it se= ems clunky. Since the pip module wouldn't be a path, it seems strange to pu= t a boolean value in a variable meant to hold a path. I definitely think th= at the scripts could print whether they're using the pip modules or the in-= source tools. Since Lazlo suggested that pip will be the default, we could = have the in-source modules notify of the fact that you're using the in-sour= ce modules. An additional feature for the pip module could be printing the = version that they are (since you can use the pip infrastructure to query th= e version of a given module within a python script). Another option would b= e simply trying the pip module first and then falling back to the in-source= module. There would be a slight speed penalty (likely around 10ms) but sin= ce this would only apply to trim and build, it should have relatively low i= mpact.

<= /div>
Lazlo:
Thank you for the excellent summary of the different pieces of the= discussion along with the links. To answer your first point, I think what = a developer does with their pip module is largely up to them. They could do= a virtual environment, they could just do what the requirements state, or = pip install from a checked out basetools.I do think there are some variable= s that the virtual environment sets up that would be a good signal whether = you're in a virtual environment or not. I agree with your approach of baset= ools development going into the out of edk2 repo and the importance of maki= ng sure package maintainers test and validate their areas with the new= setup. I would personally try to get this early into the development cycle= , (just after this next stable tag this week) to give the community and dev= elopers the most amount of time to get used to things. A trial period of on= e release makes sense.
&= nbsp;
I also agree that the gatewa= y is important in maintaining cohesion between the new and the old. Hopeful= ly that's nearing completion.

Hopefully other= stewards will weigh in but otherwise we'll move ahead with a proposed solu= tion in patches next week.

-Matthew = Carlson

On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek <lersek@redhat.com> wrote:
On 09/= 02/20 02:49, Andrew Fish via groups.io wrote:
>
>
>> On Sep 1, 2020, at 4:3= 5 PM, Matthew Carlson <matthewfcarlson@gmail.com> wrote:
>>
>&= gt; Hello all,
>>
>> A recent topic on the RFC mailing li= st went out and the work on moving Basetools/Sources/Python to a separate r= epo has started. See the RFC conversation here: https://edk2.groups.io/g= /rfc/topic/74009714#270 <https://edk2.groups.io/g/rfc/topic/74009= 714#270>
>>
>> The repo in question is here: https:= //github.com/tianocore/edk2-basetools <https://github.com/tianocore/e= dk2-basetools>
>>
>> The current plan is shortly a= fter the stable tag is created, a series of patches will come into edk2 tha= t redirects the build system into the new python module as well as adds add= itional documentation. You can see a sample of this work here: https://github.co= m/matthewfcarlson/edk2 <https://github.com/matthewfcarlson/edk2> a= s this has a branch that has the work required to use the basetools pip mod= ule. The patches won't delete the Basetools/Sources/Python folder but will = allow users to select between them. After a certain grace period, the pytho= n folder will be deleted and the pip module will be the de facto way of usi= ng basetools.
>>
>> Three questions need to be answered:<= br>>>
>> 1. After the patches that enable the pip module lan= d, how long should the grace period be?
>> 2. During the grace per= iod, should basetools commits land in both places or just in the edk2-baset= ools directory?
>> 3. How should the user be able to select which = basetools to use (the one in EDK2 or the pip module)? Currently the approac= h being considered is a simple environmental variable? One of the key consi= derations is transparency since it won't be apparent what is being used for= a particular build without some sort of mechanism to notify the developer.= With two seperate versions of Basetools, it becomes very easy for the vers= ion of basetools you're using to not be the one you expect.
>>
= >
> Matthew,
>
> I=E2=80=99ll throw out some current= developer centric ideas.
>
> 1) If you `source edksetup.sh` = (edksetup.bat) you get the current behavior, and you add an argument you ge= t the pip flavor? So maybe `edksetup.bat pip-basetools`?
> 2) We have= similar issues to this with env variables and the build command dumps them= out when it runs. Can we use the current EDK_TOOL_PATH? Or maybe add an ex= tra print to show that the pip module is being used?

I've skimmed:
- the earlier discussion linked above (rooted at <https://edk2.groups.= io/g/rfc/message/270>),

- the even earlier comments in the &q= uot;Discussion: Basetools a separate repo" thread on edk2-devel:
https://edk2.groups.io/g/devel/message/57482
http://mid.mail-archive.com/7= 34D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com
If I still understand / remember correctly, the way at least *I* would= use the new feature is the following:

- set up a new virtual python= environment,

- either install the new pip module "permanently&= quot; in the virtual environment, or else install it in "editable mode= " from a git checkout (but *still* in the virtual environment)

= - build edk2 with the virtual environment active

That is, for me any= way, the key distinguishing factor would be that I'd be in a python virtual= environment where this particular python module existed / were installed.<= br>
Does this answer question (3)? Because, in my case anyway, I wouldn'= t have to be *notified* about using the separate basetools repo vs. using t= he one resident in edk2 -- instead, I'd have to *activate* the separate bas= etools repo myself, as first step. So if that activation brings some queria= ble feature into the environment (sets a new environment variable or makes = a new python package appear in the environment), then I think it's good eno= ugh -- the usual tools that I run then can query these artifacts.

In= short (I guess): commands should use the in-tree variant, unless I activat= e the virtual environment that has the basetools PIP module installed.
<= br>I think it would be fine to require that, *if* someone intends to activa= te such a python virtual environment, *then* they do so *before* running &q= uot;edksetup.sh". So "edksetup.sh" could check for the pytho= n env having the external basetools repo / module active. Hopefully that wo= uld be "early enough".

Regarding the grace period -- q= uestions (1) and (2):

- The patches introducing the new feature to e= dk2 should be posted to the list. These patches should also add a warning t= o "edksetup.sh" that urges the developer to use the out-of-tree b= asetools repo / PIP module, in case "edksetup.sh" determines the = current choice is the in-tree variant (that is, the virtual env is inactive= , or does not contain the new PIP module)

- While the patches are pe= nding approval, BaseTools development is put on hold (no fixes, no features= ).

- For every package (subsystem) listed in Maintainers.txt, in *bo= th* edk2 *and* edk2-platfomrs, at least one "M" person is require= d to report back with a Tested-by, meaning that they built said package suc= cessfully with the new PIP module.

- When this feedback is complete,= the patch is merged, and the new PIP module becomes the default build syst= em (see the edksetup.sh warning described above).

- Optimally, the a= bove (=3D comprehensive testing feedback, and merging) would occur *early* = in the development cycle (just after the last stable tag).

- Going f= orward, bug reports and feature requests are only addressed in the new (out= -of-tree) module. If someone reports that they have to switch back, *tempor= arily*, for whatever reason, to the in-tree variant, and the in-tree varian= t no longer builds edk2 for them, then such issues can be resolved on a cas= e-by-case basis, *after* the issue is reported. Point being, we make the ou= t-of-tree system the new default because of the comprehensive and strict in= itial testing requirements (see above); after which the old system is prese= rved for a while only as a fallback. If the fallback proves lacking later o= n (but still during the grace period), then the community works to resolve = the issue in one of two ways: either help the issue reporter eliminate thei= r need to use the fallback in the first place, or backport the subject bugf= ix/feature to the fallback.

- After the *next* stable release (which= still contains both the fallback and the support for the out-of-tree PIP m= odule), the fallback is removed.

Ultimately this would make the grac= e period almost one full development cycle, in which cycle the new system s= hould be tested comprehensively, and become the default, near the beginning= of the period.

This is just my proposal. Some of the other stewards= are temporarily away; I'd suggest waiting for their feedback too.

T= o finish up, I would like to highlight something from the earlier RFC:
<= br>"""
Contribution/Dev Process:
Since this is a separ= ate repo, it will follow a slightly different contribution and code review = process.
1. Github PR process will be used for contr= ibutions and code review feedback
a. The yet to be r= eleased =E2=80=9CTianocore PR archiver=E2=80=9D will be used to send to a d= edicated list for basetools patch review archive
2. = PRs will only be committed if they keep linear history (no merge commits)3. The PR review must be approved by at least 2 membe= rs of the basetools team (not including the author)
4. &nbs= p; The PR must pass all automated checks
a. Formatti= ng/style
b. Unit tests
c. Cod= e coverage (can=E2=80=99t commit change that would decrease overall %)
d= . DCO enforcement - https://probot.github.io/apps/dco/
e.&= nbsp; See other python requirements from the Python coding st= andard
5. Github Issues will be used for non-securit= y sensitive bugs/issues/feature requests
"""

Point= (1a) is a pre-requisite for merging the edk2 patches!

We cannot mak= e the new system the default unless its development process is integrated w= ith the github-to-email gateway (webhook).

Thanks!
Laszlo

=