From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-il1-f193.google.com (mail-il1-f193.google.com [209.85.166.193]) by mx.groups.io with SMTP id smtpd.web12.15918.1599073601333529195 for ; Wed, 02 Sep 2020 12:06:41 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20161025 header.b=d1SKHBdF; spf=pass (domain: gmail.com, ip: 209.85.166.193, mailfrom: matthewfcarlson@gmail.com) Received: by mail-il1-f193.google.com with SMTP id l4so233935ilq.2; Wed, 02 Sep 2020 12:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fhgRxKslK6w2S3rJTEIE7OtM/wKXXhvLlEYySilYShY=; b=d1SKHBdFxkpyD83NxDnfKp1PBeUoirpqLxMY1LLeppxQOrRe8YvlFP2pQqy4U1LRWf ngQmgihNFB9SqlhxhbjlTcWDQnys9O/aM/ilzAUnOrlyleg5DNNqEyhB0BQ0UL+jFjc8 DOtx+kijRJFHhK5Nol3kpoOnTUTBrDI6HpQ2zjvskNf6Nxq7WZ4PABRqo6SvRp3nkK42 cpwwFnN67mUFqMUOS3md/8pncUrJfP0NPrfUgOby/TAI7+G2VzLIeRfemVnVa9EHzhNv f4Xe/pbQMbVLVfiyE3ycXxQt60exKcfCSVatxagnG/m1mFXc6+L2PvEHLIU9iHO2gSfD Dmxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fhgRxKslK6w2S3rJTEIE7OtM/wKXXhvLlEYySilYShY=; b=EJxj15RbhO9L8vPQGFqUvdmz1YRh18k6+y7Paz7RGaTSZXGxdx8JKqVR0mDFkks7BR MVr0ZqEVp7tYBfG0WhfvD/2iFOmHXsEbL2LEJ6gZCOygiY0nniso5hYZSPZ2DhT0kOSS Z1UF8oSgaa5PG0KM8Wy5bokuiZA2D1Q+TNfHtLTQoieJu5egDVjCTtBFp+JiPPUimSJf QkRTntmRcKmes+HlFLRO3cnhAsh/bwz3qjkfu9wnPB25BJSTjCP2l4q4SXWilD0HYpC4 C4P7qGLexxaYDlGeLWJ/+cKxDHvy/pYdIlVDx5NWWXd7kzwHcw2hEiBZvIhZzTltvqn5 YWnQ== X-Gm-Message-State: AOAM532gr8fZrjbiLnZcqDr9TfGYkFmceapPs0J1kxWhag1aNs7GqDXA KpOrWGJ5t6S3wuKicyD4JVifcvG7FeZTSabOLMs= X-Google-Smtp-Source: ABdhPJwdvr2dIDyDHr/gtvAqDl4F2boL8pZh1OKxtjqIQVlO88vYOzsVVaF1HIOv0U0oMymMorVuGYV9z9bgGHbkXZw= X-Received: by 2002:a92:6f0a:: with SMTP id k10mr4564004ilc.5.1599073600543; Wed, 02 Sep 2020 12:06:40 -0700 (PDT) MIME-Version: 1.0 References: <4C114D94-5320-4825-9400-1044FC5FA9AB@apple.com> <43a4bb4e-8ff1-5b9b-cfc3-22a16dd440db@redhat.com> In-Reply-To: <43a4bb4e-8ff1-5b9b-cfc3-22a16dd440db@redhat.com> From: "Matthew Carlson" Date: Wed, 2 Sep 2020 12:06:29 -0700 Message-ID: Subject: Re: [edk2-devel] Basetools as a pip module To: Laszlo Ersek Cc: devel@edk2.groups.io, afish@apple.com, rfc@edk2.groups.io Content-Type: multipart/alternative; boundary="000000000000ee1ff205ae5955ad" --000000000000ee1ff205ae5955ad Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Andrew: I think leveraging the existing edksetup is a great idea. Using the existing EDK_TOOL_PATH variable could work but it seems clunky. Since the pip module wouldn't be a path, it seems strange to put a boolean value in a variable meant to hold a path. I definitely think that the scripts could print whether they're using the pip modules or the in-source tools. Since Lazlo suggested that pip will be the default, we could have the in-source modules notify of the fact that you're using the in-source modules. An additional feature for the pip module could be printing the version that they are (since you can use the pip infrastructure to query the version of a given module within a python script). Another option would be simply trying the pip module first and then falling back to the in-source module. There would be a slight speed penalty (likely around 10ms) but since this would only apply to trim and build, it should have relatively low impact. Lazlo: Thank you for the excellent summary of the different pieces of the discussion along with the links. To answer your first point, I think what a developer does with their pip module is largely up to them. They could do a virtual environment, they could just do what the requirements state, or pip install from a checked out basetools.I do think there are some variables that the virtual environment sets up that would be a good signal whether you're in a virtual environment or not. I agree with your approach of basetools development going into the out of edk2 repo and the importance of making sure package maintainers test and validate their areas with the new setup. I would personally try to get this early into the development cycle, (just after this next stable tag this week) to give the community and developers the most amount of time to get used to things. A trial period of one release makes sense. I also agree that the gateway is important in maintaining cohesion between the new and the old. Hopefully that's nearing completion. Hopefully other stewards will weigh in but otherwise we'll move ahead with a proposed solution in patches next week. -Matthew Carlson On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek wrote: > On 09/02/20 02:49, Andrew Fish via groups.io wrote: > > > > > >> On Sep 1, 2020, at 4:35 PM, Matthew Carlson > wrote: > >> > >> Hello all, > >> > >> A recent topic on the RFC mailing list went out and the work on moving > Basetools/Sources/Python to a separate repo has started. See the RFC > conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270 < > https://edk2.groups.io/g/rfc/topic/74009714#270> > >> > >> The repo in question is here: > https://github.com/tianocore/edk2-basetools < > https://github.com/tianocore/edk2-basetools> > >> > >> The current plan is shortly after the stable tag is created, a series > of patches will come into edk2 that redirects the build system into the n= ew > python module as well as adds additional documentation. You can see a > sample of this work here: https://github.com/matthewfcarlson/edk2 < > https://github.com/matthewfcarlson/edk2> as this has a branch that has > the work required to use the basetools pip module. The patches won't dele= te > the Basetools/Sources/Python folder but will allow users to select betwee= n > them. After a certain grace period, the python folder will be deleted and > the pip module will be the de facto way of using basetools. > >> > >> Three questions need to be answered: > >> > >> 1. After the patches that enable the pip module land, how long should > the grace period be? > >> 2. During the grace period, should basetools commits land in both > places or just in the edk2-basetools directory? > >> 3. How should the user be able to select which basetools to use (the > one in EDK2 or the pip module)? Currently the approach being considered i= s > a simple environmental variable? One of the key considerations is > transparency since it won't be apparent what is being used for a particul= ar > build without some sort of mechanism to notify the developer. With two > seperate versions of Basetools, it becomes very easy for the version of > basetools you're using to not be the one you expect. > >> > > > > Matthew, > > > > I=E2=80=99ll throw out some current developer centric ideas. > > > > 1) If you `source edksetup.sh` (edksetup.bat) you get the current > behavior, and you add an argument you get the pip flavor? So maybe > `edksetup.bat pip-basetools`? > > 2) We have similar issues to this with env variables and the build > command dumps them out when it runs. Can we use the current EDK_TOOL_PATH= ? > Or maybe add an extra print to show that the pip module is being used? > > I've skimmed: > > - the earlier discussion linked above (rooted at < > https://edk2.groups.io/g/rfc/message/270>), > > - the even earlier comments in the "Discussion: Basetools a separate repo= " > thread on edk2-devel: > > https://edk2.groups.io/g/devel/message/57482 > > http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSM= SX104.ccr.corp.intel.com > > If I still understand / remember correctly, the way at least *I* would us= e > the new feature is the following: > > - set up a new virtual python environment, > > - either install the new pip module "permanently" in the virtual > environment, or else install it in "editable mode" from a git checkout (b= ut > *still* in the virtual environment) > > - build edk2 with the virtual environment active > > That is, for me anyway, the key distinguishing factor would be that I'd b= e > in a python virtual environment where this particular python module exist= ed > / were installed. > > Does this answer question (3)? Because, in my case anyway, I wouldn't hav= e > to be *notified* about using the separate basetools repo vs. using the on= e > resident in edk2 -- instead, I'd have to *activate* the separate basetool= s > repo myself, as first step. So if that activation brings some queriable > feature into the environment (sets a new environment variable or makes a > new python package appear in the environment), then I think it's good > enough -- the usual tools that I run then can query these artifacts. > > In short (I guess): commands should use the in-tree variant, unless I > activate the virtual environment that has the basetools PIP module > installed. > > I think it would be fine to require that, *if* someone intends to activat= e > such a python virtual environment, *then* they do so *before* running > "edksetup.sh". So "edksetup.sh" could check for the python env having the > external basetools repo / module active. Hopefully that would be "early > enough". > > > Regarding the grace period -- questions (1) and (2): > > - The patches introducing the new feature to edk2 should be posted to the > list. These patches should also add a warning to "edksetup.sh" that urges > the developer to use the out-of-tree basetools repo / PIP module, in case > "edksetup.sh" determines the current choice is the in-tree variant (that > is, the virtual env is inactive, or does not contain the new PIP module) > > - While the patches are pending approval, BaseTools development is put on > hold (no fixes, no features). > > - For every package (subsystem) listed in Maintainers.txt, in *both* edk2 > *and* edk2-platfomrs, at least one "M" person is required to report back > with a Tested-by, meaning that they built said package successfully with > the new PIP module. > > - When this feedback is complete, the patch is merged, and the new PIP > module becomes the default build system (see the edksetup.sh warning > described above). > > - Optimally, the above (=3D comprehensive testing feedback, and merging) > would occur *early* in the development cycle (just after the last stable > tag). > > - Going forward, bug reports and feature requests are only addressed in > the new (out-of-tree) module. If someone reports that they have to switch > back, *temporarily*, for whatever reason, to the in-tree variant, and the > in-tree variant no longer builds edk2 for them, then such issues can be > resolved on a case-by-case basis, *after* the issue is reported. Point > being, we make the out-of-tree system the new default because of the > comprehensive and strict initial testing requirements (see above); after > which the old system is preserved for a while only as a fallback. If the > fallback proves lacking later on (but still during the grace period), the= n > the community works to resolve the issue in one of two ways: either help > the issue reporter eliminate their need to use the fallback in the first > place, or backport the subject bugfix/feature to the fallback. > > - After the *next* stable release (which still contains both the fallback > and the support for the out-of-tree PIP module), the fallback is removed. > > Ultimately this would make the grace period almost one full development > cycle, in which cycle the new system should be tested comprehensively, an= d > become the default, near the beginning of the period. > > This is just my proposal. Some of the other stewards are temporarily away= ; > I'd suggest waiting for their feedback too. > > To finish up, I would like to highlight something from the earlier RFC: > > """ > Contribution/Dev Process: > Since this is a separate repo, it will follow a slightly different > contribution and code review process. > 1. Github PR process will be used for contributions and code review > feedback > a. The yet to be released =E2=80=9CTianocore PR archiver=E2=80=9D wi= ll be used to > send to a dedicated list for basetools patch review archive > 2. PRs will only be committed if they keep linear history (no merge > commits) > 3. The PR review must be approved by at least 2 members of the > basetools team (not including the author) > 4. The PR must pass all automated checks > a. Formatting/style > b. Unit tests > c. Code coverage (can=E2=80=99t commit change that would decrease ov= erall %) > d. DCO enforcement - https://probot.github.io/apps/dco/ > e. See other python requirements from the Python coding standard > 5. Github Issues will be used for non-security sensitive > bugs/issues/feature requests > """ > > Point (1a) is a pre-requisite for merging the edk2 patches! > > We cannot make the new system the default unless its development process > is integrated with the github-to-email gateway (webhook). > > Thanks! > Laszlo > > --000000000000ee1ff205ae5955ad Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Andrew:

<= div dir=3D"ltr">I think leveraging the existing edksetup is a great idea. U= sing the existing EDK_TOOL_PATH variable could work but it seems clunky. Si= nce the pip module wouldn't be a path, it seems strange to put a boolea= n value in a variable meant to hold a path. I definitely think that the scr= ipts could print whether they're using the pip modules or the in-source= tools. Since Lazlo suggested that pip will be the default, we could have t= he in-source modules notify of the fact that you're using the in-source= modules. An additional feature for the pip module could be printing the ve= rsion that they are (since you can use the pip infrastructure to query the = version of a given module within a python script). Another option would be = simply trying the pip module first and then falling back to the in-source m= odule. There would be a slight speed penalty (likely around 10ms) but since= this would only apply to trim and build, it should have relatively low imp= act.

Lazlo:
Thank you for th= e excellent summary of the different pieces of the discussion along with th= e links. To answer your first point, I think what a developer does with the= ir pip module is largely up to them. They could do a virtual environment, t= hey could just do what the requirements state, or pip install from a checke= d out basetools.I do think there are some variables that the virtual enviro= nment sets up that would be a good signal whether you're in a virtual e= nvironment or not. I agree with your approach of basetools development goin= g into the out of edk2 repo and the importance of making sure package maint= ainers=C2=A0test and validate their areas with the new setup. I would perso= nally try to get this early into the development cycle, (just after this ne= xt stable tag this week) to give the community and developers the most amou= nt of time to get used to things. A trial period of one release makes sense= .

I also agree that the gateway is important in ma= intaining cohesion between the new and the old. Hopefully that's nearin= g completion.=C2=A0

Hopefully other stewards will = weigh in but otherwise we'll move ahead with a proposed solution in pat= ches next week.

-Matthew Ca= rlson


On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek <= lersek@redhat.com> wrote:
On = 09/02/20 02:49, Andrew Fish via groups.io wrote:
>
>
>> On Sep 1, 2020, at 4:35 PM, Matthew Carlson <matthewfcarlson@gmail.com&= gt; wrote:
>>
>> Hello all,
>>
>> A recent topic on the RFC mailing list went out and the work on mo= ving Basetools/Sources/Python to a separate repo has started. See the RFC c= onversation here: https://edk2.groups.io/g/rfc/topic/= 74009714#270 <https://edk2.groups.io/g/rfc/top= ic/74009714#270>
>>
>> The repo in question is here: https://github.com= /tianocore/edk2-basetools <https://github.com/tian= ocore/edk2-basetools>
>>
>> The current plan is shortly after the stable tag is created, a ser= ies of patches will come into edk2 that redirects the build system into the= new python module as well as adds additional documentation. You can see a = sample of this work here: https://github.com/matthewfcarlson/= edk2 <https://github.com/matthewfcarlson/edk2> = as this has a branch that has the work required to use the basetools pip mo= dule. The patches won't delete the Basetools/Sources/Python folder but = will allow users to select between them. After a certain grace period, the = python folder will be deleted and the pip module will be the de facto way o= f using basetools.
>>
>> Three questions need to be answered:
>>
>> 1. After the patches that enable the pip module land, how long sho= uld the grace period be?
>> 2. During the grace period, should basetools commits land in both = places or just in the edk2-basetools directory?
>> 3. How should the user be able to select which basetools to use (t= he one in EDK2 or the pip module)? Currently the approach being considered = is a simple environmental variable? One of the key considerations is transp= arency since it won't be apparent what is being used for a particular b= uild without some sort of mechanism to notify the developer. With two seper= ate versions of Basetools, it becomes very easy for the version of basetool= s you're using to not be the one you expect.
>>
>
> Matthew,
>
> I=E2=80=99ll throw out some current developer centric ideas.
>
> 1) If you `source edksetup.sh` (edksetup.bat) you get the current beha= vior, and you add an argument you get the pip flavor? So maybe `edksetup.ba= t pip-basetools`?
> 2) We have similar issues to this with env variables and the build com= mand dumps them out when it runs. Can we use the current EDK_TOOL_PATH? Or = maybe add an extra print to show that the pip module is being used?

I've skimmed:

- the earlier discussion linked above (rooted at <https:/= /edk2.groups.io/g/rfc/message/270>),

- the even earlier comments in the "Discussion: Basetools a separate r= epo" thread on edk2-devel:

=C2=A0 https://edk2.groups.io/g/devel/message/57482<= br> =C2=A0 http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SH= SMSX104.ccr.corp.intel.com

If I still understand / remember correctly, the way at least *I* would use = the new feature is the following:

- set up a new virtual python environment,

- either install the new pip module "permanently" in the virtual = environment, or else install it in "editable mode" from a git che= ckout (but *still* in the virtual environment)

- build edk2 with the virtual environment active

That is, for me anyway, the key distinguishing factor would be that I'd= be in a python virtual environment where this particular python module exi= sted / were installed.

Does this answer question (3)? Because, in my case anyway, I wouldn't h= ave to be *notified* about using the separate basetools repo vs. using the = one resident in edk2 -- instead, I'd have to *activate* the separate ba= setools repo myself, as first step. So if that activation brings some queri= able feature into the environment (sets a new environment variable or makes= a new python package appear in the environment), then I think it's goo= d enough -- the usual tools that I run then can query these artifacts.

In short (I guess): commands should use the in-tree variant, unless I activ= ate the virtual environment that has the basetools PIP module installed.
I think it would be fine to require that, *if* someone intends to activate = such a python virtual environment, *then* they do so *before* running "= ;edksetup.sh". So "edksetup.sh" could check for the python e= nv having the external basetools repo / module active. Hopefully that would= be "early enough".


Regarding the grace period -- questions (1) and (2):

- The patches introducing the new feature to edk2 should be posted to the l= ist. These patches should also add a warning to "edksetup.sh" tha= t urges the developer to use the out-of-tree basetools repo / PIP module, i= n case "edksetup.sh" determines the current choice is the in-tree= variant (that is, the virtual env is inactive, or does not contain the new= PIP module)

- While the patches are pending approval, BaseTools development is put on h= old (no fixes, no features).

- For every package (subsystem) listed in Maintainers.txt, in *both* edk2 *= and* edk2-platfomrs, at least one "M" person is required to repor= t back with a Tested-by, meaning that they built said package successfully = with the new PIP module.

- When this feedback is complete, the patch is merged, and the new PIP modu= le becomes the default build system (see the edksetup.sh warning described = above).

- Optimally, the above (=3D comprehensive testing feedback, and merging) wo= uld occur *early* in the development cycle (just after the last stable tag)= .

- Going forward, bug reports and feature requests are only addressed in the= new (out-of-tree) module. If someone reports that they have to switch back= , *temporarily*, for whatever reason, to the in-tree variant, and the in-tr= ee variant no longer builds edk2 for them, then such issues can be resolved= on a case-by-case basis, *after* the issue is reported. Point being, we ma= ke the out-of-tree system the new default because of the comprehensive and = strict initial testing requirements (see above); after which the old system= is preserved for a while only as a fallback. If the fallback proves lackin= g later on (but still during the grace period), then the community works to= resolve the issue in one of two ways: either help the issue reporter elimi= nate their need to use the fallback in the first place, or backport the sub= ject bugfix/feature to the fallback.

- After the *next* stable release (which still contains both the fallback a= nd the support for the out-of-tree PIP module), the fallback is removed.
Ultimately this would make the grace period almost one full development cyc= le, in which cycle the new system should be tested comprehensively, and bec= ome the default, near the beginning of the period.

This is just my proposal. Some of the other stewards are temporarily away; = I'd suggest waiting for their feedback too.

To finish up, I would like to highlight something from the earlier RFC:

"""
Contribution/Dev Process:
Since this is a separate repo, it will follow a slightly different contribu= tion and code review process.
1.=C2=A0 =C2=A0 =C2=A0 Github PR process will be used for contributions and= code review feedback
a.=C2=A0 =C2=A0 =C2=A0 The yet to be released =E2=80=9CTianocore PR archive= r=E2=80=9D will be used to send to a dedicated list for basetools patch rev= iew archive
2.=C2=A0 =C2=A0 =C2=A0 PRs will only be committed if they keep linear histo= ry (no merge commits)
3.=C2=A0 =C2=A0 =C2=A0 The PR review must be approved by at least 2 members= of the basetools team (not including the author)
4.=C2=A0 =C2=A0 =C2=A0 The PR must pass all automated checks
a.=C2=A0 =C2=A0 =C2=A0 Formatting/style
b.=C2=A0 =C2=A0 =C2=A0 Unit tests
c.=C2=A0 =C2=A0 =C2=A0 Code coverage (can=E2=80=99t commit change that woul= d decrease overall %)
d.=C2=A0 =C2=A0 =C2=A0 DCO enforcement - https://probot.github.io/= apps/dco/
e.=C2=A0 =C2=A0 =C2=A0 See other python requirements from the Python coding= standard
5.=C2=A0 =C2=A0 =C2=A0 Github Issues will be used for non-security sensitiv= e bugs/issues/feature requests
"""

Point (1a) is a pre-requisite for merging the edk2 patches!

We cannot make the new system the default unless its development process is= integrated with the github-to-email gateway (webhook).

Thanks!
Laszlo

--000000000000ee1ff205ae5955ad--