From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mx.groups.io with SMTP id smtpd.web10.2380.1587087632658668191 for ; Thu, 16 Apr 2020 18:40:32 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.115, mailfrom: ray.ni@intel.com) IronPort-SDR: y+QN0xAcp96NCGuKqeNMR+nWLMN7VJyzT8xAPBs1HuN5th8W+YpWDnEzJoBF+Ul/8ikosWjSOF HfaVNAeWuf4w== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Apr 2020 18:40:27 -0700 IronPort-SDR: WmXU4m8fV/kIXRd37AxlXNlRkkWrXQ3/6BQit3XtaF56Ceerr2GByi1EXum6eLqKXQHl8mnFdX wOvK2z6cLwHg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,393,1580803200"; d="scan'208,217";a="254055985" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by orsmga003.jf.intel.com with ESMTP; 16 Apr 2020 18:40:22 -0700 Received: from fmsmsx111.amr.corp.intel.com (10.18.116.5) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 16 Apr 2020 18:40:22 -0700 Received: from shsmsx105.ccr.corp.intel.com (10.239.4.158) by fmsmsx111.amr.corp.intel.com (10.18.116.5) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 16 Apr 2020 18:40:21 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.225]) by SHSMSX105.ccr.corp.intel.com ([169.254.11.213]) with mapi id 14.03.0439.000; Fri, 17 Apr 2020 09:40:19 +0800 From: "Ni, Ray" To: "devel@edk2.groups.io" Subject: FW: Discussion: Basetools a separate repo Thread-Topic: Discussion: Basetools a separate repo Thread-Index: AQHWEpPQxknXjIyLXUqpjD+e7ZPLDKh5EqrrgAByPUCAAbZMMIAAAarggAFQOsA= Date: Fri, 17 Apr 2020 01:40:18 +0000 Message-ID: <734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com> References: <734D49CCEBEEF84792F5B80ED585239D5C4FEF0B@SHSMSX104.ccr.corp.intel.com> <734D49CCEBEEF84792F5B80ED585239D5C500738@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <734D49CCEBEEF84792F5B80ED585239D5C500738@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US, zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-04-14T19:35:11.2520130Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Privileged x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: ray.ni@intel.com Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_734D49CCEBEEF84792F5B80ED585239D5C5019CESHSMSX104ccrcor_" --_000_734D49CCEBEEF84792F5B80ED585239D5C5019CESHSMSX104ccrcor_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable From: Ni, Ray Sent: Thursday, April 16, 2020 1:41 PM To: 'Sean Brogan' ; Matthew Carlson Cc: Feng, Bob C ; edk2-devel@lists.01.org Subject: Discussion: Basetools a separate repo Sean, Matthew needs to join the edk2.groups.io and sends message from the registe= red mail address in order to send out message to everyone in the forum. I j= ust checked and didn't find any blocking messages from Matthew in the group= s.io website. I copied edk2-devel@lists.01.org in this ma= il so everyone in the forum can get notified before the meeting. Thanks, Ray From: Sean Brogan > Sent: Thursday, April 16, 2020 1:31 PM To: Ni, Ray >; Matthew Carlson > Cc: Feng, Bob C > Subject: RE: Discussion: Basetools a separate repo Ray, Can you check if the message is blocked/moderated? I think it would be goo= d to get the message out to the list prior to the design meeting. Thanks Sean From: Ni, Ray > Sent: Tuesday, April 14, 2020 8:23 PM To: Matthew Carlson >; Se= an Brogan > Cc: Feng, Bob C > Subject: [EXTERNAL] RE: Discussion: Basetools a separate repo Matthew, Meeting agenda was updated. Thanks, Ray From: Matthew Carlson > Sent: Wednesday, April 15, 2020 4:42 AM To: Ni, Ray >; Sean Brogan > Subject: Discussion: Basetools a separate repo Hello Ray, I sent this to the discuss list on tianocore, but I think I'm moderated. ---- I'm looking to discuss the movement of the basetools folder in edk2 to a se= parate repo and treated as a separate python project. I'd like to get on the agenda for the Thursday (April 16th) design meeting = with this topic. I'm planning on talking through the reasoning and current = thinking for 5 minutes and having a discussion. I expect at least 20-30 min= utes of discussion. Here's a basic overview of the what and the why behind this proposal: Why a separate repo? The recent efforts in expanding the role of CI in the platform and core cod= e of EDK2 will pay big dividends in the future, leading to higher quality c= ode and easier integrations for everyone. Having basetools as it's own repo= would simplify adding a similar CI/linting process and unit-tests to the b= asetools python code, leading to higher quality code. A second major benefit is it would allow others that write tools for UEFI a= nd Edk2 to leverage this vast resource of python code using standard python= package inclusion. It would allow those tools to be decoupled from edk2 s= ource and provide a consistent and managed user experience. The python pro= ject would be published as a Pip module for those that want to leverage the= basetools modules the same way they leverage the existing python ecosystem= . Packing basetools as a pip module, would reach the most developers and pr= ovide the most flexibility and versatility. There are numerous way this cou= ld be used; Pip is just one method suggested here. Other ways to leverage = this are described below. Why a pip module? The investment into basetools is sizable and it has some amazing functional= ity that's difficult to reproduce. For example, the DSC, FDF, INF, and DEC = parsers handle an incredible amount of edge cases. If I wanted to write a t= ool that could do a CI validation check or build a UEFI capsule, currently = I would need to clone all of EDK2 to get basetools. If it was in a separate= repo and available as a wheel, as a developer, I could include it via pip = and have that dependency managed with virtual environment or just in the gl= obal cache. In addition, other tools that currently are difficult to build = would become possible with access to the Basetools functionality. However, there have been some concerns expressed about having a global base= tools and the impact this has on developers with multiple workspaces (poten= tially at different versions). There are several tools and strategies to c= onsider for managing this dependency. Some outlined below. How will this change your workflow? If this moved there would have to be a change for all platforms using edk2 = and we have been evaluating options. These could become requirements a dev= eloper must do before building edk2 or with minimal effort could be added t= o the edksetup or build scripts. These can also be more easily isolated if= python virtual environments are used. For those just consuming released versions of basetools python code: Option A: leverage Python package management and install a released version= from Pypi into a per project virtual environment. Option B: leverage pip to install from the git repo at known tag/release (p= ip_requirements) For those wanting to do active development of basetools within an edk2 proj= ect: Option C: Clone the python package source and install the package locally (= pip install -e ./). All changes made in the local package are reflected in= to your python site packages. We have a demo of what this would look like: https://github.com/matthewfcar= lson/edk2-pytool-base/ And the EDK2 that leverages it https://github.com/matthewfcarlson/edk2/tree= /feature/pip-basetools What happens next? Right now, we're gathering feedback and seeing if anyone has an concerns or= platforms that this would not work for. We'd love to hear what you have to= say. Baring any serious concerns, we'd move forward with: 1. Create new GitHub repo on tianocore for the basetools project 2. Develop the testing, PR, and release process 3. Release the initial version to pypi 4. Delete the source folder in edk2 repo and replace with readme and met= hod to get pip version installed 5. Continually improve basetools and add more testing What's the long-term plan? The current tentative long term plan is to merge some or all of basetools i= n with the existing edk2-pytool-library repo. This is still an active conve= rsation, and we'd like to hear your thoughts. Matthew Carlson Core UEFI Microsoft --_000_734D49CCEBEEF84792F5B80ED585239D5C5019CESHSMSX104ccrcor_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

 

From: Ni, Ray
Sent: Thursday, April 16, 2020 1:41 PM
To: 'Sean Brogan' <sean.brogan@microsoft.com>; Matthew Carlson= <macarl@microsoft.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; edk2-devel@lists.01.or= g
Subject: Discussion: Basetools a separate repo

 

Sean,

Matthew needs to join the edk2.groups.io and sends m= essage from the registered mail address in order to send out message to eve= ryone in the forum. I just checked and didn’t find any blocking messa= ges from Matthew in the groups.io website.

 

I copied = edk2-devel@lists.01.org in this mail so everyone in the forum can get n= otified before the meeting.

 

Thanks,

Ray

 

From: Sean Brogan <sean.brogan@microsoft.com>
Sent: Thursday, April 16, 2020 1:31 PM
To: Ni, Ray <ray.ni@intel.com= >; Matthew Carlson <macar= l@microsoft.com>
Cc: Feng, Bob C <bob.c.fe= ng@intel.com>
Subject: RE: Discussion: Basetools a separate repo

 

Ray,

 

Can you check if the message is blocked/moderated?&n= bsp; I think it would be good to get the message out to the list prior to t= he design meeting. 

 

Thanks

Sean

 

 

From: Ni, Ray <ray.ni@intel.com>
Sent: Tuesday, April 14, 2020 8:23 PM
To: Matthew Carlson <maca= rl@microsoft.com>; Sean Brogan <sean.brogan@microsoft.com>
Cc: Feng, Bob C <bob.c.fe= ng@intel.com>
Subject: [EXTERNAL] RE: Discussion: Basetools a separate repo

 

Matthew,

Meeting agenda was updated.

 

Thanks,

Ray

 

From: Matthew Carlson <macarl@microsoft.com>
Sent: Wednesday, April 15, 2020 4:42 AM
To: Ni, Ray <ray.ni@intel.com= >; Sean Brogan <sean= .brogan@microsoft.com>
Subject: Discussion: Basetools a separate repo

 

Hello Ray,

 

I sent this to the discuss list on tianocore, but I = think I’m moderated.

 

----

 

I’m looking to discuss the movement of the bas= etools folder in edk2 to a separate repo and treated as a separate python p= roject.

 

I’d like to get on the agenda for the Thursday= (April 16th) design meeting with this topic. I’m planning= on talking through the reasoning and current thinking for 5 minutes and ha= ving a discussion. I expect at least 20-30 minutes of discussion.

 

Here’s a basic overview of the what and the wh= y behind this proposal:

 

Why a separate repo?

The recent efforts in expanding the role of CI in th= e platform and core code of EDK2 will pay big dividends in the future, lead= ing to higher quality code and easier integrations for everyone. Having bas= etools as it’s own repo would simplify adding a similar CI/linting process and unit-tests to the basetools python= code, leading to higher quality code.

 

A second major benefit is it would allow others that= write tools for UEFI and Edk2 to leverage this vast resource of python cod= e using standard python package inclusion.  It would allow those tools= to be decoupled from edk2 source and provide a consistent and managed user experience.  The python project would b= e published as a Pip module for those that want to leverage the basetools m= odules the same way they leverage the existing python ecosystem. Packing ba= setools as a pip module, would reach the most developers and provide the most flexibility and versatility. Ther= e are numerous way this could be used; Pip is just one method suggested her= e.  Other ways to leverage this are described below.

 

Why a pip module?

The investment into basetools is sizable and it has = some amazing functionality that’s difficult to reproduce. For example= , the DSC, FDF, INF, and DEC parsers handle an incredible amount of edge ca= ses. If I wanted to write a tool that could do a CI validation check or build a UEFI capsule, currently I would need t= o clone all of EDK2 to get basetools. If it was in a separate repo and avai= lable as a wheel, as a developer, I could include it via pip and have that = dependency managed with virtual environment or just in the global cache. In addition, other tools that cur= rently are difficult to build would become possible with access to the Base= tools functionality.

 

However, there have been some concerns expressed abo= ut having a global basetools and the impact this has on developers with mul= tiple workspaces (potentially at different versions).  There are sever= al tools and strategies to consider for managing this dependency.  Some outlined below. 

 

How will this change your workflow?

If this moved there would have to be a change for al= l platforms using edk2 and we have been evaluating options.  These cou= ld become requirements a developer must do before building edk2 or with min= imal effort could be added to the edksetup or build scripts.  These can also be more easily isolated if python v= irtual environments are used.

 

For those just consuming released versions of baseto= ols python code:

Option A: leverage Python= package management and install a released version from Pypi into a per pro= ject virtual environment.

Option B: leverage pip to= install from the git repo at known tag/release (pip_requirements)

 

For those wanting to do active development of baseto= ols within an edk2 project:

Option C: Clone the pytho= n package source and install the package locally (pip install -e ./). = All changes made in the local package are reflected into your python site = packages. 

 

 

We have a demo of what this would look like: https://github.com/matthewfcarlson/edk2-pytool-base/

And the EDK2 that leverages it https://github.com/matthewfcarlson/edk2/tree/feature/pip-basetools=

 

What happens next?

Right now, we’re gathering feedback and seeing= if anyone has an concerns or platforms that this would not work for. We= 217;d love to hear what you have to say. Baring any serious concerns, we= 217;d move forward with:

  1. Create new GitHub repo on tianocore for the basetools project
  2. Develop the testing, PR, and release process
    3. <= li class=3D"MsoListParagraph" style=3D"margin-left:0in;mso-list:l1 level1 l= fo3">Release the initial version to pypi
  3. Delete the so= urce folder in edk2 repo and replace with readme and method to get pip vers= ion installed
  4. Continually improve basetools and add mo= re testing

 

What’s the long-term plan?

The current tentative long term plan is to merge som= e or all of basetools in with the existing edk2-pytool-library repo. This i= s still an active conversation, and we’d like to hear your thoughts.<= o:p>

 

Matthew Carlson

Core UEFI

Microsoft

 

 

--_000_734D49CCEBEEF84792F5B80ED585239D5C5019CESHSMSX104ccrcor_--