3rd Party Python Packages

3rd Party Python Packages

[Carsey, Jaben]

Note: deviating from "RE: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function"

Laszlo,

Since I am working on some research related to this, I have a few follow up questions.

If there is significant performance improvement, how would you feel about requiring a 3rd party package to be installed (via pip I think)?
Would you feel more comfortable if BaseTools were able to run either with it or without (with performance differences)?

Basically there are potential performance improvements, but use of 3rd party python packages is instrumental for many of them.  

-Jaben

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Laszlo Ersek
> Sent: Tuesday, February 12, 2019 6:02 AM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
> 
> On 02/12/19 14:33, Gao, Liming wrote:
> > Laszlo:
> >  To install python3-distutils should resolve this issue. I expect BaseTools
> build functionality doesn't depend on the third party python lib.
> 
> I completely agree with your expectation, regarding *3rd party* python
> packages. We shouldn't expect developers to install packages from
> repositories that fall outside of their normal distro repos.
> 
> However, my understanding was that python3-distutils should be available
> as a normal (not 3rd party) component on Ubuntu 18. I think we can
> expect developers to install additional packages if those packages are
> readily available in their normal (distro-provided) repos.
> 
> > So, I suggest to check whether python3-distutils is the native python
> library. If it is native python library, why Ubuntu18 doesn't include it. I will
> work with Dandan to collect more information.
> 
> Right, that's exactly what I'm asking for. Thank you very much!
> Laszlo
> 
> >> -----Original Message-----
> >> From: Laszlo Ersek [mailto:lersek@redhat.com]
> >> Sent: Tuesday, February 12, 2019 8:24 PM
> >> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan
> <dandan.bi@intel.com>
> >> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
> >> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted
> function
> >>
> >> On 02/04/19 20:12, Laszlo Ersek wrote:
> >>> On 02/03/19 06:55, Feng, Bob C wrote:
> >>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
> >>>> On some Linux environment, there may be no distutils.util
> >>>> library for python3 that will cause build tool crash.
> >>>> This patch implement distutils.util.split_quoted
> >>>> in BaseTools so that the Basetools will be independent with
> >>>> distutils.util library.
> >>>>
> >>>> Feng, Bob C (3):
> >>>>   BaseTools: Implement splitquoted function in Build tool
> >>>>   BaseTools: Implement splitquoted function in UPT
> >>>>   BaseTools: unit test for splitquoted function
> >>>>
> >>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50
> ++++++++++++++++++++++++++++++++++++++++++++++----
> >>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47
> ++++++++++++++++++++++++++++++++++++++++++++---
> >>>>  BaseTools/Tests/TestStringSplit.py                    | 38
> ++++++++++++++++++++++++++++++++++++++
> >>>>  3 files changed, 128 insertions(+), 7 deletions(-)
> >>>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
> >>>>
> >>>
> >>> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
> >>> like the issue can be resolved by a simple package installation on
> >>> Ubuntu 18:
> >>>
> >>> https://superuser.com/questions/1319047/cant-install-virtual-
> interpreter-in-pycharm-in-linux
> >>>
> >>> """
> >>> sudo apt-get install python3-distutils
> >>> """
> >>>
> >>> I'm not a Ubuntu user myself; so all I can do here (without installing a
> >>> Ubuntu18 VM) is check the Ubuntu package directory:
> >>>
> >>> https://packages.ubuntu.com/search?keywords=python3-
> distutils&searchon=names&suite=all&section=all
> >>>
> >>> python3-distutils appears available for both "bionic (18.04LTS)" and
> >>> "cosmic (18.10)".
> >>>
> >>> Dandan, if you install python3-distutils, does that solve the issue for
> you?
> >>
> >> I'd still like to get an answer to my question, before the series is pushed.
> >>
> >> Thanks,
> >> Laszlo
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: 3rd Party Python Packages

[Laszlo Ersek]

On 02/12/19 16:22, Carsey, Jaben wrote:
> Note: deviating from "RE: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function"
> 
> Laszlo,
> 
> Since I am working on some research related to this, I have a few follow up questions.
> 
> If there is significant performance improvement, how would you feel about requiring a 3rd party package to be installed (via pip I think)?

I'd strongly disagree with the proposal, except if the add-on were optional.

If the python add-on in question is well-maintained, its upstream
maintainers should work with popular distro maintainers to get the
project packaged. Then distro users can enable the add-on (and the
dependent BaseTools goodies) without leaving their well known / trusted
repos.

> Would you feel more comfortable if BaseTools were able to run either with it or without (with performance differences)?

I certainly would.

> Basically there are potential performance improvements, but use of 3rd party python packages is instrumental for many of them.

I think that's normal; people write libs and add-ons to improve
functionality and/or performance. What matters is how distro users can
consume these add-ons.

Thanks,
Laszlo

> 
> -Jaben
> 
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
>> Laszlo Ersek
>> Sent: Tuesday, February 12, 2019 6:02 AM
>> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
>> <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
>> Cc: edk2-devel@lists.01.org
>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>>
>> On 02/12/19 14:33, Gao, Liming wrote:
>>> Laszlo:
>>>  To install python3-distutils should resolve this issue. I expect BaseTools
>> build functionality doesn't depend on the third party python lib.
>>
>> I completely agree with your expectation, regarding *3rd party* python
>> packages. We shouldn't expect developers to install packages from
>> repositories that fall outside of their normal distro repos.
>>
>> However, my understanding was that python3-distutils should be available
>> as a normal (not 3rd party) component on Ubuntu 18. I think we can
>> expect developers to install additional packages if those packages are
>> readily available in their normal (distro-provided) repos.
>>
>>> So, I suggest to check whether python3-distutils is the native python
>> library. If it is native python library, why Ubuntu18 doesn't include it. I will
>> work with Dandan to collect more information.
>>
>> Right, that's exactly what I'm asking for. Thank you very much!
>> Laszlo
>>
>>>> -----Original Message-----
>>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>>> Sent: Tuesday, February 12, 2019 8:24 PM
>>>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan
>> <dandan.bi@intel.com>
>>>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>>>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted
>> function
>>>>
>>>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>>>> On some Linux environment, there may be no distutils.util
>>>>>> library for python3 that will cause build tool crash.
>>>>>> This patch implement distutils.util.split_quoted
>>>>>> in BaseTools so that the Basetools will be independent with
>>>>>> distutils.util library.
>>>>>>
>>>>>> Feng, Bob C (3):
>>>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>>>   BaseTools: Implement splitquoted function in UPT
>>>>>>   BaseTools: unit test for splitquoted function
>>>>>>
>>>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50
>> ++++++++++++++++++++++++++++++++++++++++++++++----
>>>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47
>> ++++++++++++++++++++++++++++++++++++++++++++---
>>>>>>  BaseTools/Tests/TestStringSplit.py                    | 38
>> ++++++++++++++++++++++++++++++++++++++
>>>>>>  3 files changed, 128 insertions(+), 7 deletions(-)
>>>>>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
>>>>>>
>>>>>
>>>>> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
>>>>> like the issue can be resolved by a simple package installation on
>>>>> Ubuntu 18:
>>>>>
>>>>> https://superuser.com/questions/1319047/cant-install-virtual-
>> interpreter-in-pycharm-in-linux
>>>>>
>>>>> """
>>>>> sudo apt-get install python3-distutils
>>>>> """
>>>>>
>>>>> I'm not a Ubuntu user myself; so all I can do here (without installing a
>>>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>>>
>>>>> https://packages.ubuntu.com/search?keywords=python3-
>> distutils&searchon=names&suite=all&section=all
>>>>>
>>>>> python3-distutils appears available for both "bionic (18.04LTS)" and
>>>>> "cosmic (18.10)".
>>>>>
>>>>> Dandan, if you install python3-distutils, does that solve the issue for
>> you?
>>>>
>>>> I'd still like to get an answer to my question, before the series is pushed.
>>>>
>>>> Thanks,
>>>> Laszlo
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel