[Patch 0/3] BaseTools: Implement splitquoted function

[Patch 0/3] BaseTools: Implement splitquoted function

[Feng, Bob C]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
On some Linux environment, there may be no distutils.util
library for python3 that will cause build tool crash.
This patch implement distutils.util.split_quoted
in BaseTools so that the Basetools will be independent with
distutils.util library.

Feng, Bob C (3):
  BaseTools: Implement splitquoted function in Build tool
  BaseTools: Implement splitquoted function in UPT
  BaseTools: unit test for splitquoted function

 BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
 BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
 BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 128 insertions(+), 7 deletions(-)
 create mode 100644 BaseTools/Tests/TestStringSplit.py

-- 
2.20.1.windows.1

[Patch 1/3] BaseTools: Implement splitquoted function in Build tool

[Feng, Bob C]

https://bugzilla.tianocore.org/show_bug.cgi?id=1509
On some Linux environment, there may be no distutils.util
library for python3 that will cause build tool crash.
This patch implement distutils.util.split_quoted
in BaseTools so that the build tool will be independent with
distutils.util library.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
---
 BaseTools/Source/Python/AutoGen/UniClassObject.py | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 46 insertions(+), 4 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py b/BaseTools/Source/Python/AutoGen/UniClassObject.py
index d162387cc5..c4b39d25d4 100644
--- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
+++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
@@ -16,11 +16,10 @@
 ##
 # Import Modules
 #
 from __future__ import print_function
 import Common.LongFilePathOs as os, codecs, re
-import distutils.util
 import Common.EdkLogger as EdkLogger
 from io import BytesIO
 from Common.BuildToolError import *
 from Common.StringUtils import GetLineNo
 from Common.Misc import PathClass
@@ -41,11 +40,11 @@ NON_BREAKING_CHAR = u'\uFFF2'
 CR = u'\u000D'
 LF = u'\u000A'
 NULL = u'\u0000'
 TAB = u'\t'
 BACK_SLASH_PLACEHOLDER = u'\u0006'
-
+WHITESPACE = ' \t\n\r\v\f'
 gIncludePattern = re.compile("^#include +[\"<]+([^\"< >]+)[>\"]+$", re.MULTILINE | re.UNICODE)
 
 ## Convert a unicode string to a Hex list
 #
 # Convert a unicode string to a Hex list
@@ -216,10 +215,54 @@ def StripComments(Line):
             CommentPos = Line.find (Comment, CommentPos + 1)
         else:
             return Line[:CommentPos].strip()
     return Line.strip()
 
+#
+# This function implement distutils.util.split_quoted in another way.
+#
+def splitquoted(s):
+    words = []
+    word = []
+    InQuoted = False
+    escaped = False
+    CurrentQ = ""
+    s = s.strip()
+    for ch in s:
+        if escaped:
+            # preserve whatever is escaped;
+            # This char will become part of the current word
+            word.append(ch)
+            escaped = False
+            continue
+
+        if ch in WHITESPACE and not InQuoted:
+            # unescaped, unquoted whitespace: now
+            # we definitely have a word delimiter
+            if "".join(word):
+                words.append("".join(word))
+            word = []
+            continue
+        elif ch == "\\":
+            escaped = True
+            if not InQuoted:
+                continue
+        else:
+            # handle singly-quoted string or doubly-quoted string
+            if ch =='"' or ch == "'":
+                if not InQuoted:
+                    InQuoted = True
+                    CurrentQ = ch
+                    continue
+                elif ch == CurrentQ:
+                    InQuoted = False
+                    continue
+        word.append(ch)
+    if word:
+        words.append("".join(word))
+    return words
+
 ## UniFileClassObject
 #
 # A structure for .uni file definition
 #
 class UniFileClassObject(object):
@@ -232,16 +275,15 @@ class UniFileClassObject(object):
         self.OrderedStringListByToken = {}      #{ u'LanguageIdentifier' : {Token: StringDefClassObject} }
         self.IsCompatibleMode = IsCompatibleMode
         self.IncludePathList = IncludePathList
         if len(self.FileList) > 0:
             self.LoadUniFiles(FileList)
-
     #
     # Get Language definition
     #
     def GetLangDef(self, File, Line):
-        Lang = distutils.util.split_quoted((Line.split(u"//")[0]))
+        Lang = splitquoted((Line.split(u"//")[0]))
         if len(Lang) != 3:
             try:
                 FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
             except UnicodeError as X:
                 EdkLogger.error("build", FILE_READ_FAILURE, "File read failure: %s" % str(X), ExtraData=File);
-- 
2.20.1.windows.1

Re: [Patch 1/3] BaseTools: Implement splitquoted function in Build tool

[Carsey, Jaben]

Code looks good, but I think the function comment is lacking.  How about describe the function and what it does instead of saying it replaces distutils function (which while correct, didn't really answer my question of "what is this supposed to do").

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Feng, Bob C
> Sent: Saturday, February 02, 2019 9:55 PM
> To: edk2-devel@lists.01.org
> Cc: Gao, Liming <liming.gao@intel.com>
> Subject: [edk2] [Patch 1/3] BaseTools: Implement splitquoted function in
> Build tool
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1509
> On some Linux environment, there may be no distutils.util
> library for python3 that will cause build tool crash.
> This patch implement distutils.util.split_quoted
> in BaseTools so that the build tool will be independent with
> distutils.util library.
> 
> Contributed-under: TianoCore Contribution Agreement 1.1
> Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> ---
>  BaseTools/Source/Python/AutoGen/UniClassObject.py | 50
> ++++++++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 46 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> index d162387cc5..c4b39d25d4 100644
> --- a/BaseTools/Source/Python/AutoGen/UniClassObject.py
> +++ b/BaseTools/Source/Python/AutoGen/UniClassObject.py
> @@ -16,11 +16,10 @@
>  ##
>  # Import Modules
>  #
>  from __future__ import print_function
>  import Common.LongFilePathOs as os, codecs, re
> -import distutils.util
>  import Common.EdkLogger as EdkLogger
>  from io import BytesIO
>  from Common.BuildToolError import *
>  from Common.StringUtils import GetLineNo
>  from Common.Misc import PathClass
> @@ -41,11 +40,11 @@ NON_BREAKING_CHAR = u'\uFFF2'
>  CR = u'\u000D'
>  LF = u'\u000A'
>  NULL = u'\u0000'
>  TAB = u'\t'
>  BACK_SLASH_PLACEHOLDER = u'\u0006'
> -
> +WHITESPACE = ' \t\n\r\v\f'
>  gIncludePattern = re.compile("^#include +[\"<]+([^\"< >]+)[>\"]+$",
> re.MULTILINE | re.UNICODE)
> 
>  ## Convert a unicode string to a Hex list
>  #
>  # Convert a unicode string to a Hex list
> @@ -216,10 +215,54 @@ def StripComments(Line):
>              CommentPos = Line.find (Comment, CommentPos + 1)
>          else:
>              return Line[:CommentPos].strip()
>      return Line.strip()
> 
> +#
> +# This function implement distutils.util.split_quoted in another way.
> +#
> +def splitquoted(s):
> +    words = []
> +    word = []
> +    InQuoted = False
> +    escaped = False
> +    CurrentQ = ""
> +    s = s.strip()
> +    for ch in s:
> +        if escaped:
> +            # preserve whatever is escaped;
> +            # This char will become part of the current word
> +            word.append(ch)
> +            escaped = False
> +            continue
> +
> +        if ch in WHITESPACE and not InQuoted:
> +            # unescaped, unquoted whitespace: now
> +            # we definitely have a word delimiter
> +            if "".join(word):
> +                words.append("".join(word))
> +            word = []
> +            continue
> +        elif ch == "\\":
> +            escaped = True
> +            if not InQuoted:
> +                continue
> +        else:
> +            # handle singly-quoted string or doubly-quoted string
> +            if ch =='"' or ch == "'":
> +                if not InQuoted:
> +                    InQuoted = True
> +                    CurrentQ = ch
> +                    continue
> +                elif ch == CurrentQ:
> +                    InQuoted = False
> +                    continue
> +        word.append(ch)
> +    if word:
> +        words.append("".join(word))
> +    return words
> +
>  ## UniFileClassObject
>  #
>  # A structure for .uni file definition
>  #
>  class UniFileClassObject(object):
> @@ -232,16 +275,15 @@ class UniFileClassObject(object):
>          self.OrderedStringListByToken = {}      #{ u'LanguageIdentifier' : {Token:
> StringDefClassObject} }
>          self.IsCompatibleMode = IsCompatibleMode
>          self.IncludePathList = IncludePathList
>          if len(self.FileList) > 0:
>              self.LoadUniFiles(FileList)
> -
>      #
>      # Get Language definition
>      #
>      def GetLangDef(self, File, Line):
> -        Lang = distutils.util.split_quoted((Line.split(u"//")[0]))
> +        Lang = splitquoted((Line.split(u"//")[0]))
>          if len(Lang) != 3:
>              try:
>                  FileIn = UniFileClassObject.OpenUniFile(LongFilePath(File.Path))
>              except UnicodeError as X:
>                  EdkLogger.error("build", FILE_READ_FAILURE, "File read failure: %s"
> % str(X), ExtraData=File);
> --
> 2.20.1.windows.1
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

[Patch 2/3] BaseTools: Implement splitquoted function in UPT

[Feng, Bob C]

On some Linux environment, there may be no distutils.util
library for python3 that will cause UPT crash.
This patch implement distutils.util.split_quoted
in BaseTools so that the UPT tool will be independent with
distutils.util library.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
---
 BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 44 insertions(+), 3 deletions(-)

diff --git a/BaseTools/Source/Python/UPT/Library/UniClassObject.py b/BaseTools/Source/Python/UPT/Library/UniClassObject.py
index bd7804b753..79aa252507 100644
--- a/BaseTools/Source/Python/UPT/Library/UniClassObject.py
+++ b/BaseTools/Source/Python/UPT/Library/UniClassObject.py
@@ -18,11 +18,10 @@ from __future__ import print_function
 
 ##
 # Import Modules
 #
 import os, codecs, re
-import distutils.util
 from Logger import ToolError
 from Logger import Log as EdkLogger
 from Logger import StringTable as ST
 from Library.StringUtils import GetLineNo
 from Library.Misc import PathClass
@@ -36,11 +35,11 @@ from Library.ParserValidate import CheckUTF16FileHeader
 UNICODE_WIDE_CHAR = u'\\wide'
 UNICODE_NARROW_CHAR = u'\\narrow'
 UNICODE_NON_BREAKING_CHAR = u'\\nbr'
 UNICODE_UNICODE_CR = '\r'
 UNICODE_UNICODE_LF = '\n'
-
+WHITESPACE = ' \t\n\r\v\f'
 NARROW_CHAR = u'\uFFF0'
 WIDE_CHAR = u'\uFFF1'
 NON_BREAKING_CHAR = u'\uFFF2'
 CR = u'\u000D'
 LF = u'\u000A'
@@ -296,11 +295,53 @@ class StringDefClassObject(object):
                 self.StringValue = self.StringValue + '\r\n' + Value
             else:
                 self.StringValue = Value
             self.StringValueByteList = UniToHexList(self.StringValue)
             self.Length = len(self.StringValueByteList)
+#
+# This function implement distutils.util.split_quoted in another way.
+#
+def splitquoted(s):
+    words = []
+    word = []
+    InQuoted = False
+    escaped = False
+    CurrentQ = ""
+    s = s.strip()
+    for ch in s:
+        if escaped:
+            # preserve whatever is escaped;
+            # This char will become part of the current word
+            word.append(ch)
+            escaped = False
+            continue
 
+        if ch in WHITESPACE and not InQuoted:
+            # unescaped, unquoted whitespace: now
+            # we definitely have a word delimiter
+            if "".join(word):
+                words.append("".join(word))
+            word = []
+            continue
+        elif ch == "\\":
+            escaped = True
+            if not InQuoted:
+                continue
+        else:
+            # handle singly-quoted string or doubly-quoted string
+            if ch =='"' or ch == "'":
+                if not InQuoted:
+                    InQuoted = True
+                    CurrentQ = ch
+                    continue
+                elif ch == CurrentQ:
+                    InQuoted = False
+                    continue
+        word.append(ch)
+    if word:
+        words.append("".join(word))
+    return words
 ## UniFileClassObject
 #
 # A structure for .uni file definition
 #
 class UniFileClassObject(object):
@@ -324,11 +365,11 @@ class UniFileClassObject(object):
 
     #
     # Get Language definition
     #
     def GetLangDef(self, File, Line):
-        Lang = distutils.util.split_quoted((Line.split(u"//")[0]))
+        Lang = splitquoted((Line.split(u"//")[0]))
         if len(Lang) != 3:
             try:
                 FileIn = codecs.open(File.Path, mode='rb', encoding='utf_8').readlines()
             except UnicodeError as Xstr:
                 FileIn = codecs.open(File.Path, mode='rb', encoding='utf_16').readlines()
-- 
2.20.1.windows.1

[Patch 3/3] BaseTools: unit test for splitquoted function

[Feng, Bob C]

unit test for splitquoted function

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
---
 BaseTools/Tests/TestStringSplit.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/BaseTools/Tests/TestStringSplit.py b/BaseTools/Tests/TestStringSplit.py
new file mode 100644
index 0000000000..f24200ecfc
--- /dev/null
+++ b/BaseTools/Tests/TestStringSplit.py
@@ -0,0 +1,38 @@
+## @file
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# This program and the accompanying materials
+# are licensed and made available under the terms and conditions of the BSD License
+# which accompanies this distribution.  The full text of the license may be found at
+# http://opensource.org/licenses/bsd-license.php
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+# WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+import unittest
+from AutoGen.UniClassObject import splitquoted
+
+class TestStringSplit(unittest.TestCase):
+    def test_split_string(self):
+        TestStr1 = """ 'hello "world"' """
+        ExpStr1 = ['hello "world"']
+        self.assertEqual(ExpStr1, splitquoted(TestStr1))
+
+        TestStr1 = """ "hello 'world'" """
+        ExpStr1 = ["hello 'world'"]
+        self.assertEqual(ExpStr1, splitquoted(TestStr1))
+
+        TestStr1 = """ "hello 'world'"  'hello "abcd"' """
+        ExpStr1 = ["hello 'world'", 'hello "abcd"']
+        self.assertEqual(ExpStr1, splitquoted(TestStr1))
+
+        TestStr1 = """ "hello 'world'"  'hello "abcd"'  \t\n\r\v\f  \x64 """
+        ExpStr1 = ["hello 'world'", 'hello "abcd"', '\x64']
+        self.assertEqual(ExpStr1, splitquoted(TestStr1))
+
+        TestStr1 = """ "hello 'world'"  'hello "abcd"  \t\n\r\v\f  \x64' """
+        ExpStr1 = ["hello 'world'", 'hello "abcd"  \t\n\r\v\f  \x64']
+        self.assertEqual(ExpStr1, splitquoted(TestStr1))
+
+if __name__ == '__main__':
+    unittest.main()
-- 
2.20.1.windows.1

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/03/19 06:55, Feng, Bob C wrote:
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
> On some Linux environment, there may be no distutils.util
> library for python3 that will cause build tool crash.
> This patch implement distutils.util.split_quoted
> in BaseTools so that the Basetools will be independent with
> distutils.util library.
> 
> Feng, Bob C (3):
>   BaseTools: Implement splitquoted function in Build tool
>   BaseTools: Implement splitquoted function in UPT
>   BaseTools: unit test for splitquoted function
> 
>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 128 insertions(+), 7 deletions(-)
>  create mode 100644 BaseTools/Tests/TestStringSplit.py
> 

Is this really necessary? BZ#1509 references Ubuntu18; however it looks
like the issue can be resolved by a simple package installation on
Ubuntu 18:

https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux

"""
sudo apt-get install python3-distutils
"""

I'm not a Ubuntu user myself; so all I can do here (without installing a
Ubuntu18 VM) is check the Ubuntu package directory:

https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all

python3-distutils appears available for both "bionic (18.04LTS)" and
"cosmic (18.10)".

Dandan, if you install python3-distutils, does that solve the issue for you?

Thanks
Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/04/19 20:12, Laszlo Ersek wrote:
> On 02/03/19 06:55, Feng, Bob C wrote:
>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>> On some Linux environment, there may be no distutils.util
>> library for python3 that will cause build tool crash.
>> This patch implement distutils.util.split_quoted
>> in BaseTools so that the Basetools will be independent with
>> distutils.util library.
>>
>> Feng, Bob C (3):
>>   BaseTools: Implement splitquoted function in Build tool
>>   BaseTools: Implement splitquoted function in UPT
>>   BaseTools: unit test for splitquoted function
>>
>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 128 insertions(+), 7 deletions(-)
>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
>>
> 
> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
> like the issue can be resolved by a simple package installation on
> Ubuntu 18:
> 
> https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux
> 
> """
> sudo apt-get install python3-distutils
> """
> 
> I'm not a Ubuntu user myself; so all I can do here (without installing a
> Ubuntu18 VM) is check the Ubuntu package directory:
> 
> https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all
> 
> python3-distutils appears available for both "bionic (18.04LTS)" and
> "cosmic (18.10)".
> 
> Dandan, if you install python3-distutils, does that solve the issue for you?

I'd still like to get an answer to my question, before the series is pushed.

Thanks,
Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Gao, Liming]

Laszlo:
 To install python3-distutils should resolve this issue. I expect BaseTools build functionality doesn't depend on the third party python lib. So, I suggest to check whether python3-distutils is the native python library. If it is native python library, why Ubuntu18 doesn't include it. I will work with Dandan to collect more information. 

Thanks
Liming
> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Tuesday, February 12, 2019 8:24 PM
> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
> 
> On 02/04/19 20:12, Laszlo Ersek wrote:
> > On 02/03/19 06:55, Feng, Bob C wrote:
> >> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
> >> On some Linux environment, there may be no distutils.util
> >> library for python3 that will cause build tool crash.
> >> This patch implement distutils.util.split_quoted
> >> in BaseTools so that the Basetools will be independent with
> >> distutils.util library.
> >>
> >> Feng, Bob C (3):
> >>   BaseTools: Implement splitquoted function in Build tool
> >>   BaseTools: Implement splitquoted function in UPT
> >>   BaseTools: unit test for splitquoted function
> >>
> >>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
> >>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
> >>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
> >>  3 files changed, 128 insertions(+), 7 deletions(-)
> >>  create mode 100644 BaseTools/Tests/TestStringSplit.py
> >>
> >
> > Is this really necessary? BZ#1509 references Ubuntu18; however it looks
> > like the issue can be resolved by a simple package installation on
> > Ubuntu 18:
> >
> > https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux
> >
> > """
> > sudo apt-get install python3-distutils
> > """
> >
> > I'm not a Ubuntu user myself; so all I can do here (without installing a
> > Ubuntu18 VM) is check the Ubuntu package directory:
> >
> > https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all
> >
> > python3-distutils appears available for both "bionic (18.04LTS)" and
> > "cosmic (18.10)".
> >
> > Dandan, if you install python3-distutils, does that solve the issue for you?
> 
> I'd still like to get an answer to my question, before the series is pushed.
> 
> Thanks,
> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/12/19 14:33, Gao, Liming wrote:
> Laszlo:
>  To install python3-distutils should resolve this issue. I expect BaseTools build functionality doesn't depend on the third party python lib.

I completely agree with your expectation, regarding *3rd party* python
packages. We shouldn't expect developers to install packages from
repositories that fall outside of their normal distro repos.

However, my understanding was that python3-distutils should be available
as a normal (not 3rd party) component on Ubuntu 18. I think we can
expect developers to install additional packages if those packages are
readily available in their normal (distro-provided) repos.

> So, I suggest to check whether python3-distutils is the native python library. If it is native python library, why Ubuntu18 doesn't include it. I will work with Dandan to collect more information. 

Right, that's exactly what I'm asking for. Thank you very much!
Laszlo

>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Tuesday, February 12, 2019 8:24 PM
>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>>
>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>> On some Linux environment, there may be no distutils.util
>>>> library for python3 that will cause build tool crash.
>>>> This patch implement distutils.util.split_quoted
>>>> in BaseTools so that the Basetools will be independent with
>>>> distutils.util library.
>>>>
>>>> Feng, Bob C (3):
>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>   BaseTools: Implement splitquoted function in UPT
>>>>   BaseTools: unit test for splitquoted function
>>>>
>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>>>>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>>>>  3 files changed, 128 insertions(+), 7 deletions(-)
>>>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
>>>>
>>>
>>> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
>>> like the issue can be resolved by a simple package installation on
>>> Ubuntu 18:
>>>
>>> https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux
>>>
>>> """
>>> sudo apt-get install python3-distutils
>>> """
>>>
>>> I'm not a Ubuntu user myself; so all I can do here (without installing a
>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>
>>> https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all
>>>
>>> python3-distutils appears available for both "bionic (18.04LTS)" and
>>> "cosmic (18.10)".
>>>
>>> Dandan, if you install python3-distutils, does that solve the issue for you?
>>
>> I'd still like to get an answer to my question, before the series is pushed.
>>
>> Thanks,
>> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Philippe Mathieu-Daudé]

On 2/12/19 3:02 PM, Laszlo Ersek wrote:
> On 02/12/19 14:33, Gao, Liming wrote:
>> Laszlo:
>>  To install python3-distutils should resolve this issue. I expect BaseTools build functionality doesn't depend on the third party python lib.
> 
> I completely agree with your expectation, regarding *3rd party* python
> packages. We shouldn't expect developers to install packages from
> repositories that fall outside of their normal distro repos.
> 
> However, my understanding was that python3-distutils should be available
> as a normal (not 3rd party) component on Ubuntu 18. I think we can
> expect developers to install additional packages if those packages are
> readily available in their normal (distro-provided) repos.

The documentation is not precise about the python package to install, it
simply states "Install Python 2.7.10":

https://github.com/tianocore/tianocore.github.io/wiki/Getting-Started-with-EDK-II

--

Except python, there is a precise list of packages to install for this
distro: "sudo apt-get install build-essential uuid-dev iasl git gcc-5
nasm", per:

https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC#Install_required_software_from_apt

Maybe we simply need to update the doc to ask python3 and add
python3-distutils in the list?

> 
>> So, I suggest to check whether python3-distutils is the native python library. If it is native python library, why Ubuntu18 doesn't include it. I will work with Dandan to collect more information. 
> 
> Right, that's exactly what I'm asking for. Thank you very much!
> Laszlo
> 
>>> -----Original Message-----
>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>> Sent: Tuesday, February 12, 2019 8:24 PM
>>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
>>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>>>
>>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>>> On some Linux environment, there may be no distutils.util
>>>>> library for python3 that will cause build tool crash.
>>>>> This patch implement distutils.util.split_quoted
>>>>> in BaseTools so that the Basetools will be independent with
>>>>> distutils.util library.
>>>>>
>>>>> Feng, Bob C (3):
>>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>>   BaseTools: Implement splitquoted function in UPT
>>>>>   BaseTools: unit test for splitquoted function
>>>>>
>>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>>>>>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>>>>>  3 files changed, 128 insertions(+), 7 deletions(-)
>>>>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
>>>>>
>>>>
>>>> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
>>>> like the issue can be resolved by a simple package installation on
>>>> Ubuntu 18:
>>>>
>>>> https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux
>>>>
>>>> """
>>>> sudo apt-get install python3-distutils
>>>> """
>>>>
>>>> I'm not a Ubuntu user myself; so all I can do here (without installing a
>>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>>
>>>> https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all
>>>>
>>>> python3-distutils appears available for both "bionic (18.04LTS)" and
>>>> "cosmic (18.10)".
>>>>
>>>> Dandan, if you install python3-distutils, does that solve the issue for you?
>>>
>>> I'd still like to get an answer to my question, before the series is pushed.
>>>
>>> Thanks,
>>> Laszlo
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
> 

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/12/19 18:16, Philippe Mathieu-Daudé wrote:
> On 2/12/19 3:02 PM, Laszlo Ersek wrote:
>> On 02/12/19 14:33, Gao, Liming wrote:
>>> Laszlo:
>>>  To install python3-distutils should resolve this issue. I expect BaseTools build functionality doesn't depend on the third party python lib.
>>
>> I completely agree with your expectation, regarding *3rd party* python
>> packages. We shouldn't expect developers to install packages from
>> repositories that fall outside of their normal distro repos.
>>
>> However, my understanding was that python3-distutils should be available
>> as a normal (not 3rd party) component on Ubuntu 18. I think we can
>> expect developers to install additional packages if those packages are
>> readily available in their normal (distro-provided) repos.
> 
> The documentation is not precise about the python package to install, it
> simply states "Install Python 2.7.10":
> 
> https://github.com/tianocore/tianocore.github.io/wiki/Getting-Started-with-EDK-II
> 
> --
> 
> Except python, there is a precise list of packages to install for this
> distro: "sudo apt-get install build-essential uuid-dev iasl git gcc-5
> nasm", per:
> 
> https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC#Install_required_software_from_apt
> 
> Maybe we simply need to update the doc to ask python3 and add
> python3-distutils in the list?

I didn't realize we had Ubuntu-specific instructions for setting up the
environment :) So yes, if Liming & Hao confirm python3-distutils is
available as part of Ubuntu 18, then the docs should likely be updated
as you say.

Thanks
Laszlo

> 
>>
>>> So, I suggest to check whether python3-distutils is the native python library. If it is native python library, why Ubuntu18 doesn't include it. I will work with Dandan to collect more information. 
>>
>> Right, that's exactly what I'm asking for. Thank you very much!
>> Laszlo
>>
>>>> -----Original Message-----
>>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>>> Sent: Tuesday, February 12, 2019 8:24 PM
>>>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
>>>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>>>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>>>>
>>>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>>>> On some Linux environment, there may be no distutils.util
>>>>>> library for python3 that will cause build tool crash.
>>>>>> This patch implement distutils.util.split_quoted
>>>>>> in BaseTools so that the Basetools will be independent with
>>>>>> distutils.util library.
>>>>>>
>>>>>> Feng, Bob C (3):
>>>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>>>   BaseTools: Implement splitquoted function in UPT
>>>>>>   BaseTools: unit test for splitquoted function
>>>>>>
>>>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>>>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>>>>>>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>>>>>>  3 files changed, 128 insertions(+), 7 deletions(-)
>>>>>>  create mode 100644 BaseTools/Tests/TestStringSplit.py
>>>>>>
>>>>>
>>>>> Is this really necessary? BZ#1509 references Ubuntu18; however it looks
>>>>> like the issue can be resolved by a simple package installation on
>>>>> Ubuntu 18:
>>>>>
>>>>> https://superuser.com/questions/1319047/cant-install-virtual-interpreter-in-pycharm-in-linux
>>>>>
>>>>> """
>>>>> sudo apt-get install python3-distutils
>>>>> """
>>>>>
>>>>> I'm not a Ubuntu user myself; so all I can do here (without installing a
>>>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>>>
>>>>> https://packages.ubuntu.com/search?keywords=python3-distutils&searchon=names&suite=all&section=all
>>>>>
>>>>> python3-distutils appears available for both "bionic (18.04LTS)" and
>>>>> "cosmic (18.10)".
>>>>>
>>>>> Dandan, if you install python3-distutils, does that solve the issue for you?
>>>>
>>>> I'd still like to get an answer to my question, before the series is pushed.
>>>>
>>>> Thanks,
>>>> Laszlo
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>>

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Feng, Bob C]

Hi Laszlo, Liming, Jaben and Dandan,

I found this is a Ubuntu18 bug. Refer to https://bugs.launchpad.net/ubuntu/+source/fdroidserver/+bug/1762183

And Ubuntu fixed this bug via a Ubuntu18.04.1 update package which was published on 2018-08-09. Refer to https://launchpad.net/ubuntu/+source/fdroidserver/1.0.9-1~18.04.1

While the latest Ubuntu 18.04 release (ubuntu-18.04.1-desktop-amd64.iso)  on http://releases.ubuntu.com/18.04/ was published on 2018-07-25.  So there is no distutils.util library on Ubuntu18.04 default installation. But I think it's clear that distutils.util is not *3rd party* python library. 

I have tried that the command "sudo apt upgrade"  can't fix this bug while the command "sudo apt-get install python3-distutils" works.

Thanks,
Bob

-----Original Message-----
From: Laszlo Ersek [mailto:lersek@redhat.com] 
Sent: Tuesday, February 12, 2019 10:02 PM
To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
Cc: edk2-devel@lists.01.org
Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function

On 02/12/19 14:33, Gao, Liming wrote:
> Laszlo:
>  To install python3-distutils should resolve this issue. I expect BaseTools build functionality doesn't depend on the third party python lib.

I completely agree with your expectation, regarding *3rd party* python packages. We shouldn't expect developers to install packages from repositories that fall outside of their normal distro repos.

However, my understanding was that python3-distutils should be available as a normal (not 3rd party) component on Ubuntu 18. I think we can expect developers to install additional packages if those packages are readily available in their normal (distro-provided) repos.

> So, I suggest to check whether python3-distutils is the native python library. If it is native python library, why Ubuntu18 doesn't include it. I will work with Dandan to collect more information. 

Right, that's exactly what I'm asking for. Thank you very much!
Laszlo

>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Tuesday, February 12, 2019 8:24 PM
>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan 
>> <dandan.bi@intel.com>
>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted 
>> function
>>
>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>> On some Linux environment, there may be no distutils.util library 
>>>> for python3 that will cause build tool crash.
>>>> This patch implement distutils.util.split_quoted in BaseTools so 
>>>> that the Basetools will be independent with distutils.util library.
>>>>
>>>> Feng, Bob C (3):
>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>   BaseTools: Implement splitquoted function in UPT
>>>>   BaseTools: unit test for splitquoted function
>>>>
>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>>>>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>>>>  3 files changed, 128 insertions(+), 7 deletions(-)  create mode 
>>>> 100644 BaseTools/Tests/TestStringSplit.py
>>>>
>>>
>>> Is this really necessary? BZ#1509 references Ubuntu18; however it 
>>> looks like the issue can be resolved by a simple package 
>>> installation on Ubuntu 18:
>>>
>>> https://superuser.com/questions/1319047/cant-install-virtual-interpr
>>> eter-in-pycharm-in-linux
>>>
>>> """
>>> sudo apt-get install python3-distutils """
>>>
>>> I'm not a Ubuntu user myself; so all I can do here (without 
>>> installing a
>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>
>>> https://packages.ubuntu.com/search?keywords=python3-distutils&search
>>> on=names&suite=all&section=all
>>>
>>> python3-distutils appears available for both "bionic (18.04LTS)" and 
>>> "cosmic (18.10)".
>>>
>>> Dandan, if you install python3-distutils, does that solve the issue for you?
>>
>> I'd still like to get an answer to my question, before the series is pushed.
>>
>> Thanks,
>> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Gao, Liming]

Bob:
  So, this is OS issue. I prefer to update wiki page to describe how to resolve it. 

Thanks
Liming
>-----Original Message-----
>From: Feng, Bob C
>Sent: Thursday, February 14, 2019 10:51 AM
>To: Laszlo Ersek <lersek@redhat.com>; Gao, Liming <liming.gao@intel.com>;
>Bi, Dandan <dandan.bi@intel.com>; Carsey, Jaben <jaben.carsey@intel.com>
>Cc: edk2-devel@lists.01.org
>Subject: RE: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>
>Hi Laszlo, Liming, Jaben and Dandan,
>
>I found this is a Ubuntu18 bug. Refer to
>https://bugs.launchpad.net/ubuntu/+source/fdroidserver/+bug/1762183
>
>And Ubuntu fixed this bug via a Ubuntu18.04.1 update package which was
>published on 2018-08-09. Refer to
>https://launchpad.net/ubuntu/+source/fdroidserver/1.0.9-1~18.04.1
>
>While the latest Ubuntu 18.04 release (ubuntu-18.04.1-desktop-amd64.iso)
>on http://releases.ubuntu.com/18.04/ was published on 2018-07-25.  So there
>is no distutils.util library on Ubuntu18.04 default installation. But I think it's
>clear that distutils.util is not *3rd party* python library.
>
>I have tried that the command "sudo apt upgrade"  can't fix this bug while the
>command "sudo apt-get install python3-distutils" works.
>
>Thanks,
>Bob
>
>-----Original Message-----
>From: Laszlo Ersek [mailto:lersek@redhat.com]
>Sent: Tuesday, February 12, 2019 10:02 PM
>To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
><bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>
>Cc: edk2-devel@lists.01.org
>Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>
>On 02/12/19 14:33, Gao, Liming wrote:
>> Laszlo:
>>  To install python3-distutils should resolve this issue. I expect BaseTools
>build functionality doesn't depend on the third party python lib.
>
>I completely agree with your expectation, regarding *3rd party* python
>packages. We shouldn't expect developers to install packages from
>repositories that fall outside of their normal distro repos.
>
>However, my understanding was that python3-distutils should be available as
>a normal (not 3rd party) component on Ubuntu 18. I think we can expect
>developers to install additional packages if those packages are readily available
>in their normal (distro-provided) repos.
>
>> So, I suggest to check whether python3-distutils is the native python library.
>If it is native python library, why Ubuntu18 doesn't include it. I will work with
>Dandan to collect more information.
>
>Right, that's exactly what I'm asking for. Thank you very much!
>Laszlo
>
>>> -----Original Message-----
>>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>>> Sent: Tuesday, February 12, 2019 8:24 PM
>>> To: Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan
>>> <dandan.bi@intel.com>
>>> Cc: edk2-devel@lists.01.org; Gao, Liming <liming.gao@intel.com>
>>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted
>>> function
>>>
>>> On 02/04/19 20:12, Laszlo Ersek wrote:
>>>> On 02/03/19 06:55, Feng, Bob C wrote:
>>>>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
>>>>> On some Linux environment, there may be no distutils.util library
>>>>> for python3 that will cause build tool crash.
>>>>> This patch implement distutils.util.split_quoted in BaseTools so
>>>>> that the Basetools will be independent with distutils.util library.
>>>>>
>>>>> Feng, Bob C (3):
>>>>>   BaseTools: Implement splitquoted function in Build tool
>>>>>   BaseTools: Implement splitquoted function in UPT
>>>>>   BaseTools: unit test for splitquoted function
>>>>>
>>>>>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50
>++++++++++++++++++++++++++++++++++++++++++++++----
>>>>>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47
>++++++++++++++++++++++++++++++++++++++++++++---
>>>>>  BaseTools/Tests/TestStringSplit.py                    | 38
>++++++++++++++++++++++++++++++++++++++
>>>>>  3 files changed, 128 insertions(+), 7 deletions(-)  create mode
>>>>> 100644 BaseTools/Tests/TestStringSplit.py
>>>>>
>>>>
>>>> Is this really necessary? BZ#1509 references Ubuntu18; however it
>>>> looks like the issue can be resolved by a simple package
>>>> installation on Ubuntu 18:
>>>>
>>>> https://superuser.com/questions/1319047/cant-install-virtual-interpr
>>>> eter-in-pycharm-in-linux
>>>>
>>>> """
>>>> sudo apt-get install python3-distutils """
>>>>
>>>> I'm not a Ubuntu user myself; so all I can do here (without
>>>> installing a
>>>> Ubuntu18 VM) is check the Ubuntu package directory:
>>>>
>>>> https://packages.ubuntu.com/search?keywords=python3-
>distutils&search
>>>> on=names&suite=all&section=all
>>>>
>>>> python3-distutils appears available for both "bionic (18.04LTS)" and
>>>> "cosmic (18.10)".
>>>>
>>>> Dandan, if you install python3-distutils, does that solve the issue for you?
>>>
>>> I'd still like to get an answer to my question, before the series is pushed.
>>>
>>> Thanks,
>>> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/14/19 04:23, Gao, Liming wrote:
> Bob:
>   So, this is OS issue. I prefer to update wiki page to describe how to resolve it. 

Great idea. I think this is probably the affected section:

https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC#Install_required_software_from_apt

We should simply add "python3-distutils" to the list.

Possibly, also mention "python3-distutils" in
"BaseTools/Conf/tools_def.template".

Thanks
Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Gao, Liming]

Laszlo:
  I have updated wiki page to mention python3-distutils. I suggest to update BaseTools\ReadMe.txt for it. 

Thanks
Liming
> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Thursday, February 14, 2019 11:35 PM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Carsey, Jaben
> <jaben.carsey@intel.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
> 
> On 02/14/19 04:23, Gao, Liming wrote:
> > Bob:
> >   So, this is OS issue. I prefer to update wiki page to describe how to resolve it.
> 
> Great idea. I think this is probably the affected section:
> 
> https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC#Install_required_software_from_apt
> 
> We should simply add "python3-distutils" to the list.
> 
> Possibly, also mention "python3-distutils" in
> "BaseTools/Conf/tools_def.template".
> 
> Thanks
> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

On 02/15/19 15:35, Gao, Liming wrote:
> Laszlo:
>   I have updated wiki page to mention python3-distutils.

Thanks!

> I suggest to update BaseTools\ReadMe.txt for it. 

Ah, haha, there's such a file as "BaseTools/ReadMe.txt"? :) I don't
think I've ever noticed it. :)

But, now I agree that "BaseTools/ReadMe.txt" is a better place for
exposing this dependency.

Thanks!
Laszlo

> 
> Thanks
> Liming
>> -----Original Message-----
>> From: Laszlo Ersek [mailto:lersek@redhat.com]
>> Sent: Thursday, February 14, 2019 11:35 PM
>> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Carsey, Jaben
>> <jaben.carsey@intel.com>
>> Cc: edk2-devel@lists.01.org
>> Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
>>
>> On 02/14/19 04:23, Gao, Liming wrote:
>>> Bob:
>>>   So, this is OS issue. I prefer to update wiki page to describe how to resolve it.
>>
>> Great idea. I think this is probably the affected section:
>>
>> https://github.com/tianocore/tianocore.github.io/wiki/Using-EDK-II-with-Native-GCC#Install_required_software_from_apt
>>
>> We should simply add "python3-distutils" to the list.
>>
>> Possibly, also mention "python3-distutils" in
>> "BaseTools/Conf/tools_def.template".
>>
>> Thanks
>> Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Laszlo Ersek]

Hi Bob,

On 02/14/19 03:51, Feng, Bob C wrote:
> Hi Laszlo, Liming, Jaben and Dandan,
>
> I found this is a Ubuntu18 bug. Refer to
> https://bugs.launchpad.net/ubuntu/+source/fdroidserver/+bug/1762183

That Ubuntu bug is *related*, but it's not a bug that matters for us.

The bug above is about another package; it is called "fdroidserver". The
"fdroidserver" package has a hard runtime dependency on the python
"distutils.util" module.

Package management systems on Linux distributions track inter-package
dependencies. That is, if the meta-data on the "fdroidserver" package
explicitly lists the "python3-distutils" package as a dependency, then
the package management system will automatically install in
"python3-distutils" when the user requests "fdroidserver".

However, if the meta-data on the "fdroidserver" package are incorrect
(buggy), then the package management system will have no clue. And, if
the "python3-distutils" package is not already installed for some other
reason, then the user will get an installation or runtime error, when
they try to install or run "fdroidserver".

So, to be clear, the bug report you reference describes a *similar*
situation to ours (a missing dependency), but it's not the same case.
The bug you refer to is in the "fdroidserver" package, and they fixed it
in Ubuntu by updating / correcting the meta-data on the "fdroidserver"
package. The "python3-distutils" package, or other parts of the OS, were
not touched.

> And Ubuntu fixed this bug via a Ubuntu18.04.1 update package which was
> published on 2018-08-09. Refer to
> https://launchpad.net/ubuntu/+source/fdroidserver/1.0.9-1~18.04.1

The link you provide confirms what I wrote above. It is a changelog for
the "fdroidserver" package, and the relevant entry says "fix missing
Python distutils dependency".

> While the latest Ubuntu 18.04 release
> (ubuntu-18.04.1-desktop-amd64.iso)  on
> http://releases.ubuntu.com/18.04/ was published on 2018-07-25.  So
> there is no distutils.util library on Ubuntu18.04 default
> installation.

No, this statement can't be correct. I'm pretty sure that the
"python3-distutils" package *was* available when Ubuntu 18.04 -- not
18.04.1 -- was originally released.

I've just downloaded

  http://old-releases.ubuntu.com/releases/bionic/ubuntu-18.04-desktop-amd64.iso

While the package is not on the ISO, a whole lot of *other* packages are
also not there -- for example I can't see any python at all. So I'm
thinking that python3-distutils was only available from the network.

But, I'm pretty sure python3-distutils *was* available from the network,
when 18.04 was originally released.

What was indeed broken in the original Ubuntu 18.04 release was the
meta-data on the "fdroidserver" package.

> But I think it's clear that distutils.util is not *3rd party* python
> library.

I agree; and that is what matters.

> I have tried that the command "sudo apt upgrade"  can't fix this bug

This "apt" command would only be relevant if you had the *old*
fdroidserver package installed (with the missing dependency in its
meta-data). Then, "apt" would install the *new* fdroidserver package for
you, and it would also act on the now-visible "python3-distutils"
dependency. Thus, "apt" would automatically install in
python3-distutils, as a dependency.

For installing "python3-distutils" *in itself*, the above "apt" command
is totally useless.

> while the command "sudo apt-get install python3-distutils" works.

Yes, it does, because here you are specifically requesting the
python3-distutils package.

And that's what matters. Ubuntu users simply need to install
python3-distutils manually, from their official package repositories, if
they want to use BaseTools (as part of the upstream edk2 git repo).

So I think this edk2 patch set is not necessary, after all.

Thanks,
Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Feng, Bob C]

Hi Laszlo,

Thanks for you detailed clarification. 

This patch will not be pushed to edk2 master and Ubuntu user need to install python3-distutils package via "sudo apt-get install python3-distutils" to resolve this issue.

Thanks,
Bob

-----Original Message-----
From: Laszlo Ersek [mailto:lersek@redhat.com] 
Sent: Thursday, February 14, 2019 11:32 PM
To: Feng, Bob C <bob.c.feng@intel.com>
Cc: Gao, Liming <liming.gao@intel.com>; Bi, Dandan <dandan.bi@intel.com>; Carsey, Jaben <jaben.carsey@intel.com>; edk2-devel@lists.01.org
Subject: Re: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function

Hi Bob,

On 02/14/19 03:51, Feng, Bob C wrote:
> Hi Laszlo, Liming, Jaben and Dandan,
>
> I found this is a Ubuntu18 bug. Refer to
> https://bugs.launchpad.net/ubuntu/+source/fdroidserver/+bug/1762183

That Ubuntu bug is *related*, but it's not a bug that matters for us.

The bug above is about another package; it is called "fdroidserver". The "fdroidserver" package has a hard runtime dependency on the python "distutils.util" module.

Package management systems on Linux distributions track inter-package dependencies. That is, if the meta-data on the "fdroidserver" package explicitly lists the "python3-distutils" package as a dependency, then the package management system will automatically install in "python3-distutils" when the user requests "fdroidserver".

However, if the meta-data on the "fdroidserver" package are incorrect (buggy), then the package management system will have no clue. And, if the "python3-distutils" package is not already installed for some other reason, then the user will get an installation or runtime error, when they try to install or run "fdroidserver".

So, to be clear, the bug report you reference describes a *similar* situation to ours (a missing dependency), but it's not the same case.
The bug you refer to is in the "fdroidserver" package, and they fixed it in Ubuntu by updating / correcting the meta-data on the "fdroidserver"
package. The "python3-distutils" package, or other parts of the OS, were not touched.

> And Ubuntu fixed this bug via a Ubuntu18.04.1 update package which was 
> published on 2018-08-09. Refer to
> https://launchpad.net/ubuntu/+source/fdroidserver/1.0.9-1~18.04.1

The link you provide confirms what I wrote above. It is a changelog for the "fdroidserver" package, and the relevant entry says "fix missing Python distutils dependency".

> While the latest Ubuntu 18.04 release
> (ubuntu-18.04.1-desktop-amd64.iso)  on 
> http://releases.ubuntu.com/18.04/ was published on 2018-07-25.  So 
> there is no distutils.util library on Ubuntu18.04 default 
> installation.

No, this statement can't be correct. I'm pretty sure that the "python3-distutils" package *was* available when Ubuntu 18.04 -- not
18.04.1 -- was originally released.

I've just downloaded

  http://old-releases.ubuntu.com/releases/bionic/ubuntu-18.04-desktop-amd64.iso

While the package is not on the ISO, a whole lot of *other* packages are also not there -- for example I can't see any python at all. So I'm thinking that python3-distutils was only available from the network.

But, I'm pretty sure python3-distutils *was* available from the network, when 18.04 was originally released.

What was indeed broken in the original Ubuntu 18.04 release was the meta-data on the "fdroidserver" package.

> But I think it's clear that distutils.util is not *3rd party* python 
> library.

I agree; and that is what matters.

> I have tried that the command "sudo apt upgrade"  can't fix this bug

This "apt" command would only be relevant if you had the *old* fdroidserver package installed (with the missing dependency in its meta-data). Then, "apt" would install the *new* fdroidserver package for you, and it would also act on the now-visible "python3-distutils"
dependency. Thus, "apt" would automatically install in python3-distutils, as a dependency.

For installing "python3-distutils" *in itself*, the above "apt" command is totally useless.

> while the command "sudo apt-get install python3-distutils" works.

Yes, it does, because here you are specifically requesting the python3-distutils package.

And that's what matters. Ubuntu users simply need to install python3-distutils manually, from their official package repositories, if they want to use BaseTools (as part of the upstream edk2 git repo).

So I think this edk2 patch set is not necessary, after all.

Thanks,
Laszlo

Re: [Patch 0/3] BaseTools: Implement splitquoted function

[Gao, Liming]

Reviewed-by: Liming Gao <liming.gao@intel.com>

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Feng, Bob C
> Sent: Sunday, February 3, 2019 1:55 PM
> To: edk2-devel@lists.01.org
> Subject: [edk2] [Patch 0/3] BaseTools: Implement splitquoted function
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1509
> On some Linux environment, there may be no distutils.util
> library for python3 that will cause build tool crash.
> This patch implement distutils.util.split_quoted
> in BaseTools so that the Basetools will be independent with
> distutils.util library.
> 
> Feng, Bob C (3):
>   BaseTools: Implement splitquoted function in Build tool
>   BaseTools: Implement splitquoted function in UPT
>   BaseTools: unit test for splitquoted function
> 
>  BaseTools/Source/Python/AutoGen/UniClassObject.py     | 50 ++++++++++++++++++++++++++++++++++++++++++++++----
>  BaseTools/Source/Python/UPT/Library/UniClassObject.py | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>  BaseTools/Tests/TestStringSplit.py                    | 38 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 128 insertions(+), 7 deletions(-)
>  create mode 100644 BaseTools/Tests/TestStringSplit.py
> 
> --
> 2.20.1.windows.1
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel