[PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[Heng Luo]

Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=2656

1. Add GetPackagesPath.py, it will be used to get package pathes from
  special directories. A sub directory is a qualified package path
  when an EDKII Package can be found under it.
2. Add SetPackagesPath.bat and SetPackagesPath.sh, these scripts call
  GetPackagesPath.py to collect all package paths under specified
  directories and append them to PACKAGES_PATH environment variable.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Amy Chan <amy.chan@intel.com>
Signed-off-by: Heng Luo <heng.luo@intel.com>
---
 BaseTools/Scripts/GetPackagesPath.py  | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 BaseTools/Scripts/SetPackagesPath.bat | 33 +++++++++++++++++++++++++++++++++
 BaseTools/Scripts/SetPackagesPath.sh  | 40 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 167 insertions(+)

diff --git a/BaseTools/Scripts/GetPackagesPath.py b/BaseTools/Scripts/GetPackagesPath.py
new file mode 100644
index 0000000000..82d66a6106
--- /dev/null
+++ b/BaseTools/Scripts/GetPackagesPath.py
@@ -0,0 +1,94 @@
+## @file
+# Get all recursive package paths from special directories.
+#
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import os
+import glob
+import argparse
+
+#
+# Globals for help information
+#
+__prog__ = 'GetPackagesPath.py'
+__copyright__ = 'Copyright (c) 2020, Intel Corporation. All rights reserved.'
+__description__ = 'Gets all recursive package paths in specified directory.\n'
+
+def list_packages_path(root):
+    """ Gets all recursive package paths in specified directory.
+        A directory is a package path if it satisfies conditions below:
+        1. it is a directory
+        2. it is not an EDK II Package. An EDK II Package (directory) is
+           a directory that contains an EDK II package declaration (DEC) file.
+        3. it contains at least one first level EDK II Package.
+        Note: A directory is not package path but its subdirectory could be.
+        Example: edk2-platforms/Features is not package path
+        but edk2-platforms/Features/Intel is.
+
+        :param root: The specified directory to find package paths in it
+        :type root: String
+        :returns: Return all recursive package paths
+        :rtype: String list
+    """
+
+    if (not os.path.exists(root)) or (not os.path.isdir(root)):
+         return []
+
+    if glob.glob(os.path.join(root, '*.dec')):
+        # it is an EDK II Package
+        return []
+
+    paths = []
+    contain_package = False
+    for filename in os.listdir(root):
+        filepath = os.path.join(root, filename)
+        if os.path.isdir(filepath):
+            if glob.glob(os.path.join(filepath, '*.dec')):
+                # it is an EDK II Package
+                contain_package = True
+            else:
+                # gets package paths for subdirectory if it is not package
+                paths = paths + list_packages_path(filepath)
+
+    if contain_package:
+        # root is a package path because it contains EDK II Package
+        # in first level folder, inset it to head of list
+        paths.insert(0, root)
+
+    # return package paths
+    return paths
+
+def get_packages_path(directories):
+    """ For each direcory in directories, gets all recursive package paths
+        in this directory and joins them into one string.
+
+        :param directories: the list of directory
+        :type directories: String list
+        :returns: Return string of package paths
+        :rtype: String
+    """
+
+    packages_path = ''
+    for directory in directories:
+        directory = os.path.abspath(directory)
+        paths = list_packages_path(directory)
+        for path in paths:
+            if packages_path == '':
+                packages_path = path
+            else:
+                packages_path += os.pathsep + path
+    return packages_path
+
+if __name__ == '__main__':
+    # Create command line argument parser object
+    parser = argparse.ArgumentParser(
+            prog=__prog__,
+            description=__description__ + __copyright__,
+            conflict_handler='resolve'
+    )
+    parser.add_argument('directory', nargs='+',
+            help='Specified directory where package packages are got from')
+    args = parser.parse_args()
+    print(get_packages_path(args.directory))
diff --git a/BaseTools/Scripts/SetPackagesPath.bat b/BaseTools/Scripts/SetPackagesPath.bat
new file mode 100644
index 0000000000..a8c45612c9
--- /dev/null
+++ b/BaseTools/Scripts/SetPackagesPath.bat
@@ -0,0 +1,33 @@
+@REM @file
+@REM Windows batch file to set PACKAGES_PATH environment
+@REM
+@REM Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+@REM SPDX-License-Identifier: BSD-2-Clause-Patent
+@REM
+@REM This script calls GetPackagesPath.py to collect all package paths under
+@REM specified directories and append them to PACKAGES_PATH environment
+@REM variable. A sub directory is a qualified package path when an EDKII
+@REM Package can be found under it.
+
+@echo off
+@if /I "%1"=="" @goto Usage
+@if /I "%1"=="-h" @goto Usage
+@if /I "%1"=="--help" @goto Usage
+@if /I "%1"=="/?" @goto Usage
+
+for /f %%i in ('python %EDK_TOOLS_PATH%\Scripts\GetPackagesPath.py %*') do (
+    if defined PACKAGES_PATH (
+        set "PACKAGES_PATH=%PACKAGES_PATH%;%%i"
+    ) else (
+        set "PACKAGES_PATH=%%i"
+    )
+)
+@goto End
+
+:Usage
+@echo Usage: SetPackagesPath.bat directory [directory ...]
+@echo Copyright(c) 2020, Intel Corporation. All rights reserved.
+@echo Options:
+@echo   --help, -h     Print this help screen and exit
+
+:End
diff --git a/BaseTools/Scripts/SetPackagesPath.sh b/BaseTools/Scripts/SetPackagesPath.sh
new file mode 100644
index 0000000000..e3e337254e
--- /dev/null
+++ b/BaseTools/Scripts/SetPackagesPath.sh
@@ -0,0 +1,40 @@
+
+#
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+# This script calls GetPackagesPath.py to collect all package paths under
+# specified directories and append them to PACKAGES_PATH environment
+# variable. A sub directory is a qualified package path when an EDKII
+# Package can be found under it.
+#
+# Note: This script must be \'sourced\' so the environment can be changed:
+# source SetPackagesPath.sh
+# . SetPackagesPath.sh
+
+function Usage()
+{
+    echo "Usage: source SetPackagesPath.sh directory [directory ...]"
+    echo "Copyright(c) 2020, Intel Corporation. All rights reserved."
+    echo "Options:"
+    echo "  --help, -h     Print this help screen and exit"
+    echo "Please note: This script must be \'sourced\' so the environment can be changed."
+    echo ". SetPackagesPath.sh"
+    echo "source SetPackagesPath.sh"
+}
+
+function SetEnv()
+{
+    local paths=$(python $EDK_TOOLS_PATH/Scripts/GetPackagesPath.py $@)
+    if [ "$PACKAGES_PATH" ]; then
+        PACKAGES_PATH=$PACKAGES_PATH:$paths
+    else
+        PACKAGES_PATH=$paths
+    fi
+}
+
+if [ $# -eq 0 -o "$1" == "-h" -o "$1" == "--help" -o "$1" == "/?" ]; then
+    Usage
+else
+    SetEnv $@
+fi
-- 
2.24.0.windows.2

Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[Sean]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

I am not a fan of this.  The community and design meetings have had a few discussions about tools (PyTools has been presented twice) and a few brief discussions about common patterns to build platforms but I don't think there is real alignment.  Each "platform" has its own way of doing things.

As for this patch, I don't really want to see a bunch of scripts added to edk2 basetools that are not aligned with a community agreed direction (or any clear direction).  It has also been brought up that basetools as a python package/project is a challenge to work with and has some fundamental problems.  Adding more in a similar design pattern is not how we start fixing it.   As an example for edk2-pytools, an RFC was offered and even then it was added as its own repositories to avoid more directly to the edk2 repo.

@Mike K - Any progress on getting a tools subtream setup?  Any governance ideas to help align these efforts.

Thanks
Sean

[-- Attachment #2: Type: text/html, Size: 1043 bytes --]

Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[Ni, Ray]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

Sean,
PyTools is good. This patch is to support those platforms that haven’t adopted PyTools for their platform build.

Thanks,
Ray

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean via groups.io
Sent: Thursday, April 2, 2020 11:59 PM
To: Luo, Heng <heng.luo@intel.com>; devel@edk2.groups.io
Subject: Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

I am not a fan of this.  The community and design meetings have had a few discussions about tools (PyTools has been presented twice) and a few brief discussions about common patterns to build platforms but I don't think there is real alignment.  Each "platform" has its own way of doing things.

As for this patch, I don't really want to see a bunch of scripts added to edk2 basetools that are not aligned with a community agreed direction (or any clear direction).  It has also been brought up that basetools as a python package/project is a challenge to work with and has some fundamental problems.  Adding more in a similar design pattern is not how we start fixing it.   As an example for edk2-pytools, an RFC was offered and even then it was added as its own repositories to avoid more directly to the edk2 repo.

@Mike K - Any progress on getting a tools subtream setup?  Any governance ideas to help align these efforts.

Thanks
Sean


[-- Attachment #2: Type: text/html, Size: 4295 bytes --]

Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[Bob Feng]

[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]

Hi Sean,

That the Basetools works correctly depends on the PACAKGES_PATH is set correctly. I think this patch provides common function and provides assistance for the Basetools.

Thanks,
Bob

From: Ni, Ray
Sent: Tuesday, April 7, 2020 10:16 AM
To: devel@edk2.groups.io; sean.brogan@microsoft.com; Luo, Heng <heng.luo@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>
Subject: RE: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

Sean,
PyTools is good. This patch is to support those platforms that haven’t adopted PyTools for their platform build.

Thanks,
Ray

From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Sean via groups.io
Sent: Thursday, April 2, 2020 11:59 PM
To: Luo, Heng <heng.luo@intel.com<mailto:heng.luo@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
Subject: Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

I am not a fan of this.  The community and design meetings have had a few discussions about tools (PyTools has been presented twice) and a few brief discussions about common patterns to build platforms but I don't think there is real alignment.  Each "platform" has its own way of doing things.

As for this patch, I don't really want to see a bunch of scripts added to edk2 basetools that are not aligned with a community agreed direction (or any clear direction).  It has also been brought up that basetools as a python package/project is a challenge to work with and has some fundamental problems.  Adding more in a similar design pattern is not how we start fixing it.   As an example for edk2-pytools, an RFC was offered and even then it was added as its own repositories to avoid more directly to the edk2 repo.

@Mike K - Any progress on getting a tools subtream setup?  Any governance ideas to help align these efforts.

Thanks
Sean


[-- Attachment #2: Type: text/html, Size: 6081 bytes --]

Re: [edk2-devel] [PATCH] BaseTools/Scripts: Add scripts to set PACKAGES_PATH environment

[Sean]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

Bob and Ray,

My point isn't that there is anything wrong with script.  This script is designed to work with edk2-platforms repo or other repos of similar layout.   This is a "platform" thing that is needed on some platforms (not edk2 core platforms).  To me this means it does't belong in edk2 repo.

Anytime new functionality is proposed I think it is important to evaluate the stakeholders of the given repository that is targeted and make sure the functionality is aligned.  In my opinion when edk2 get too prescriptive about how to do things outside of edk2 or tries to prematurely provide functionality for all consumers to use then we often see this overhead added to the tree and low adoption rate.

I would propose to add this to edk2-platforms since that is where this is needed and build up a nice "platform" design pattern.   If it provides great value then talk about moving up in the repository stack.

Finally, I know this is a trivial and small script.  The overhead is minimal but i see this as a barometer for how the community aligns to support all isn't consumers.  Many of us have to maintain forks with significant changes because edk2 carries too many prescriptive ideas.  I want to push Tianocore to have better alignment with different repositories, their functionality and focus.  I want to see downstream consumers spend less time maintaining their fork and shipping code more closely aligned with the open source edk2.

thanks
Sean

[-- Attachment #2: Type: text/html, Size: 1615 bytes --]