[edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support

[edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support

[Ashley E Desimone]

V5 updates the imports in patch 3/3 to reflect the change
in file names resulting from V4

V4 addresses Nate's comments regarding typo of 'maintenance'
and added edkrepo.common.workspace_maintenance to setup.py
in patch 1/3

V3 Addresses Prince's comments regarding typos in patch 2/3

V2 Replaces use of management with maitenance per feedback
V2 Introduces patch 1/1 which removes a circular import between
common_repo_functions.py and manifest_repos_maitenance.py by
adding workspace_maiteneance.py.

Adds initial support for manifest repository maitenance while
retaining support for commands to use the existing edkrepo.cfg
format and manifest repo support functions.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>

Ashley E Desimone (3):
  EdkRepo: Initial commit of workspace_maintenance.py
  EdkRepo: Add
    edkrepo/common/workspace_maintenance/manifest_repos_maintenance
  EdkRepo: Update pull_latest_manifest_repo to use
    pull_single_manifest_repo

 edkrepo/common/common_repo_functions.py            | 33 ++----------
 edkrepo/common/humble.py                           |  8 ---
 .../humble/manifest_repos_maintenance_humble.py    | 23 +++++++++
 .../manifest_repos_maintenance.py                  | 59 ++++++++++++++++++++++
 .../workspace_maintenance/workspace_maintenance.py | 30 +++++++++++
 setup.py                                           |  5 +-
 6 files changed, 119 insertions(+), 39 deletions(-)
 create mode 100644 edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
 create mode 100644 edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
 create mode 100644 edkrepo/common/workspace_maintenance/workspace_maintenance.py

-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V5 1/3] EdkRepo: Initial commit of workspace_maintenance.py

[Ashley E Desimone]

Add workspace_maintenance.py including the implementation
of generate_name_for_obsolete_backup()

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../workspace_maintenance/workspace_maintenance.py | 30 ++++++++++++++++++++++
 setup.py                                           |  4 +--
 2 files changed, 32 insertions(+), 2 deletions(-)
 create mode 100644 edkrepo/common/workspace_maintenance/workspace_maintenance.py

diff --git a/edkrepo/common/workspace_maintenance/workspace_maintenance.py b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
new file mode 100644
index 0000000..d45759a
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+#
+## @file
+# workspace_maintenance.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains shared workspace maintenance functions. '''
+
+import os
+
+def generate_name_for_obsolete_backup(absolute_path):
+    if not os.path.exists(absolute_path):
+        raise ValueError("{} does not exist".format(absolute_path))
+    original_name = os.path.basename(absolute_path)
+    dir_name = os.path.dirname(absolute_path)
+    unique_name = ""
+    unique_name_found = False
+    index = 1
+    while not unique_name_found:
+        if index == 1:
+            unique_name = "{}_old".format(original_name)
+        else:
+            unique_name = "{}_old{}".format(original_name, index)
+        if not os.path.exists(os.path.join(dir_name, unique_name)):
+            unique_name_found = True
+        index += 1
+    return unique_name
\ No newline at end of file
diff --git a/setup.py b/setup.py
index 06a382b..af173bb 100755
--- a/setup.py
+++ b/setup.py
@@ -12,8 +12,8 @@ setup(name='edkrepo',
       version='2.0.0',
       description='The edkrepo tools',
       packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
-                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.config', 'edkrepo.config.humble',
-                'edkrepo_manifest_parser', 'project_utils'],
+                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maintenance',
+                'edkrepo.config', 'edkrepo.config.humble', 'edkrepo_manifest_parser', 'project_utils'],
       package_data={
          },
       include_package_data=True,
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V5 2/3] EdkRepo: Add edkrepo/common/workspace_maintenance/manifest_repos_maintenance

[Ashley E Desimone]

Add a directory to edkrepo/common to store workspace maintenance
functionatlity.
Add edkrepo/common/workspace_maintenance/humble directory to store
related strings and included manifest_repos_maintenance_humble.py
Add edkrepo/common/workspace_maintenance/manifest_repos_maitenance.py
to support the mmaintenance of multiple manifest repositories.
Added pull_single_manifest_repo() to manifest_repos_maintenance.py
Updated setup.py to include the new directories.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../humble/manifest_repos_maintenance_humble.py    | 23 +++++++++
 .../manifest_repos_maintenance.py                  | 59 ++++++++++++++++++++++
 setup.py                                           |  3 +-
 3 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
 create mode 100644 edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py

diff --git a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
new file mode 100644
index 0000000..440fd8a
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_maintenance_humble.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains user facing strings for manifest_repos_mgmt.py '''
+
+from colorama import Fore
+from colorama import Style
+
+CLONE_SINGLE_MAN_REPO = 'Cloning global manifest repository to: {} from: {}'
+SYNC_SINGLE_MAN_REPO = 'Syncing the global manifest repository: {}'
+SINGLE_MAN_REPO_DIRTY = ('Uncommited changes present in the global manifest '
+                         'repository: {} Resolve these changes and attempt your'
+                         ' operation again.')
+SINGLE_MAN_REPO_NOT_CFG_BRANCH = ('The current active branch, {}, is not the '
+                                  'specified branch for global manifst repository: {}')
+SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH = 'Checking out the specified branch: {} prior to syncing'
+SINGLE_MAN_REPO_MOVED = '{}{}WARNING:{}{} The global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
new file mode 100644
index 0000000..6e26d4f
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_maintenance.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import os
+import traceback
+import shutil
+
+import git
+from git import Repo
+
+import edkrepo.config.config_factory as cfg
+from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException
+from edkrepo.common.progress_handler import GitProgressHandler
+import edkrepo.common.workspace_maintenance.humble.manifest_repos_maintenance_humble as humble
+from edkrepo.common.workspace_maintenance.workspace_maintenance import generate_name_for_obsolete_backup
+
+
+def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
+    '''
+    Clones or syncs a single global manifest repository as defined in either
+    the edkrepo.cfg or the edkrepo_user.cfg
+    '''
+    # If a relative path is used join to the edkrepo global data directory path
+    if not os.path.isabs(local_path):
+        local_path = os.path.join(cfg.get_edkrepo_global_data_directory(), local_path)
+    # Clone the repository if it does not exist locally
+    if not os.path.exists(local_path):
+        print(humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+        repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+    # Sync the repository if it exists locally
+    else:
+        repo = Repo(local_path)
+        if url in repo.remotes['origin'].urls:
+            if repo.is_dirty(untracked_files=True) and not reset_hard:
+                raise EdkrepoUncommitedChangesException(humble.SINGLE_MAN_REPO_DIRTY.format(local_path))
+            elif repo.is_dirty(untracked_files=True) and reset_hard:
+                repo.git.reset('--hard')
+            print(humble.SYNC_SINGLE_MAN_REPO.format(local_path))
+            if repo.active_branch.name != branch:
+                print(humble.SINGLE_MAN_REPO_NOT_CFG_BRANCH.format(repo.active_branch.name, local_path))
+                print(humble.SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH.format(branch))
+                repo.git.checkout(branch)
+            repo.remotes.origin.pull()
+        # If the URL specified for this manifest repo has moved back up the existing
+        # local copy and clone the new repository
+        else:
+            new_path = generate_name_for_obsolete_backup(local_path)
+            new_path = os.path.join(os.path.dirname(local_path), new_path)
+            print(humble.SINGLE_MAN_REPO_MOVED.format(new_path))
+            shutil.move(local_path, new_path)
+            print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+            repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+
diff --git a/setup.py b/setup.py
index af173bb..5053b1a 100755
--- a/setup.py
+++ b/setup.py
@@ -13,7 +13,8 @@ setup(name='edkrepo',
       description='The edkrepo tools',
       packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
                 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maintenance',
-                'edkrepo.config', 'edkrepo.config.humble', 'edkrepo_manifest_parser', 'project_utils'],
+                'edkrepo.common.workspace_maintenance.humble', 'edkrepo.config', 'edkrepo.config.humble',
+                'edkrepo_manifest_parser', 'project_utils'],
       package_data={
          },
       include_package_data=True,
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V5 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo

[Ashley E Desimone]

Updates the implementation for pull_latest_manifest_repo to
call pull_single_manifest repo.
Removes definitions of strings used by pull_latest_manifest_repo
from common/humble.py and from the imports of
common_repo_functions.py

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 edkrepo/common/common_repo_functions.py | 33 ++++-----------------------------
 edkrepo/common/humble.py                |  8 --------
 2 files changed, 4 insertions(+), 37 deletions(-)

diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
index 0d39291..2aa5f5f 100644
--- a/edkrepo/common/common_repo_functions.py
+++ b/edkrepo/common/common_repo_functions.py
@@ -46,12 +46,10 @@ from edkrepo.common.humble import COMMIT_TEMPLATE_NOT_FOUND, COMMIT_TEMPLATE_CUS
 from edkrepo.common.humble import ADD_PRIMARY_REMOTE, REMOVE_PRIMARY_REMOTE
 from edkrepo.common.humble import FETCH_PRIMARY_REMOTE, MIRROR_PRIMARY_SHA, TAG_AND_BRANCH_SPECIFIED
 from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_REPO, HOOK_NOT_FOUND_ERROR, SUBMODULE_FAILURE
-from edkrepo.common.humble import MANIFEST_REPO_DIRTY, MANIFEST_REPO_MOVED, CLONING_MANIFEST_REPO, SYNCING_MANIFEST_REPO
 from edkrepo.common.humble import INCLUDED_URL_LINE, INCLUDED_INSTEAD_OF_LINE, INCLUDED_FILE_NAME
 from edkrepo.common.humble import ERROR_WRITING_INCLUDE, MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED
 from edkrepo.common.humble import VERIFY_GLOBAL, VERIFY_ARCHIVED, VERIFY_PROJ, VERIFY_PROJ_FAIL
-from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL, MANIFEST_REPO_NOT_CONFIG_BRANCH
-from edkrepo.common.humble import MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH
+from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL
 from edkrepo.common.pathfix import get_actual_path
 from project_utils.sparse import BuildInfo, process_sparse_checkout
 from edkrepo.config.config_factory import get_workspace_path
@@ -61,6 +59,7 @@ from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml
 from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException, EdkrepoGitException, EdkrepoWarningException
 from edkrepo.common.edkrepo_exception import EdkrepoFoundMultipleException, EdkrepoHookNotFoundException
 from edkrepo.common.edkrepo_exception import EdkrepoGitConfigSetupException, EdkrepoManifestInvalidException
+from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import pull_single_manifest_repo
 from edkrepo.common.ui_functions import init_color_console
 from edkrepo_manifest_parser import edk_manifest
 from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrepo
@@ -77,32 +76,8 @@ def pull_latest_manifest_repo(args, config, reset_hard=False):
     branch = config['cfg_file'].manifest_repo_branch
     local_path = config['cfg_file'].manifest_repo_local_path
     init_color_console(False)
-    if not (os.path.isabs(local_path)):
-        #since only a relative path was specified it must be joined to the Edkrepo Application Data Directory
-        local_path = os.path.join(get_edkrepo_global_data_directory(), local_path)
-    if not os.path.exists(local_path):
-        print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
-        repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
-    else:
-        repo = Repo(local_path)
-        if repo_url in repo.remotes['origin'].urls:
-            if repo.is_dirty(untracked_files=True) and not reset_hard:
-                raise EdkrepoWarningException(MANIFEST_REPO_DIRTY)
-            elif repo.is_dirty(untracked_files=True) and reset_hard:
-                repo.git.reset('--hard')
-            print (SYNCING_MANIFEST_REPO)
-            if repo.active_branch.name != branch:
-                print(MANIFEST_REPO_NOT_CONFIG_BRANCH.format(repo.active_branch.name))
-                print(MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH.format(branch))
-                repo.git.checkout(branch)
-            repo.remotes.origin.pull()
-        else:
-            new_path = generate_name_for_obsolete_backup(local_path)
-            new_path = os.path.join(os.path.dirname(local_path), new_path)
-            print(MANIFEST_REPO_MOVED.format(new_path))
-            shutil.move(local_path, new_path)
-            print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
-            repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
+    pull_single_manifest_repo(repo_url, branch, local_path, reset_hard)
+
 
 def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_hooks, config, skip_submodule, manifest):
     for repo_to_clone in repos_to_clone:
diff --git a/edkrepo/common/humble.py b/edkrepo/common/humble.py
index 64b9519..8ca38bb 100644
--- a/edkrepo/common/humble.py
+++ b/edkrepo/common/humble.py
@@ -34,14 +34,6 @@ MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED = 'BRANCH or TAG name present with COMMIT I
 TAG_AND_BRANCH_SPECIFIED = 'BRANCH AND TAG name present in combination field for {} repo. Using TAG.\n'
 CHECKING_CONNECTION = 'Checking connection to remote url: {}\n'
 
-# Informational messages and warnings for pull_latest_manifest_repo()
-MANIFEST_REPO_DIRTY = 'Uncommited changes present in the global manifest repository. Run edkrepo update-manifest-repo --hard to revert these changes and sync the global manifest repository.\n'
-MANIFEST_REPO_MOVED = '{}{}WARNING:{}{} The Global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
-CLONING_MANIFEST_REPO = 'Cloning global manifest repository to: {} from: {}\n'
-SYNCING_MANIFEST_REPO = 'Syncing the global manifest repository.\n'
-MANIFEST_REPO_NOT_CONFIG_BRANCH = 'The current active branch, {}, is not the specified manifest repository branch'
-MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH = 'Checking out the specified manifest repository branch, {}, prior to syncing'
-
 #Error messages for sync_command.py
 SYNC_EXIT = 'Exiting without performing sync operations.'
 SYNC_UNCOMMITED_CHANGES = UNCOMMITED_CHANGES + SYNC_EXIT
-- 
2.16.2.windows.1

Re: [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support

[Nate DeSimone]

For the patch series...

Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>

-----Original Message-----
From: Desimone, Ashley E <ashley.e.desimone@intel.com> 
Sent: Thursday, April 23, 2020 4:10 PM
To: devel@edk2.groups.io
Cc: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bjorge, Erik C <erik.c.bjorge@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support

V5 updates the imports in patch 3/3 to reflect the change in file names resulting from V4

V4 addresses Nate's comments regarding typo of 'maintenance'
and added edkrepo.common.workspace_maintenance to setup.py in patch 1/3

V3 Addresses Prince's comments regarding typos in patch 2/3

V2 Replaces use of management with maitenance per feedback
V2 Introduces patch 1/1 which removes a circular import between common_repo_functions.py and manifest_repos_maitenance.py by adding workspace_maiteneance.py.

Adds initial support for manifest repository maitenance while retaining support for commands to use the existing edkrepo.cfg format and manifest repo support functions.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>

Ashley E Desimone (3):
  EdkRepo: Initial commit of workspace_maintenance.py
  EdkRepo: Add
    edkrepo/common/workspace_maintenance/manifest_repos_maintenance
  EdkRepo: Update pull_latest_manifest_repo to use
    pull_single_manifest_repo

 edkrepo/common/common_repo_functions.py            | 33 ++----------
 edkrepo/common/humble.py                           |  8 ---
 .../humble/manifest_repos_maintenance_humble.py    | 23 +++++++++
 .../manifest_repos_maintenance.py                  | 59 ++++++++++++++++++++++
 .../workspace_maintenance/workspace_maintenance.py | 30 +++++++++++
 setup.py                                           |  5 +-
 6 files changed, 119 insertions(+), 39 deletions(-)  create mode 100644 edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
 create mode 100644 edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
 create mode 100644 edkrepo/common/workspace_maintenance/workspace_maintenance.py

--
2.16.2.windows.1