[edk2-staging/EdkRepo] [PATCH V3 0/3] Add initial manifest repository support

[edk2-staging/EdkRepo] [PATCH V3 0/3] Add initial manifest repository support

[Ashley E Desimone]

V3 Addresses Prince's comments regarding typos in patch 2/3

V2 Replaces use of management with maitenance per feedback
V2 Introduces patch 1/1 which removes a circular import between
common_repo_functions.py and manifest_repos_maitenance.py by
adding workspace_maiteneance.py.

Adds initial support for manifest repository maitenance while
retaining support for commands to use the existing edkrepo.cfg
format and manifest repo support functions.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>

Ashley E Desimone (3):
  EdkRepo: Initial commit of workspace_maitenance.py
  EdkRepo: Add
    edkrepo/common/workspace_maitenance/manifest_repos_maitenance
  EdkRepo: Update pull_latest_manifest_repo to use
    pull_single_manifest_repo

 edkrepo/common/common_repo_functions.py            | 33 ++----------
 edkrepo/common/humble.py                           |  8 ---
 .../humble/manifest_repos_maitenance_humble.py     | 23 +++++++++
 .../manifest_repos_maitenance.py                   | 59 ++++++++++++++++++++++
 .../workspace_maitenance/workspace_maitenance.py   | 30 +++++++++++
 setup.py                                           |  5 +-
 6 files changed, 119 insertions(+), 39 deletions(-)
 create mode 100644 edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
 create mode 100644 edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
 create mode 100644 edkrepo/common/workspace_maitenance/workspace_maitenance.py

-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V3 1/3] EdkRepo: Initial commit of workspace_maitenance.py

[Ashley E Desimone]

Add workspace_maitenance.py including the implementation
of generate_name_for_obsolete_backup()

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../workspace_maitenance/workspace_maitenance.py   | 30 ++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 edkrepo/common/workspace_maitenance/workspace_maitenance.py

diff --git a/edkrepo/common/workspace_maitenance/workspace_maitenance.py b/edkrepo/common/workspace_maitenance/workspace_maitenance.py
new file mode 100644
index 0000000..83204f8
--- /dev/null
+++ b/edkrepo/common/workspace_maitenance/workspace_maitenance.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+#
+## @file
+# workspace_maitenance.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains shared workspace maitenance functions. '''
+
+import os
+
+def generate_name_for_obsolete_backup(absolute_path):
+    if not os.path.exists(absolute_path):
+        raise ValueError("{} does not exist".format(absolute_path))
+    original_name = os.path.basename(absolute_path)
+    dir_name = os.path.dirname(absolute_path)
+    unique_name = ""
+    unique_name_found = False
+    index = 1
+    while not unique_name_found:
+        if index == 1:
+            unique_name = "{}_old".format(original_name)
+        else:
+            unique_name = "{}_old{}".format(original_name, index)
+        if not os.path.exists(os.path.join(dir_name, unique_name)):
+            unique_name_found = True
+        index += 1
+    return unique_name
\ No newline at end of file
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V3 2/3] EdkRepo: Add edkrepo/common/workspace_maitenance/manifest_repos_maitenance

[Ashley E Desimone]

Add a directory to edkrepo/common to store workspace maitenance
functionatlity.
Add edkrepo/common/workspace_matenance/humble directory to store
related strings and included manifest_repos_maitenance_humble.py
Add edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
to support the maitenance of multiple manifest repositories.
Added pull_single_manifest_repo() to manifest_repos_maitenance.py
Updated setup.py to include the new directories.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../humble/manifest_repos_maitenance_humble.py     | 23 +++++++++
 .../manifest_repos_maitenance.py                   | 59 ++++++++++++++++++++++
 setup.py                                           |  5 +-
 3 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100644 edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
 create mode 100644 edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py

diff --git a/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py b/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
new file mode 100644
index 0000000..ee7b4bb
--- /dev/null
+++ b/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_mgmt_humble.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains user facing strings for manifest_repos_mgmt.py '''
+
+from colorama import Fore
+from colorama import Style
+
+CLONE_SINGLE_MAN_REPO = 'Cloning global manifest repository to: {} from: {}'
+SYNC_SINGLE_MAN_REPO = 'Syncing the global manifest repository: {}'
+SINGLE_MAN_REPO_DIRTY = ('Uncommited changes present in the global manifest '
+                         'repository: {} Resolve these changes and attempt your'
+                         ' operation again.')
+SINGLE_MAN_REPO_NOT_CFG_BRANCH = ('The current active branch, {}, is not the '
+                                  'specified branch for global manifst repository: {}')
+SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH = 'Checking out the specified branch: {} prior to syncing'
+SINGLE_MAN_REPO_MOVED = '{}{}WARNING:{}{} The global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
\ No newline at end of file
diff --git a/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py b/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
new file mode 100644
index 0000000..c70a413
--- /dev/null
+++ b/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_mgmt.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import os
+import traceback
+import shutil
+
+import git
+from git import Repo
+
+import edkrepo.config.config_factory as cfg
+from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException
+from edkrepo.common.progress_handler import GitProgressHandler
+import edkrepo.common.workspace_maitenance.humble.manifest_repos_maitenance_humble as humble
+from edkrepo.common.workspace_maitenance.workspace_maitenance import generate_name_for_obsolete_backup
+
+
+def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
+    '''
+    Clones or syncs a single global manifest repository as defined in either
+    the edkrepo.cfg or the edkrepo_user.cfg
+    '''
+    # If a relative path is used join to the edkrepo global data directory path
+    if not os.path.isabs(local_path):
+        local_path = os.path.join(cfg.get_edkrepo_global_data_directory(), local_path)
+    # Clone the repository if it does not exist locally
+    if not os.path.exists(local_path):
+        print(humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+        repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+    # Sync the repository if it exists locally
+    else:
+        repo = Repo(local_path)
+        if url in repo.remotes['origin'].urls:
+            if repo.is_dirty(untracked_files=True) and not reset_hard:
+                raise EdkrepoUncommitedChangesException(humble.SINGLE_MAN_REPO_DIRTY.format(local_path))
+            elif repo.is_dirty(untracked_files=True) and reset_hard:
+                repo.git.reset('--hard')
+            print(humble.SYNC_SINGLE_MAN_REPO.format(local_path))
+            if repo.active_branch.name != branch:
+                print(humble.SINGLE_MAN_REPO_NOT_CFG_BRANCH.format(repo.active_branch.name, local_path))
+                print(humble.SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH.format(branch))
+                repo.git.checkout(branch)
+            repo.remotes.origin.pull()
+        # If the URL specified for this manifest repo has moved back up the existing
+        # local copy and clone the new repository
+        else:
+            new_path = generate_name_for_obsolete_backup(local_path)
+            new_path = os.path.join(os.path.dirname(local_path), new_path)
+            print(humble.SINGLE_MAN_REPO_MOVED.format(new_path))
+            shutil.move(local_path, new_path)
+            print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+            repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+
diff --git a/setup.py b/setup.py
index 06a382b..f8bb0fc 100755
--- a/setup.py
+++ b/setup.py
@@ -12,8 +12,9 @@ setup(name='edkrepo',
       version='2.0.0',
       description='The edkrepo tools',
       packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
-                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.config', 'edkrepo.config.humble',
-                'edkrepo_manifest_parser', 'project_utils'],
+                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maitenance',
+                'edkrepo.common.workspace_maitenance.humble', 'edkrepo.config', 'edkrepo.config.humble',
+                'edkrepo_manifest_parser', 'project_utils'],
       package_data={
          },
       include_package_data=True,
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V3 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo

[Ashley E Desimone]

Updates the implementation for pull_latest_manifest_repo to
call pull_single_manifest repo.
Removes definitions of strings used by pull_latest_manifest_repo
from common/humble.py and from the imports of
common_repo_functions.py

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 edkrepo/common/common_repo_functions.py | 33 ++++-----------------------------
 edkrepo/common/humble.py                |  8 --------
 2 files changed, 4 insertions(+), 37 deletions(-)

diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
index 0d39291..39b192c 100644
--- a/edkrepo/common/common_repo_functions.py
+++ b/edkrepo/common/common_repo_functions.py
@@ -46,12 +46,10 @@ from edkrepo.common.humble import COMMIT_TEMPLATE_NOT_FOUND, COMMIT_TEMPLATE_CUS
 from edkrepo.common.humble import ADD_PRIMARY_REMOTE, REMOVE_PRIMARY_REMOTE
 from edkrepo.common.humble import FETCH_PRIMARY_REMOTE, MIRROR_PRIMARY_SHA, TAG_AND_BRANCH_SPECIFIED
 from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_REPO, HOOK_NOT_FOUND_ERROR, SUBMODULE_FAILURE
-from edkrepo.common.humble import MANIFEST_REPO_DIRTY, MANIFEST_REPO_MOVED, CLONING_MANIFEST_REPO, SYNCING_MANIFEST_REPO
 from edkrepo.common.humble import INCLUDED_URL_LINE, INCLUDED_INSTEAD_OF_LINE, INCLUDED_FILE_NAME
 from edkrepo.common.humble import ERROR_WRITING_INCLUDE, MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED
 from edkrepo.common.humble import VERIFY_GLOBAL, VERIFY_ARCHIVED, VERIFY_PROJ, VERIFY_PROJ_FAIL
-from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL, MANIFEST_REPO_NOT_CONFIG_BRANCH
-from edkrepo.common.humble import MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH
+from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL
 from edkrepo.common.pathfix import get_actual_path
 from project_utils.sparse import BuildInfo, process_sparse_checkout
 from edkrepo.config.config_factory import get_workspace_path
@@ -61,6 +59,7 @@ from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml
 from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException, EdkrepoGitException, EdkrepoWarningException
 from edkrepo.common.edkrepo_exception import EdkrepoFoundMultipleException, EdkrepoHookNotFoundException
 from edkrepo.common.edkrepo_exception import EdkrepoGitConfigSetupException, EdkrepoManifestInvalidException
+from edkrepo.common.workspace_maitenance.manifest_repos_maitenance import pull_single_manifest_repo
 from edkrepo.common.ui_functions import init_color_console
 from edkrepo_manifest_parser import edk_manifest
 from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrepo
@@ -77,32 +76,8 @@ def pull_latest_manifest_repo(args, config, reset_hard=False):
     branch = config['cfg_file'].manifest_repo_branch
     local_path = config['cfg_file'].manifest_repo_local_path
     init_color_console(False)
-    if not (os.path.isabs(local_path)):
-        #since only a relative path was specified it must be joined to the Edkrepo Application Data Directory
-        local_path = os.path.join(get_edkrepo_global_data_directory(), local_path)
-    if not os.path.exists(local_path):
-        print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
-        repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
-    else:
-        repo = Repo(local_path)
-        if repo_url in repo.remotes['origin'].urls:
-            if repo.is_dirty(untracked_files=True) and not reset_hard:
-                raise EdkrepoWarningException(MANIFEST_REPO_DIRTY)
-            elif repo.is_dirty(untracked_files=True) and reset_hard:
-                repo.git.reset('--hard')
-            print (SYNCING_MANIFEST_REPO)
-            if repo.active_branch.name != branch:
-                print(MANIFEST_REPO_NOT_CONFIG_BRANCH.format(repo.active_branch.name))
-                print(MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH.format(branch))
-                repo.git.checkout(branch)
-            repo.remotes.origin.pull()
-        else:
-            new_path = generate_name_for_obsolete_backup(local_path)
-            new_path = os.path.join(os.path.dirname(local_path), new_path)
-            print(MANIFEST_REPO_MOVED.format(new_path))
-            shutil.move(local_path, new_path)
-            print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
-            repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
+    pull_single_manifest_repo(repo_url, branch, local_path, reset_hard)
+
 
 def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_hooks, config, skip_submodule, manifest):
     for repo_to_clone in repos_to_clone:
diff --git a/edkrepo/common/humble.py b/edkrepo/common/humble.py
index 64b9519..8ca38bb 100644
--- a/edkrepo/common/humble.py
+++ b/edkrepo/common/humble.py
@@ -34,14 +34,6 @@ MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED = 'BRANCH or TAG name present with COMMIT I
 TAG_AND_BRANCH_SPECIFIED = 'BRANCH AND TAG name present in combination field for {} repo. Using TAG.\n'
 CHECKING_CONNECTION = 'Checking connection to remote url: {}\n'
 
-# Informational messages and warnings for pull_latest_manifest_repo()
-MANIFEST_REPO_DIRTY = 'Uncommited changes present in the global manifest repository. Run edkrepo update-manifest-repo --hard to revert these changes and sync the global manifest repository.\n'
-MANIFEST_REPO_MOVED = '{}{}WARNING:{}{} The Global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
-CLONING_MANIFEST_REPO = 'Cloning global manifest repository to: {} from: {}\n'
-SYNCING_MANIFEST_REPO = 'Syncing the global manifest repository.\n'
-MANIFEST_REPO_NOT_CONFIG_BRANCH = 'The current active branch, {}, is not the specified manifest repository branch'
-MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH = 'Checking out the specified manifest repository branch, {}, prior to syncing'
-
 #Error messages for sync_command.py
 SYNC_EXIT = 'Exiting without performing sync operations.'
 SYNC_UNCOMMITED_CHANGES = UNCOMMITED_CHANGES + SYNC_EXIT
-- 
2.16.2.windows.1

Re: [edk2-staging/EdkRepo] [PATCH V3 1/3] EdkRepo: Initial commit of workspace_maitenance.py

[Nate DeSimone]

Hi Ashley,

There is a misspelling, "maitenance" should be "maintenance". Other than that, looks good.

Thanks,
Nate

On 4/22/20, 4:29 PM, "Desimone, Ashley E" <ashley.e.desimone@intel.com> wrote:

    Add workspace_maitenance.py including the implementation
    of generate_name_for_obsolete_backup()

    Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
    Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
    Cc: Puja Pandya <puja.pandya@intel.com>
    Cc: Erik Bjorge <erik.c.bjorge@intel.com>
    Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
    Cc: Prince Agyeman <prince.agyeman@intel.com>
    ---
     .../workspace_maitenance/workspace_maitenance.py   | 30 ++++++++++++++++++++++
     1 file changed, 30 insertions(+)
     create mode 100644 edkrepo/common/workspace_maitenance/workspace_maitenance.py

    diff --git a/edkrepo/common/workspace_maitenance/workspace_maitenance.py b/edkrepo/common/workspace_maitenance/workspace_maitenance.py
    new file mode 100644
    index 0000000..83204f8
    --- /dev/null
    +++ b/edkrepo/common/workspace_maitenance/workspace_maitenance.py
    @@ -0,0 +1,30 @@
    +#!/usr/bin/env python3
    +#
    +## @file
    +# workspace_maitenance.py
    +#
    +# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
    +# SPDX-License-Identifier: BSD-2-Clause-Patent
    +#
    +
    +''' Contains shared workspace maitenance functions. '''
    +
    +import os
    +
    +def generate_name_for_obsolete_backup(absolute_path):
    +    if not os.path.exists(absolute_path):
    +        raise ValueError("{} does not exist".format(absolute_path))
    +    original_name = os.path.basename(absolute_path)
    +    dir_name = os.path.dirname(absolute_path)
    +    unique_name = ""
    +    unique_name_found = False
    +    index = 1
    +    while not unique_name_found:
    +        if index == 1:
    +            unique_name = "{}_old".format(original_name)
    +        else:
    +            unique_name = "{}_old{}".format(original_name, index)
    +        if not os.path.exists(os.path.join(dir_name, unique_name)):
    +            unique_name_found = True
    +        index += 1
    +    return unique_name
    \ No newline at end of file
    -- 
    2.16.2.windows.1

Re: [edk2-staging/EdkRepo] [PATCH V3 2/3] EdkRepo: Add edkrepo/common/workspace_maitenance/manifest_repos_maitenance

[Nate DeSimone]

Hi Ashley,

There is a misspelling, "maitenance" should be "maintenance". Also, you need to update the filename at the top of manifest_repos_maitenance_humble.py Other than that, looks good.

Thanks,
Nate

On 4/22/20, 4:29 PM, "Desimone, Ashley E" <ashley.e.desimone@intel.com> wrote:

    Add a directory to edkrepo/common to store workspace maitenance
    functionatlity.
    Add edkrepo/common/workspace_matenance/humble directory to store
    related strings and included manifest_repos_maitenance_humble.py
    Add edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
    to support the maitenance of multiple manifest repositories.
    Added pull_single_manifest_repo() to manifest_repos_maitenance.py
    Updated setup.py to include the new directories.

    Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
    Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
    Cc: Puja Pandya <puja.pandya@intel.com>
    Cc: Erik Bjorge <erik.c.bjorge@intel.com>
    Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
    Cc: Prince Agyeman <prince.agyeman@intel.com>
    ---
     .../humble/manifest_repos_maitenance_humble.py     | 23 +++++++++
     .../manifest_repos_maitenance.py                   | 59 ++++++++++++++++++++++
     setup.py                                           |  5 +-
     3 files changed, 85 insertions(+), 2 deletions(-)
     create mode 100644 edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
     create mode 100644 edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py

    diff --git a/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py b/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
    new file mode 100644
    index 0000000..ee7b4bb
    --- /dev/null
    +++ b/edkrepo/common/workspace_maitenance/humble/manifest_repos_maitenance_humble.py
    @@ -0,0 +1,23 @@
    +#!/usr/bin/env python3
    +#
    +## @file
    +# manifest_repos_mgmt_humble.py
    +#
    +# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
    +# SPDX-License-Identifier: BSD-2-Clause-Patent
    +#
    +
    +''' Contains user facing strings for manifest_repos_mgmt.py '''
    +
    +from colorama import Fore
    +from colorama import Style
    +
    +CLONE_SINGLE_MAN_REPO = 'Cloning global manifest repository to: {} from: {}'
    +SYNC_SINGLE_MAN_REPO = 'Syncing the global manifest repository: {}'
    +SINGLE_MAN_REPO_DIRTY = ('Uncommited changes present in the global manifest '
    +                         'repository: {} Resolve these changes and attempt your'
    +                         ' operation again.')
    +SINGLE_MAN_REPO_NOT_CFG_BRANCH = ('The current active branch, {}, is not the '
    +                                  'specified branch for global manifst repository: {}')
    +SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH = 'Checking out the specified branch: {} prior to syncing'
    +SINGLE_MAN_REPO_MOVED = '{}{}WARNING:{}{} The global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
    \ No newline at end of file
    diff --git a/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py b/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
    new file mode 100644
    index 0000000..c70a413
    --- /dev/null
    +++ b/edkrepo/common/workspace_maitenance/manifest_repos_maitenance.py
    @@ -0,0 +1,59 @@
    +#!/usr/bin/env python3
    +#
    +## @file
    +# manifest_repos_mgmt.py
    +#
    +# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
    +# SPDX-License-Identifier: BSD-2-Clause-Patent
    +#
    +
    +import os
    +import traceback
    +import shutil
    +
    +import git
    +from git import Repo
    +
    +import edkrepo.config.config_factory as cfg
    +from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException
    +from edkrepo.common.progress_handler import GitProgressHandler
    +import edkrepo.common.workspace_maitenance.humble.manifest_repos_maitenance_humble as humble
    +from edkrepo.common.workspace_maitenance.workspace_maitenance import generate_name_for_obsolete_backup
    +
    +
    +def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
    +    '''
    +    Clones or syncs a single global manifest repository as defined in either
    +    the edkrepo.cfg or the edkrepo_user.cfg
    +    '''
    +    # If a relative path is used join to the edkrepo global data directory path
    +    if not os.path.isabs(local_path):
    +        local_path = os.path.join(cfg.get_edkrepo_global_data_directory(), local_path)
    +    # Clone the repository if it does not exist locally
    +    if not os.path.exists(local_path):
    +        print(humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
    +        repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
    +    # Sync the repository if it exists locally
    +    else:
    +        repo = Repo(local_path)
    +        if url in repo.remotes['origin'].urls:
    +            if repo.is_dirty(untracked_files=True) and not reset_hard:
    +                raise EdkrepoUncommitedChangesException(humble.SINGLE_MAN_REPO_DIRTY.format(local_path))
    +            elif repo.is_dirty(untracked_files=True) and reset_hard:
    +                repo.git.reset('--hard')
    +            print(humble.SYNC_SINGLE_MAN_REPO.format(local_path))
    +            if repo.active_branch.name != branch:
    +                print(humble.SINGLE_MAN_REPO_NOT_CFG_BRANCH.format(repo.active_branch.name, local_path))
    +                print(humble.SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH.format(branch))
    +                repo.git.checkout(branch)
    +            repo.remotes.origin.pull()
    +        # If the URL specified for this manifest repo has moved back up the existing
    +        # local copy and clone the new repository
    +        else:
    +            new_path = generate_name_for_obsolete_backup(local_path)
    +            new_path = os.path.join(os.path.dirname(local_path), new_path)
    +            print(humble.SINGLE_MAN_REPO_MOVED.format(new_path))
    +            shutil.move(local_path, new_path)
    +            print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
    +            repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
    +
    diff --git a/setup.py b/setup.py
    index 06a382b..f8bb0fc 100755
    --- a/setup.py
    +++ b/setup.py
    @@ -12,8 +12,9 @@ setup(name='edkrepo',
           version='2.0.0',
           description='The edkrepo tools',
           packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
    -                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.config', 'edkrepo.config.humble',
    -                'edkrepo_manifest_parser', 'project_utils'],
    +                'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maitenance',
    +                'edkrepo.common.workspace_maitenance.humble', 'edkrepo.config', 'edkrepo.config.humble',
    +                'edkrepo_manifest_parser', 'project_utils'],
           package_data={
              },
           include_package_data=True,
    -- 
    2.16.2.windows.1

Re: [edk2-staging/EdkRepo] [PATCH V3 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo

[Nate DeSimone]

Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>

On 4/22/20, 4:29 PM, "Desimone, Ashley E" <ashley.e.desimone@intel.com> wrote:

    Updates the implementation for pull_latest_manifest_repo to
    call pull_single_manifest repo.
    Removes definitions of strings used by pull_latest_manifest_repo
    from common/humble.py and from the imports of
    common_repo_functions.py

    Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
    Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
    Cc: Puja Pandya <puja.pandya@intel.com>
    Cc: Erik Bjorge <erik.c.bjorge@intel.com>
    Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
    Cc: Prince Agyeman <prince.agyeman@intel.com>
    ---
     edkrepo/common/common_repo_functions.py | 33 ++++-----------------------------
     edkrepo/common/humble.py                |  8 --------
     2 files changed, 4 insertions(+), 37 deletions(-)

    diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
    index 0d39291..39b192c 100644
    --- a/edkrepo/common/common_repo_functions.py
    +++ b/edkrepo/common/common_repo_functions.py
    @@ -46,12 +46,10 @@ from edkrepo.common.humble import COMMIT_TEMPLATE_NOT_FOUND, COMMIT_TEMPLATE_CUS
     from edkrepo.common.humble import ADD_PRIMARY_REMOTE, REMOVE_PRIMARY_REMOTE
     from edkrepo.common.humble import FETCH_PRIMARY_REMOTE, MIRROR_PRIMARY_SHA, TAG_AND_BRANCH_SPECIFIED
     from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_REPO, HOOK_NOT_FOUND_ERROR, SUBMODULE_FAILURE
    -from edkrepo.common.humble import MANIFEST_REPO_DIRTY, MANIFEST_REPO_MOVED, CLONING_MANIFEST_REPO, SYNCING_MANIFEST_REPO
     from edkrepo.common.humble import INCLUDED_URL_LINE, INCLUDED_INSTEAD_OF_LINE, INCLUDED_FILE_NAME
     from edkrepo.common.humble import ERROR_WRITING_INCLUDE, MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED
     from edkrepo.common.humble import VERIFY_GLOBAL, VERIFY_ARCHIVED, VERIFY_PROJ, VERIFY_PROJ_FAIL
    -from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL, MANIFEST_REPO_NOT_CONFIG_BRANCH
    -from edkrepo.common.humble import MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH
    +from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL
     from edkrepo.common.pathfix import get_actual_path
     from project_utils.sparse import BuildInfo, process_sparse_checkout
     from edkrepo.config.config_factory import get_workspace_path
    @@ -61,6 +59,7 @@ from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml
     from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException, EdkrepoGitException, EdkrepoWarningException
     from edkrepo.common.edkrepo_exception import EdkrepoFoundMultipleException, EdkrepoHookNotFoundException
     from edkrepo.common.edkrepo_exception import EdkrepoGitConfigSetupException, EdkrepoManifestInvalidException
    +from edkrepo.common.workspace_maitenance.manifest_repos_maitenance import pull_single_manifest_repo
     from edkrepo.common.ui_functions import init_color_console
     from edkrepo_manifest_parser import edk_manifest
     from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrepo
    @@ -77,32 +76,8 @@ def pull_latest_manifest_repo(args, config, reset_hard=False):
         branch = config['cfg_file'].manifest_repo_branch
         local_path = config['cfg_file'].manifest_repo_local_path
         init_color_console(False)
    -    if not (os.path.isabs(local_path)):
    -        #since only a relative path was specified it must be joined to the Edkrepo Application Data Directory
    -        local_path = os.path.join(get_edkrepo_global_data_directory(), local_path)
    -    if not os.path.exists(local_path):
    -        print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
    -        repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
    -    else:
    -        repo = Repo(local_path)
    -        if repo_url in repo.remotes['origin'].urls:
    -            if repo.is_dirty(untracked_files=True) and not reset_hard:
    -                raise EdkrepoWarningException(MANIFEST_REPO_DIRTY)
    -            elif repo.is_dirty(untracked_files=True) and reset_hard:
    -                repo.git.reset('--hard')
    -            print (SYNCING_MANIFEST_REPO)
    -            if repo.active_branch.name != branch:
    -                print(MANIFEST_REPO_NOT_CONFIG_BRANCH.format(repo.active_branch.name))
    -                print(MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH.format(branch))
    -                repo.git.checkout(branch)
    -            repo.remotes.origin.pull()
    -        else:
    -            new_path = generate_name_for_obsolete_backup(local_path)
    -            new_path = os.path.join(os.path.dirname(local_path), new_path)
    -            print(MANIFEST_REPO_MOVED.format(new_path))
    -            shutil.move(local_path, new_path)
    -            print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
    -            repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
    +    pull_single_manifest_repo(repo_url, branch, local_path, reset_hard)
    +

     def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_hooks, config, skip_submodule, manifest):
         for repo_to_clone in repos_to_clone:
    diff --git a/edkrepo/common/humble.py b/edkrepo/common/humble.py
    index 64b9519..8ca38bb 100644
    --- a/edkrepo/common/humble.py
    +++ b/edkrepo/common/humble.py
    @@ -34,14 +34,6 @@ MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED = 'BRANCH or TAG name present with COMMIT I
     TAG_AND_BRANCH_SPECIFIED = 'BRANCH AND TAG name present in combination field for {} repo. Using TAG.\n'
     CHECKING_CONNECTION = 'Checking connection to remote url: {}\n'

    -# Informational messages and warnings for pull_latest_manifest_repo()
    -MANIFEST_REPO_DIRTY = 'Uncommited changes present in the global manifest repository. Run edkrepo update-manifest-repo --hard to revert these changes and sync the global manifest repository.\n'
    -MANIFEST_REPO_MOVED = '{}{}WARNING:{}{} The Global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
    -CLONING_MANIFEST_REPO = 'Cloning global manifest repository to: {} from: {}\n'
    -SYNCING_MANIFEST_REPO = 'Syncing the global manifest repository.\n'
    -MANIFEST_REPO_NOT_CONFIG_BRANCH = 'The current active branch, {}, is not the specified manifest repository branch'
    -MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH = 'Checking out the specified manifest repository branch, {}, prior to syncing'
    -
     #Error messages for sync_command.py
     SYNC_EXIT = 'Exiting without performing sync operations.'
     SYNC_UNCOMMITED_CHANGES = UNCOMMITED_CHANGES + SYNC_EXIT
    -- 
    2.16.2.windows.1