* [edk2-staging/EdkRepo] [PATCH V5 1/3] EdkRepo: Initial commit of workspace_maintenance.py
2020-04-23 23:09 [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Ashley E Desimone
@ 2020-04-23 23:09 ` Ashley E Desimone
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 2/3] EdkRepo: Add edkrepo/common/workspace_maintenance/manifest_repos_maintenance Ashley E Desimone
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Ashley E Desimone @ 2020-04-23 23:09 UTC (permalink / raw)
To: devel
Cc: Nate DeSimone, Puja Pandya, Erik Bjorge, Bret Barkelew,
Prince Agyeman
Add workspace_maintenance.py including the implementation
of generate_name_for_obsolete_backup()
Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
.../workspace_maintenance/workspace_maintenance.py | 30 ++++++++++++++++++++++
setup.py | 4 +--
2 files changed, 32 insertions(+), 2 deletions(-)
create mode 100644 edkrepo/common/workspace_maintenance/workspace_maintenance.py
diff --git a/edkrepo/common/workspace_maintenance/workspace_maintenance.py b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
new file mode 100644
index 0000000..d45759a
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+#
+## @file
+# workspace_maintenance.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains shared workspace maintenance functions. '''
+
+import os
+
+def generate_name_for_obsolete_backup(absolute_path):
+ if not os.path.exists(absolute_path):
+ raise ValueError("{} does not exist".format(absolute_path))
+ original_name = os.path.basename(absolute_path)
+ dir_name = os.path.dirname(absolute_path)
+ unique_name = ""
+ unique_name_found = False
+ index = 1
+ while not unique_name_found:
+ if index == 1:
+ unique_name = "{}_old".format(original_name)
+ else:
+ unique_name = "{}_old{}".format(original_name, index)
+ if not os.path.exists(os.path.join(dir_name, unique_name)):
+ unique_name_found = True
+ index += 1
+ return unique_name
\ No newline at end of file
diff --git a/setup.py b/setup.py
index 06a382b..af173bb 100755
--- a/setup.py
+++ b/setup.py
@@ -12,8 +12,8 @@ setup(name='edkrepo',
version='2.0.0',
description='The edkrepo tools',
packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
- 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.config', 'edkrepo.config.humble',
- 'edkrepo_manifest_parser', 'project_utils'],
+ 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maintenance',
+ 'edkrepo.config', 'edkrepo.config.humble', 'edkrepo_manifest_parser', 'project_utils'],
package_data={
},
include_package_data=True,
--
2.16.2.windows.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [edk2-staging/EdkRepo] [PATCH V5 2/3] EdkRepo: Add edkrepo/common/workspace_maintenance/manifest_repos_maintenance
2020-04-23 23:09 [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Ashley E Desimone
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 1/3] EdkRepo: Initial commit of workspace_maintenance.py Ashley E Desimone
@ 2020-04-23 23:09 ` Ashley E Desimone
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo Ashley E Desimone
2020-04-23 23:29 ` [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Nate DeSimone
3 siblings, 0 replies; 5+ messages in thread
From: Ashley E Desimone @ 2020-04-23 23:09 UTC (permalink / raw)
To: devel
Cc: Nate DeSimone, Puja Pandya, Erik Bjorge, Bret Barkelew,
Prince Agyeman
Add a directory to edkrepo/common to store workspace maintenance
functionatlity.
Add edkrepo/common/workspace_maintenance/humble directory to store
related strings and included manifest_repos_maintenance_humble.py
Add edkrepo/common/workspace_maintenance/manifest_repos_maitenance.py
to support the mmaintenance of multiple manifest repositories.
Added pull_single_manifest_repo() to manifest_repos_maintenance.py
Updated setup.py to include the new directories.
Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
.../humble/manifest_repos_maintenance_humble.py | 23 +++++++++
.../manifest_repos_maintenance.py | 59 ++++++++++++++++++++++
setup.py | 3 +-
3 files changed, 84 insertions(+), 1 deletion(-)
create mode 100644 edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
create mode 100644 edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
diff --git a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
new file mode 100644
index 0000000..440fd8a
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
@@ -0,0 +1,23 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_maintenance_humble.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+''' Contains user facing strings for manifest_repos_mgmt.py '''
+
+from colorama import Fore
+from colorama import Style
+
+CLONE_SINGLE_MAN_REPO = 'Cloning global manifest repository to: {} from: {}'
+SYNC_SINGLE_MAN_REPO = 'Syncing the global manifest repository: {}'
+SINGLE_MAN_REPO_DIRTY = ('Uncommited changes present in the global manifest '
+ 'repository: {} Resolve these changes and attempt your'
+ ' operation again.')
+SINGLE_MAN_REPO_NOT_CFG_BRANCH = ('The current active branch, {}, is not the '
+ 'specified branch for global manifst repository: {}')
+SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH = 'Checking out the specified branch: {} prior to syncing'
+SINGLE_MAN_REPO_MOVED = '{}{}WARNING:{}{} The global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
new file mode 100644
index 0000000..6e26d4f
--- /dev/null
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+#
+## @file
+# manifest_repos_maintenance.py
+#
+# Copyright (c) 2017- 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import os
+import traceback
+import shutil
+
+import git
+from git import Repo
+
+import edkrepo.config.config_factory as cfg
+from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException
+from edkrepo.common.progress_handler import GitProgressHandler
+import edkrepo.common.workspace_maintenance.humble.manifest_repos_maintenance_humble as humble
+from edkrepo.common.workspace_maintenance.workspace_maintenance import generate_name_for_obsolete_backup
+
+
+def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
+ '''
+ Clones or syncs a single global manifest repository as defined in either
+ the edkrepo.cfg or the edkrepo_user.cfg
+ '''
+ # If a relative path is used join to the edkrepo global data directory path
+ if not os.path.isabs(local_path):
+ local_path = os.path.join(cfg.get_edkrepo_global_data_directory(), local_path)
+ # Clone the repository if it does not exist locally
+ if not os.path.exists(local_path):
+ print(humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+ repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+ # Sync the repository if it exists locally
+ else:
+ repo = Repo(local_path)
+ if url in repo.remotes['origin'].urls:
+ if repo.is_dirty(untracked_files=True) and not reset_hard:
+ raise EdkrepoUncommitedChangesException(humble.SINGLE_MAN_REPO_DIRTY.format(local_path))
+ elif repo.is_dirty(untracked_files=True) and reset_hard:
+ repo.git.reset('--hard')
+ print(humble.SYNC_SINGLE_MAN_REPO.format(local_path))
+ if repo.active_branch.name != branch:
+ print(humble.SINGLE_MAN_REPO_NOT_CFG_BRANCH.format(repo.active_branch.name, local_path))
+ print(humble.SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH.format(branch))
+ repo.git.checkout(branch)
+ repo.remotes.origin.pull()
+ # If the URL specified for this manifest repo has moved back up the existing
+ # local copy and clone the new repository
+ else:
+ new_path = generate_name_for_obsolete_backup(local_path)
+ new_path = os.path.join(os.path.dirname(local_path), new_path)
+ print(humble.SINGLE_MAN_REPO_MOVED.format(new_path))
+ shutil.move(local_path, new_path)
+ print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
+ repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
+
diff --git a/setup.py b/setup.py
index af173bb..5053b1a 100755
--- a/setup.py
+++ b/setup.py
@@ -13,7 +13,8 @@ setup(name='edkrepo',
description='The edkrepo tools',
packages=['edkrepo', 'edkrepo.commands', 'edkrepo.commands.arguments', 'edkrepo.commands.humble',
'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.common.workspace_maintenance',
- 'edkrepo.config', 'edkrepo.config.humble', 'edkrepo_manifest_parser', 'project_utils'],
+ 'edkrepo.common.workspace_maintenance.humble', 'edkrepo.config', 'edkrepo.config.humble',
+ 'edkrepo_manifest_parser', 'project_utils'],
package_data={
},
include_package_data=True,
--
2.16.2.windows.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [edk2-staging/EdkRepo] [PATCH V5 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo
2020-04-23 23:09 [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Ashley E Desimone
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 1/3] EdkRepo: Initial commit of workspace_maintenance.py Ashley E Desimone
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 2/3] EdkRepo: Add edkrepo/common/workspace_maintenance/manifest_repos_maintenance Ashley E Desimone
@ 2020-04-23 23:09 ` Ashley E Desimone
2020-04-23 23:29 ` [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Nate DeSimone
3 siblings, 0 replies; 5+ messages in thread
From: Ashley E Desimone @ 2020-04-23 23:09 UTC (permalink / raw)
To: devel
Cc: Nate DeSimone, Puja Pandya, Erik Bjorge, Bret Barkelew,
Prince Agyeman
Updates the implementation for pull_latest_manifest_repo to
call pull_single_manifest repo.
Removes definitions of strings used by pull_latest_manifest_repo
from common/humble.py and from the imports of
common_repo_functions.py
Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
edkrepo/common/common_repo_functions.py | 33 ++++-----------------------------
edkrepo/common/humble.py | 8 --------
2 files changed, 4 insertions(+), 37 deletions(-)
diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
index 0d39291..2aa5f5f 100644
--- a/edkrepo/common/common_repo_functions.py
+++ b/edkrepo/common/common_repo_functions.py
@@ -46,12 +46,10 @@ from edkrepo.common.humble import COMMIT_TEMPLATE_NOT_FOUND, COMMIT_TEMPLATE_CUS
from edkrepo.common.humble import ADD_PRIMARY_REMOTE, REMOVE_PRIMARY_REMOTE
from edkrepo.common.humble import FETCH_PRIMARY_REMOTE, MIRROR_PRIMARY_SHA, TAG_AND_BRANCH_SPECIFIED
from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_REPO, HOOK_NOT_FOUND_ERROR, SUBMODULE_FAILURE
-from edkrepo.common.humble import MANIFEST_REPO_DIRTY, MANIFEST_REPO_MOVED, CLONING_MANIFEST_REPO, SYNCING_MANIFEST_REPO
from edkrepo.common.humble import INCLUDED_URL_LINE, INCLUDED_INSTEAD_OF_LINE, INCLUDED_FILE_NAME
from edkrepo.common.humble import ERROR_WRITING_INCLUDE, MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED
from edkrepo.common.humble import VERIFY_GLOBAL, VERIFY_ARCHIVED, VERIFY_PROJ, VERIFY_PROJ_FAIL
-from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL, MANIFEST_REPO_NOT_CONFIG_BRANCH
-from edkrepo.common.humble import MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH
+from edkrepo.common.humble import VERIFY_PROJ_NOT_IN_INDEX, VERIFY_GLOBAL_FAIL
from edkrepo.common.pathfix import get_actual_path
from project_utils.sparse import BuildInfo, process_sparse_checkout
from edkrepo.config.config_factory import get_workspace_path
@@ -61,6 +59,7 @@ from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml
from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException, EdkrepoGitException, EdkrepoWarningException
from edkrepo.common.edkrepo_exception import EdkrepoFoundMultipleException, EdkrepoHookNotFoundException
from edkrepo.common.edkrepo_exception import EdkrepoGitConfigSetupException, EdkrepoManifestInvalidException
+from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import pull_single_manifest_repo
from edkrepo.common.ui_functions import init_color_console
from edkrepo_manifest_parser import edk_manifest
from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrepo
@@ -77,32 +76,8 @@ def pull_latest_manifest_repo(args, config, reset_hard=False):
branch = config['cfg_file'].manifest_repo_branch
local_path = config['cfg_file'].manifest_repo_local_path
init_color_console(False)
- if not (os.path.isabs(local_path)):
- #since only a relative path was specified it must be joined to the Edkrepo Application Data Directory
- local_path = os.path.join(get_edkrepo_global_data_directory(), local_path)
- if not os.path.exists(local_path):
- print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
- repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
- else:
- repo = Repo(local_path)
- if repo_url in repo.remotes['origin'].urls:
- if repo.is_dirty(untracked_files=True) and not reset_hard:
- raise EdkrepoWarningException(MANIFEST_REPO_DIRTY)
- elif repo.is_dirty(untracked_files=True) and reset_hard:
- repo.git.reset('--hard')
- print (SYNCING_MANIFEST_REPO)
- if repo.active_branch.name != branch:
- print(MANIFEST_REPO_NOT_CONFIG_BRANCH.format(repo.active_branch.name))
- print(MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH.format(branch))
- repo.git.checkout(branch)
- repo.remotes.origin.pull()
- else:
- new_path = generate_name_for_obsolete_backup(local_path)
- new_path = os.path.join(os.path.dirname(local_path), new_path)
- print(MANIFEST_REPO_MOVED.format(new_path))
- shutil.move(local_path, new_path)
- print (CLONING_MANIFEST_REPO.format(local_path, repo_url))
- repo = Repo.clone_from(repo_url, local_path, progress=GitProgressHandler(), branch=branch)
+ pull_single_manifest_repo(repo_url, branch, local_path, reset_hard)
+
def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_hooks, config, skip_submodule, manifest):
for repo_to_clone in repos_to_clone:
diff --git a/edkrepo/common/humble.py b/edkrepo/common/humble.py
index 64b9519..8ca38bb 100644
--- a/edkrepo/common/humble.py
+++ b/edkrepo/common/humble.py
@@ -34,14 +34,6 @@ MULTIPLE_SOURCE_ATTRIBUTES_SPECIFIED = 'BRANCH or TAG name present with COMMIT I
TAG_AND_BRANCH_SPECIFIED = 'BRANCH AND TAG name present in combination field for {} repo. Using TAG.\n'
CHECKING_CONNECTION = 'Checking connection to remote url: {}\n'
-# Informational messages and warnings for pull_latest_manifest_repo()
-MANIFEST_REPO_DIRTY = 'Uncommited changes present in the global manifest repository. Run edkrepo update-manifest-repo --hard to revert these changes and sync the global manifest repository.\n'
-MANIFEST_REPO_MOVED = '{}{}WARNING:{}{} The Global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
-CLONING_MANIFEST_REPO = 'Cloning global manifest repository to: {} from: {}\n'
-SYNCING_MANIFEST_REPO = 'Syncing the global manifest repository.\n'
-MANIFEST_REPO_NOT_CONFIG_BRANCH = 'The current active branch, {}, is not the specified manifest repository branch'
-MANIFEST_REPO_CHECKOUT_CONFIG_BRANCH = 'Checking out the specified manifest repository branch, {}, prior to syncing'
-
#Error messages for sync_command.py
SYNC_EXIT = 'Exiting without performing sync operations.'
SYNC_UNCOMMITED_CHANGES = UNCOMMITED_CHANGES + SYNC_EXIT
--
2.16.2.windows.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support
2020-04-23 23:09 [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support Ashley E Desimone
` (2 preceding siblings ...)
2020-04-23 23:09 ` [edk2-staging/EdkRepo] [PATCH V5 3/3] EdkRepo: Update pull_latest_manifest_repo to use pull_single_manifest_repo Ashley E Desimone
@ 2020-04-23 23:29 ` Nate DeSimone
3 siblings, 0 replies; 5+ messages in thread
From: Nate DeSimone @ 2020-04-23 23:29 UTC (permalink / raw)
To: Desimone, Ashley E, devel@edk2.groups.io
Cc: Pandya, Puja, Bjorge, Erik C, Bret Barkelew, Agyeman, Prince
For the patch series...
Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
-----Original Message-----
From: Desimone, Ashley E <ashley.e.desimone@intel.com>
Sent: Thursday, April 23, 2020 4:10 PM
To: devel@edk2.groups.io
Cc: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bjorge, Erik C <erik.c.bjorge@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH v5 0/3] Add initial manifest repository support
V5 updates the imports in patch 3/3 to reflect the change in file names resulting from V4
V4 addresses Nate's comments regarding typo of 'maintenance'
and added edkrepo.common.workspace_maintenance to setup.py in patch 1/3
V3 Addresses Prince's comments regarding typos in patch 2/3
V2 Replaces use of management with maitenance per feedback
V2 Introduces patch 1/1 which removes a circular import between common_repo_functions.py and manifest_repos_maitenance.py by adding workspace_maiteneance.py.
Adds initial support for manifest repository maitenance while retaining support for commands to use the existing edkrepo.cfg format and manifest repo support functions.
Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Ashley E Desimone (3):
EdkRepo: Initial commit of workspace_maintenance.py
EdkRepo: Add
edkrepo/common/workspace_maintenance/manifest_repos_maintenance
EdkRepo: Update pull_latest_manifest_repo to use
pull_single_manifest_repo
edkrepo/common/common_repo_functions.py | 33 ++----------
edkrepo/common/humble.py | 8 ---
.../humble/manifest_repos_maintenance_humble.py | 23 +++++++++
.../manifest_repos_maintenance.py | 59 ++++++++++++++++++++++
.../workspace_maintenance/workspace_maintenance.py | 30 +++++++++++
setup.py | 5 +-
6 files changed, 119 insertions(+), 39 deletions(-) create mode 100644 edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
create mode 100644 edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
create mode 100644 edkrepo/common/workspace_maintenance/workspace_maintenance.py
--
2.16.2.windows.1
^ permalink raw reply [flat|nested] 5+ messages in thread