[edk2-staging/EdkRepo] [PATCH V2 0/8] EdkRepo: Support for consuming multiple manifest repositories

[edk2-staging/EdkRepo] [PATCH V2 0/8] EdkRepo: Support for consuming multiple manifest repositories

[Ashley E Desimone]

V2 Adds an additional patch (8/8) to address all of the
feedback from Erik and nate

Add the required support functionaility for consuming multiple
manifest repositories defined in both the edkrepo.cfg and
edkrepo_user.cfg files.

Includes support for determining the source manifest repository,
pulling only the manifest repository relevant to the workspace,
and finding projects within all manifest repositories.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>

Ashley E Desimone (8):
  EdkRepo: Add check for conflicting/duplicated manifest repo
    definitions
  EdkRepo: Add downloading all available manifest repositories
  EdkRepo: Add optional field to edkrepo_manifst to track the source
    manifest repo
  EdkRepo: Add list_available_manifest_repos()
  EdkRepo: Add ability to find projects across all manifest repositories
  EdkRepo: Add ability to determine the source manifest of a workspace
  EdkRepo: Add the ability to pull only the global manifest repository
    for a given workspace.
  EdkRepo: Rename man to manifest

 .../humble/manifest_repos_maintenance_humble.py    |   6 +
 .../manifest_repos_maintenance.py                  | 186 ++++++++++++++++++++-
 .../workspace_maintenance/workspace_maintenance.py |  17 +-
 edkrepo/config/tool_config.py                      |  10 ++
 edkrepo_manifest_parser/edk_manifest.py            |  26 ++-
 5 files changed, 240 insertions(+), 5 deletions(-)
 create mode 100644 edkrepo/config/tool_config.py

-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V2 1/8] EdkRepo: Add check for conflicting/duplicated manifest repo definitions

[Ashley E Desimone]

Add a functions to check for conflicting or duplicated manifest
repository definitions in the edkrepo.cfg and the edkrepo_user.cfg
files. Two manifest repositories definitions are
considered conflicting if they have the same name and at least one
of URL, branch or local path differ. Two manifest repository
definitions are considered duplicates if the name, URL, branch
and local path are the same.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../manifest_repos_maintenance.py                  | 25 ++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 6e26d4f..ad6ddbc 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -57,3 +57,28 @@ def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
             print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
             repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
 
+def detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
+    '''
+    Determines whether there is are conflicting or duplicated manifest
+    repositories listed in the edkrepo.cfg and the edkrepo_user.cfg.
+    '''
+    conflicts = []
+    duplicates = []
+    if not edkrepo_user_cfg.manifest_repo_list:
+        return conflicts, duplicates
+    else:
+        config_repos = set(edkrepo_cfg.manifest_repo_list)
+        user_cfg_repos = set(edkrepo_user_cfg.manifest_repo_list)
+    if config_repos.isdisjoint(user_cfg_repos):
+        return conflicts, duplicates
+    else:
+        for repo in config_repos.intersection(user_cfg_repos):
+            if edkrepo_cfg.get_manifest_repo_url(repo) != edkrepo_user_cfg.get_manifest_repo_url(repo):
+                conflicts.append(repo)
+            elif edkrepo_cfg.get_manifest_repo_branch(repo) != edkrepo_user_cfg.get_manifest_repo_branch(repo):
+                conflicts.append(repo)
+            elif edkrepo_cfg.get_manifest_repo_local_path(repo) != edkrepo_user_cfg.get_manifest_repo_local_path(repo):
+                conflicts.append(repo)
+            else:
+                duplicates.append(repo)
+    return conflicts, duplicates
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V2 2/8] EdkRepo: Add downloading all available manifest repositories

[Ashley E Desimone]

Add a function that will download all available manifest
repositories defined in either the edkrepo.cfg or the
edkrepo_user.cfg

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../humble/manifest_repos_maintenance_humble.py    |  4 +++
 .../manifest_repos_maintenance.py                  | 38 ++++++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
index 440fd8a..e592f19 100644
--- a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
+++ b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
@@ -21,3 +21,7 @@ SINGLE_MAN_REPO_NOT_CFG_BRANCH = ('The current active branch, {}, is not the '
                                   'specified branch for global manifst repository: {}')
 SINGLE_MAN_REPO_CHECKOUT_CFG_BRANCH = 'Checking out the specified branch: {} prior to syncing'
 SINGLE_MAN_REPO_MOVED = '{}{}WARNING:{}{} The global manifest repository has moved. Backing up previous global manifest repository to: {{}}{}\n'.format(Style.BRIGHT, Fore.RED, Style.RESET_ALL, Fore.RED, Style.RESET_ALL)
+CONFLICT_NO_CLONE = ('The definition of global manifest repository, {}, '
+                     'in the edkrepo_user.cfg does not match the definition in the edkrepo.cfg. '
+                     'This global manifest repository will not be downloaded or updated. '
+                     'Resolve the conflict and then re-run the failed operation')
diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index ad6ddbc..24ad76a 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -57,6 +57,44 @@ def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
             print (humble.CLONE_SINGLE_MAN_REPO.format(local_path, url))
             repo = Repo.clone_from(url, local_path, progress=GitProgressHandler(), branch=branch)
 
+def pull_all_manifest_repos(edkrepo_cfg, edkrepo_user_cfg, reset_hard=False):
+    '''
+    Clones or syncs all global manifest repositories defined in both the
+    edkrepo_cfg and the edkrepo_user.cfg)
+    '''
+    cfg_man_repos = []
+    user_cfg_man_repos = []
+    conflicts, duplicates = detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg)
+    if not conflicts and not duplicates:
+        cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+        user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+    elif conflicts:
+        for conflict in conflicts:
+            # In the case of a conflict do not pull conflicting repo
+            print(humble.CONFLICT_NO_CLONE.format(conflict))
+            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+            cfg_man_repos.remove(conflict)
+            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+            user_cfg_man_repos.remove(conflict)
+    elif duplicates:
+        for duplicate in duplicates:
+            # the duplicate needs to be ignored in on of the repo lists so it is
+            # not cloned/pulled twice
+            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+            user_cfg_man_repos.remove(conflict)
+    for repo in cfg_man_repos:
+        pull_single_manifest_repo(edkrepo_cfg.get_manifest_repo_url(repo),
+                                  edkrepo_cfg.get_manifest_repo_branch(repo),
+                                  edkrepo_cfg.get_manifest_repo_local_path(repo),
+                                  reset_hard)
+    for repo in user_cfg_man_repos:
+        pull_single_manifest_repo(edkrepo_user_cfg.get_manifest_repo_url(repo),
+                                  edkrepo_user_cfg.get_manifest_repo_branch(repo),
+                                  edkrepo_user_cfg.get_manifest_repo_local_path(repo),
+                                  reset_hard)
+
+
 def detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
     '''
     Determines whether there is are conflicting or duplicated manifest
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH 3/8] EdkRepo: Add optional field to edkrepo_manifst to track the source manifest repo

[Ashley E Desimone]

Add the SourceManifestRepository to the edkrepo manfiest general config
for use by edkrepo to track the source manifest repository for the workspace.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 edkrepo_manifest_parser/edk_manifest.py | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/edkrepo_manifest_parser/edk_manifest.py b/edkrepo_manifest_parser/edk_manifest.py
index 080448f..dcf9c29 100644
--- a/edkrepo_manifest_parser/edk_manifest.py
+++ b/edkrepo_manifest_parser/edk_manifest.py
@@ -21,7 +21,7 @@ import copy
 # All the namedtuple data structures that consumers of this module will need.
 #
 ProjectInfo = namedtuple('ProjectInfo', ['codename', 'description', 'dev_leads', 'reviewers', 'org', 'short_name'])
-GeneralConfig = namedtuple('GeneralConfig', ['default_combo', 'current_combo', 'pin_path'])
+GeneralConfig = namedtuple('GeneralConfig', ['default_combo', 'current_combo', 'pin_path', 'source_man_repo'])
 RemoteRepo = namedtuple('RemoteRepo', ['name', 'url'])
 RepoHook = namedtuple('RepoHook', ['source', 'dest_path', 'dest_file', 'remote_url'])
 Combination = namedtuple('Combination', ['name', 'description'])
@@ -406,6 +406,24 @@ class ManifestXml(BaseXmlHelper):
         self._tree.write(filename)
         self.__general_config.current_combo = combo_name
 
+    def write_source_manifest_repo(self, manifest_repo, filename=None):
+        '''
+        Writes the name of the source manifest repository to the
+        general config sections of the manifest file.
+        '''
+        if filename is None:
+            filename = self._fileref
+        subroot = self._tree.find('GeneralConfig')
+        if subroot is None:
+            raise KeyError(GENERAL_CONFIG_MISSING_ERROR)
+
+        element = subroot.find('SourceManifestRepository')
+        if element is None:
+            element = ET.SubElement(subroot, 'SourceManifestRepository')
+        element.attrib['manifest_repo'] = manifest_repo
+        self._tree.write(filename)
+        self.__general_config.source_man_repo = manifest_repo
+
     def generate_pin_xml(self, description, combo_name, repo_source_list, filename=None):
 
         pin_tree = ET.ElementTree(ET.Element('Pin'))
@@ -605,10 +623,14 @@ class _GeneralConfig():
             self.curr_combo = element.find('CurrentClonedCombo').attrib['combination']
         except:
             self.curr_combo = None
+        try:
+            self.source_man_repo = element.find('SourceManifestRepository').attrib['manifest_repo']
+        except:
+            self.source_man_repo = None
 
     @property
     def tuple(self):
-        return GeneralConfig(self.default_combo, self.curr_combo, self.pin_path)
+        return GeneralConfig(self.default_combo, self.curr_combo, self.pin_path, self.source_man_repo)
 
 class _RemoteRepo():
     def __init__(self, element):
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V2 4/8] EdkRepo: Add list_available_manifest_repos()

[Ashley E Desimone]

Add the ability to calculate a list of available
manifest repositories from the contents of the
edkrepo.cfg and the edkrepo_user.cfg files.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../manifest_repos_maintenance.py                  | 53 ++++++++++++++--------
 1 file changed, 34 insertions(+), 19 deletions(-)

diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 24ad76a..4bded46 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -64,25 +64,10 @@ def pull_all_manifest_repos(edkrepo_cfg, edkrepo_user_cfg, reset_hard=False):
     '''
     cfg_man_repos = []
     user_cfg_man_repos = []
-    conflicts, duplicates = detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg)
-    if not conflicts and not duplicates:
-        cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
-        user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
-    elif conflicts:
-        for conflict in conflicts:
-            # In the case of a conflict do not pull conflicting repo
-            print(humble.CONFLICT_NO_CLONE.format(conflict))
-            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
-            cfg_man_repos.remove(conflict)
-            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
-            user_cfg_man_repos.remove(conflict)
-    elif duplicates:
-        for duplicate in duplicates:
-            # the duplicate needs to be ignored in on of the repo lists so it is
-            # not cloned/pulled twice
-            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
-            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
-            user_cfg_man_repos.remove(conflict)
+    conflicts = []
+    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    for conflict in conflicts:
+        print(humble.CONFLICT_NO_CLONE.format(conflict))
     for repo in cfg_man_repos:
         pull_single_manifest_repo(edkrepo_cfg.get_manifest_repo_url(repo),
                                   edkrepo_cfg.get_manifest_repo_branch(repo),
@@ -120,3 +105,33 @@ def detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
             else:
                 duplicates.append(repo)
     return conflicts, duplicates
+
+def list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg):
+    '''
+    Checks for conflicts/duplicates within all manifest repositories defined in
+    both the edkrepo.cfg and the edkrepo_user.cfg and resturns a list of available
+    manifest_repos for each and a list of conflicting manifest repository entries.
+    '''
+    cfg_man_repos = []
+    user_cfg_man_repos = []
+    conflicts, duplicates = detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg)
+    if not conflicts and not duplicates:
+        cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+        user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+    elif conflicts:
+        for conflict in conflicts:
+            # In the case of a conflict do not pull conflicting repo
+            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+            cfg_man_repos.remove(conflict)
+            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+            user_cfg_man_repos.remove(conflict)
+    elif duplicates:
+        for duplicate in duplicates:
+            # the duplicate needs to be ignored in on of the repo lists so it is
+            # not cloned/pulled twice
+            cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
+            user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
+            user_cfg_man_repos.remove(duplicate)
+    return cfg_man_repos, user_cfg_man_repos, conflicts
+
+
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH 5/8] EdkRepo: Add ability to find projects across all manifest repositories

[Ashley E Desimone]

Add find_project_in_all_indicies() to search for and return
a tuple (source repo, source config, path to manifest) if a
matching project is found.

Add find_project_in_single_index() to find the path to a project
within a single manifest repositories index file and return a whether
the specified project was found and if so its path within the manifest
repository.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../manifest_repos_maintenance.py                  | 70 +++++++++++++++++++++-
 .../workspace_maintenance/workspace_maintenance.py | 17 +++++-
 2 files changed, 84 insertions(+), 3 deletions(-)

diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 4bded46..9b441ac 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -15,11 +15,12 @@ import git
 from git import Repo
 
 import edkrepo.config.config_factory as cfg
-from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException
+from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException, EdkrepoInvalidParametersException
 from edkrepo.common.progress_handler import GitProgressHandler
 import edkrepo.common.workspace_maintenance.humble.manifest_repos_maintenance_humble as humble
 from edkrepo.common.workspace_maintenance.workspace_maintenance import generate_name_for_obsolete_backup
-
+from edkrepo.common.workspace_maintenance.workspace_maintenance import case_insensitive_single_match
+from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml
 
 def pull_single_manifest_repo(url, branch, local_path, reset_hard=False):
     '''
@@ -135,3 +136,68 @@ def list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg):
     return cfg_man_repos, user_cfg_man_repos, conflicts
 
 
+def find_project_in_single_index (project, index_file, manifest_dir):
+    '''
+    Finds a project in a single global manifest repositories index file. If found
+    returns (True, path to file) if not returns (False, None)
+    '''
+    global_manifest_path = None
+    try:
+        proj_name = case_insensitive_single_match(project, index_file.project_list)
+    except:
+        proj_name = None
+    if proj_name:
+        ci_index_xml_rel_path = os.path.normpath(index_file.get_project_xml(proj_name))
+        global_manifest_path = os.path.join(manifest_dir, ci_index_xml_rel_path)
+        return True, global_manifest_path
+    else:
+        return False, global_manifest_path
+
+
+def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, except_msg_man_repo, except_msg_not_found, man_repo=None):
+    '''
+    Finds the project in all manifest repositories listed in the edkrepo.efg and
+    edkrepo_user.cfg. If a project with the same name is found uses man_repo to select
+    the correct entry
+    '''
+    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    projects = {}
+    for repo in cfg_man_repos:
+        manifest_dir = edkrepo_cfg.manifest_repo_abs_path(repo)
+        index_file = CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xml'))
+        found, man_path = find_project_in_single_index(project, index_file, manifest_dir)
+        if found:
+            projects[repo] = ('edkrepo_cfg', man_path)
+    for repo in user_cfg_man_repos:
+        manifest_dir = edkrepo_user_cfg.manifest_repo_abs_path(repo)
+        index_file = CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xml'))
+        found, man_path = find_project_in_single_index(project, index_file, manifest_dir)
+        if found:
+            projects[repo] = ('edkrepo_user_cfg', man_path)
+    if len(projects.keys()) == 1:
+        repo = list(projects.keys())[0]
+        return repo, projects[repo][0], projects[repo][1]
+    elif len(projects.keys()) > 1 and man_repo:
+        try:
+            return man_repo, projects[man_repo][0], projects[man_repo][1]
+        except KeyError:
+            raise EdkrepoInvalidParametersException(except_msg_man_repo)
+    elif os.path.isabs(project):
+        return None, None, project
+    elif os.path.isfile(os.path.join(os.getcwd(), project)):
+        return None, None, os.path.join(os.getcwd(), project)
+    elif not os.path.dirname(project):
+        for repo in cfg_man_repos:
+            if (man_repo and (repo == man_repo)) or not man_repo:
+                for dirpath, dirname, filenames in os.walk(edkrepo_cfg.manifest_repo_abs_path(repo)):
+                    if project in filenames:
+                        return repo, 'edkrepo_cfg', os.path.join(dirpath, project)
+        for repo in user_cfg_man_repos:
+            if (man_repo and (repo == man_repo)) or not man_repo:
+                for dirpath, dirname, filenames in os.walk(edkrepo_user_cfg.manifest_repo_abs_path(repo)):
+                    if project in filenames:
+                        return repo, 'edkrepo_user_cfg', os.path.join(dirpath, project)
+
+
+
+
diff --git a/edkrepo/common/workspace_maintenance/workspace_maintenance.py b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
index 6e20d43..ba62f6d 100644
--- a/edkrepo/common/workspace_maintenance/workspace_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/workspace_maintenance.py
@@ -10,6 +10,10 @@
 ''' Contains shared workspace maintenance functions. '''
 
 import os
+import unicodedata
+
+from edkrepo.common.edkrepo_exception import EdkrepoFoundMultipleException, EdkrepoNotFoundException
+from edkrepo.common.humble import GEN_A_NOT_IN_B, GEN_FOUND_MULT_A_IN_B
 
 def generate_name_for_obsolete_backup(absolute_path):
     if not os.path.exists(absolute_path):
@@ -27,4 +31,15 @@ def generate_name_for_obsolete_backup(absolute_path):
         if not os.path.exists(os.path.join(dir_name, unique_name)):
             unique_name_found = True
         index += 1
-    return unique_name
\ No newline at end of file
+    return unique_name
+
+def case_insensitive_equal(str1, str2):
+    return unicodedata.normalize("NFKD", str1.casefold()) == unicodedata.normalize("NFKD", str2.casefold())
+
+def case_insensitive_single_match(str1, str_list):
+    matches = [x for x in str_list if case_insensitive_equal(str1, x)]
+    if len(matches) == 0:
+        raise EdkrepoNotFoundException(GEN_A_NOT_IN_B.format(str1, str_list))
+    elif len(matches) > 1:
+        raise EdkrepoFoundMultipleException(GEN_FOUND_MULT_A_IN_B.format(str1, str_list))
+    return matches[0]
-- 
2.16.2.windows.1

[edk2-stagin/EdkRepo] [PATCH 6/8] EdkRepo: Add ability to determine the source manifest of a workspace

[Ashley E Desimone]

Add find_source_man_repo() to check if for the source manifest
repo is contained in the workspaces project manifest file.
If it is not determine the value and write it to the manifest.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../humble/manifest_repos_maintenance_humble.py           |  2 ++
 .../workspace_maintenance/manifest_repos_maintenance.py   | 15 +++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
index e592f19..05e76b1 100644
--- a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
+++ b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
@@ -25,3 +25,5 @@ CONFLICT_NO_CLONE = ('The definition of global manifest repository, {}, '
                      'in the edkrepo_user.cfg does not match the definition in the edkrepo.cfg. '
                      'This global manifest repository will not be downloaded or updated. '
                      'Resolve the conflict and then re-run the failed operation')
+SOURCE_MAN_REPO_NOT_FOUND = 'Could not determine the source global manifest repository for project: {}'
+PROJ_NOT_IN_REPO = 'Project: {} does not exist in any global manifest repository'
\ No newline at end of file
diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 9b441ac..7b3f866 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -199,5 +199,20 @@ def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, except_
                         return repo, 'edkrepo_user_cfg', os.path.join(dirpath, project)
 
 
+def find_source_man_repo (project_manifest, edkrepo_cfg, edkrepo_user_cfg):
+    '''
+    Finds the source manifest repo for a given project.
+    '''
+    if project_manifest.general_config.source_man_repo:
+       return project_manifest.general_config.source_man_repo
+    else:
+        src_man_repo, src_config, src_man_path = find_project_in_all_indices(project_manifest.project_info.codename,
+                                                                             edkrepo_cfg,
+                                                                             edkrepo_user_cfg,
+                                                                             humble.PROJ_NOT_IN_REPO.format(project_manifest.project_info.codename),
+                                                                             humble.SOURCE_MAN_REPO_NOT_FOUND.format(project_manifest.project_info.codename),
+                                                                             man_repo=None)
+        project_manifest.write_source_manifest_repo(src_man_repo)
+        return src_man_repo    
 
 
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V2 7/8] EdkRepo: Add the ability to pull only the global manifest repository for a given workspace.

[Ashley E Desimone]

Add pull_workspace_man_repo() to pull only the global manifest
repository affiliated with a single workspace.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../manifest_repos_maintenance.py                  | 23 ++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 7b3f866..7a7f946 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -199,7 +199,7 @@ def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, except_
                         return repo, 'edkrepo_user_cfg', os.path.join(dirpath, project)
 
 
-def find_source_man_repo (project_manifest, edkrepo_cfg, edkrepo_user_cfg):
+def find_source_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None):
     '''
     Finds the source manifest repo for a given project.
     '''
@@ -213,6 +213,25 @@ def find_source_man_repo (project_manifest, edkrepo_cfg, edkrepo_user_cfg):
                                                                              humble.SOURCE_MAN_REPO_NOT_FOUND.format(project_manifest.project_info.codename),
                                                                              man_repo=None)
         project_manifest.write_source_manifest_repo(src_man_repo)
-        return src_man_repo    
+        return src_man_repo
+
+def pull_workspace_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None, reset_hard=False):
+    '''
+    Pulls only the global manifest repo for the current workspace.
+    '''
+    src_man_repo = find_source_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo)
+    config_repos, user_config_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    if src_man_repo in config_repos:
+        pull_single_manifest_repo(edkrepo_cfg.get_manifest_repo_url(src_man_repo),
+                                  edkrepo_cfg.get_manifest_repo_branch(src_man_repo),
+                                  edkrepo_cfg.get_manifest_repo_local_path(src_man_repo),
+                                  reset_hard)
+    elif src_man_repo in user_config_repos:
+        pull_single_manifest_repo(edkrepo_user_cfg.get_manifest_repo_url(src_man_repo),
+                                  edkrepo_user_cfg.get_manifest_repo_branch(src_man_repo),
+                                  edkrepo_user_cfg.get_manifest_repo_local_path(src_man_repo),
+                                  reset_hard)
+    elif src_man_repo in conflicts:
+        raise EdkrepoInvalidParametersException(humble.CONFLICT_NO_CLONE.format(src_man_repo))
 
 
-- 
2.16.2.windows.1

[edk2-staging/EdkRepo] [PATCH V2 8/8] EdkRepo: Rename man to manifest

[Ashley E Desimone]

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
---
 .../humble/manifest_repos_maintenance_humble.py    |  2 +-
 .../manifest_repos_maintenance.py                  | 32 ++++++++++++----------
 edkrepo/config/tool_config.py                      | 10 +++++++
 edkrepo_manifest_parser/edk_manifest.py            | 10 +++----
 4 files changed, 34 insertions(+), 20 deletions(-)
 create mode 100644 edkrepo/config/tool_config.py

diff --git a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
index 05e76b1..290b463 100644
--- a/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
+++ b/edkrepo/common/workspace_maintenance/humble/manifest_repos_maintenance_humble.py
@@ -25,5 +25,5 @@ CONFLICT_NO_CLONE = ('The definition of global manifest repository, {}, '
                      'in the edkrepo_user.cfg does not match the definition in the edkrepo.cfg. '
                      'This global manifest repository will not be downloaded or updated. '
                      'Resolve the conflict and then re-run the failed operation')
-SOURCE_MAN_REPO_NOT_FOUND = 'Could not determine the source global manifest repository for project: {}'
+SOURCE_MANIFEST_REPO_NOT_FOUND = 'Could not determine the source global manifest repository for project: {}'
 PROJ_NOT_IN_REPO = 'Project: {} does not exist in any global manifest repository'
\ No newline at end of file
diff --git a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
index 7a7f946..2a5a70f 100644
--- a/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
+++ b/edkrepo/common/workspace_maintenance/manifest_repos_maintenance.py
@@ -15,7 +15,9 @@ import git
 from git import Repo
 
 import edkrepo.config.config_factory as cfg
+from edkrepo.config.tool_config import CI_INDEX_FILE_NAME
 from edkrepo.common.edkrepo_exception import EdkrepoUncommitedChangesException, EdkrepoInvalidParametersException
+from edkrepo.common.edkrepo_exception import EdkrepoManifestNotFoundException
 from edkrepo.common.progress_handler import GitProgressHandler
 import edkrepo.common.workspace_maintenance.humble.manifest_repos_maintenance_humble as humble
 from edkrepo.common.workspace_maintenance.workspace_maintenance import generate_name_for_obsolete_backup
@@ -66,7 +68,7 @@ def pull_all_manifest_repos(edkrepo_cfg, edkrepo_user_cfg, reset_hard=False):
     cfg_man_repos = []
     user_cfg_man_repos = []
     conflicts = []
-    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_manifest_repos(edkrepo_cfg, edkrepo_user_cfg)
     for conflict in conflicts:
         print(humble.CONFLICT_NO_CLONE.format(conflict))
     for repo in cfg_man_repos:
@@ -81,7 +83,7 @@ def pull_all_manifest_repos(edkrepo_cfg, edkrepo_user_cfg, reset_hard=False):
                                   reset_hard)
 
 
-def detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
+def detect_manifest_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
     '''
     Determines whether there is are conflicting or duplicated manifest
     repositories listed in the edkrepo.cfg and the edkrepo_user.cfg.
@@ -107,7 +109,7 @@ def detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg):
                 duplicates.append(repo)
     return conflicts, duplicates
 
-def list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg):
+def list_available_manifest_repos(edkrepo_cfg, edkrepo_user_cfg):
     '''
     Checks for conflicts/duplicates within all manifest repositories defined in
     both the edkrepo.cfg and the edkrepo_user.cfg and resturns a list of available
@@ -115,7 +117,7 @@ def list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg):
     '''
     cfg_man_repos = []
     user_cfg_man_repos = []
-    conflicts, duplicates = detect_man_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg)
+    conflicts, duplicates = detect_manifest_repo_conflicts_duplicates(edkrepo_cfg, edkrepo_user_cfg)
     if not conflicts and not duplicates:
         cfg_man_repos.extend(edkrepo_cfg.manifest_repo_list)
         user_cfg_man_repos.extend(edkrepo_user_cfg.manifest_repo_list)
@@ -160,17 +162,17 @@ def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, except_
     edkrepo_user.cfg. If a project with the same name is found uses man_repo to select
     the correct entry
     '''
-    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    cfg_man_repos, user_cfg_man_repos, conflicts = list_available_manifest_repos(edkrepo_cfg, edkrepo_user_cfg)
     projects = {}
     for repo in cfg_man_repos:
         manifest_dir = edkrepo_cfg.manifest_repo_abs_path(repo)
-        index_file = CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xml'))
+        index_file = CiIndexXml(os.path.join(manifest_dir, CI_INDEX_FILE_NAME))
         found, man_path = find_project_in_single_index(project, index_file, manifest_dir)
         if found:
             projects[repo] = ('edkrepo_cfg', man_path)
     for repo in user_cfg_man_repos:
         manifest_dir = edkrepo_user_cfg.manifest_repo_abs_path(repo)
-        index_file = CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xml'))
+        index_file = CiIndexXml(os.path.join(manifest_dir, CI_INDEX_FILE_NAME))
         found, man_path = find_project_in_single_index(project, index_file, manifest_dir)
         if found:
             projects[repo] = ('edkrepo_user_cfg', man_path)
@@ -197,30 +199,32 @@ def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, except_
                 for dirpath, dirname, filenames in os.walk(edkrepo_user_cfg.manifest_repo_abs_path(repo)):
                     if project in filenames:
                         return repo, 'edkrepo_user_cfg', os.path.join(dirpath, project)
+    else:
+        raise EdkrepoManifestNotFoundException(humble.PROJ_NOT_IN_REPO.format(project))
 
 
-def find_source_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None):
+def find_source_manifest_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None):
     '''
     Finds the source manifest repo for a given project.
     '''
-    if project_manifest.general_config.source_man_repo:
-       return project_manifest.general_config.source_man_repo
+    if project_manifest.general_config.source_manifest_repo:
+       return project_manifest.general_config.source_manifest_repo
     else:
         src_man_repo, src_config, src_man_path = find_project_in_all_indices(project_manifest.project_info.codename,
                                                                              edkrepo_cfg,
                                                                              edkrepo_user_cfg,
                                                                              humble.PROJ_NOT_IN_REPO.format(project_manifest.project_info.codename),
-                                                                             humble.SOURCE_MAN_REPO_NOT_FOUND.format(project_manifest.project_info.codename),
+                                                                             humble.SOURCE_MANIFEST_REPO_NOT_FOUND.format(project_manifest.project_info.codename),
                                                                              man_repo=None)
         project_manifest.write_source_manifest_repo(src_man_repo)
         return src_man_repo
 
-def pull_workspace_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None, reset_hard=False):
+def pull_workspace_manifest_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo=None, reset_hard=False):
     '''
     Pulls only the global manifest repo for the current workspace.
     '''
-    src_man_repo = find_source_man_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo)
-    config_repos, user_config_repos, conflicts = list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg)
+    src_man_repo = find_source_manifest_repo(project_manifest, edkrepo_cfg, edkrepo_user_cfg, man_repo)
+    config_repos, user_config_repos, conflicts = list_available_manifest_repos(edkrepo_cfg, edkrepo_user_cfg)
     if src_man_repo in config_repos:
         pull_single_manifest_repo(edkrepo_cfg.get_manifest_repo_url(src_man_repo),
                                   edkrepo_cfg.get_manifest_repo_branch(src_man_repo),
diff --git a/edkrepo/config/tool_config.py b/edkrepo/config/tool_config.py
new file mode 100644
index 0000000..eee1326
--- /dev/null
+++ b/edkrepo/config/tool_config.py
@@ -0,0 +1,10 @@
+#!/usr/bin/env python3
+#
+## @file
+# tool)config.py
+#
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+CI_INDEX_FILE_NAME = 'CiIndex.xml'
\ No newline at end of file
diff --git a/edkrepo_manifest_parser/edk_manifest.py b/edkrepo_manifest_parser/edk_manifest.py
index dcf9c29..f7118ae 100644
--- a/edkrepo_manifest_parser/edk_manifest.py
+++ b/edkrepo_manifest_parser/edk_manifest.py
@@ -21,7 +21,7 @@ import copy
 # All the namedtuple data structures that consumers of this module will need.
 #
 ProjectInfo = namedtuple('ProjectInfo', ['codename', 'description', 'dev_leads', 'reviewers', 'org', 'short_name'])
-GeneralConfig = namedtuple('GeneralConfig', ['default_combo', 'current_combo', 'pin_path', 'source_man_repo'])
+GeneralConfig = namedtuple('GeneralConfig', ['default_combo', 'current_combo', 'pin_path', 'source_manifest_repo'])
 RemoteRepo = namedtuple('RemoteRepo', ['name', 'url'])
 RepoHook = namedtuple('RepoHook', ['source', 'dest_path', 'dest_file', 'remote_url'])
 Combination = namedtuple('Combination', ['name', 'description'])
@@ -422,7 +422,7 @@ class ManifestXml(BaseXmlHelper):
             element = ET.SubElement(subroot, 'SourceManifestRepository')
         element.attrib['manifest_repo'] = manifest_repo
         self._tree.write(filename)
-        self.__general_config.source_man_repo = manifest_repo
+        self.__general_config.source_manifest_repo = manifest_repo
 
     def generate_pin_xml(self, description, combo_name, repo_source_list, filename=None):
 
@@ -624,13 +624,13 @@ class _GeneralConfig():
         except:
             self.curr_combo = None
         try:
-            self.source_man_repo = element.find('SourceManifestRepository').attrib['manifest_repo']
+            self.source_manifest_repo = element.find('SourceManifestRepository').attrib['manifest_repo']
         except:
-            self.source_man_repo = None
+            self.source_manifest_repo = None
 
     @property
     def tuple(self):
-        return GeneralConfig(self.default_combo, self.curr_combo, self.pin_path, self.source_man_repo)
+        return GeneralConfig(self.default_combo, self.curr_combo, self.pin_path, self.source_manifest_repo)
 
 class _RemoteRepo():
     def __init__(self, element):
-- 
2.16.2.windows.1

Re: [edk2-staging/EdkRepo] [PATCH V2 0/8] EdkRepo: Support for consuming multiple manifest repositories

[Nate DeSimone]

For the series...

Reviewed-by: Nate DeSimone <nathaniel.l.desimone@intel.com>

-----Original Message-----
From: Desimone, Ashley E <ashley.e.desimone@intel.com> 
Sent: Thursday, April 30, 2020 10:13 PM
To: devel@edk2.groups.io
Cc: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bjorge, Erik C <erik.c.bjorge@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH V2 0/8] EdkRepo: Support for consuming multiple manifest repositories

V2 Adds an additional patch (8/8) to address all of the feedback from Erik and nate

Add the required support functionaility for consuming multiple manifest repositories defined in both the edkrepo.cfg and edkrepo_user.cfg files.

Includes support for determining the source manifest repository, pulling only the manifest repository relevant to the workspace, and finding projects within all manifest repositories.

Signed-off-by: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>

Ashley E Desimone (8):
  EdkRepo: Add check for conflicting/duplicated manifest repo
    definitions
  EdkRepo: Add downloading all available manifest repositories
  EdkRepo: Add optional field to edkrepo_manifst to track the source
    manifest repo
  EdkRepo: Add list_available_manifest_repos()
  EdkRepo: Add ability to find projects across all manifest repositories
  EdkRepo: Add ability to determine the source manifest of a workspace
  EdkRepo: Add the ability to pull only the global manifest repository
    for a given workspace.
  EdkRepo: Rename man to manifest

 .../humble/manifest_repos_maintenance_humble.py    |   6 +
 .../manifest_repos_maintenance.py                  | 186 ++++++++++++++++++++-
 .../workspace_maintenance/workspace_maintenance.py |  17 +-
 edkrepo/config/tool_config.py                      |  10 ++
 edkrepo_manifest_parser/edk_manifest.py            |  26 ++-
 5 files changed, 240 insertions(+), 5 deletions(-)  create mode 100644 edkrepo/config/tool_config.py

--
2.16.2.windows.1