From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mx.groups.io with SMTP id smtpd.web10.3625.1575412606238568738 for ; Tue, 03 Dec 2019 14:36:46 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.100, mailfrom: ashley.e.desimone@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Dec 2019 14:36:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,275,1571727600"; d="scan'208";a="262721421" Received: from orsmsx101.amr.corp.intel.com ([10.22.225.128]) by FMSMGA003.fm.intel.com with ESMTP; 03 Dec 2019 14:36:45 -0800 Received: from orsmsx159.amr.corp.intel.com (10.22.240.24) by ORSMSX101.amr.corp.intel.com (10.22.225.128) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 3 Dec 2019 14:36:44 -0800 Received: from orsmsx116.amr.corp.intel.com ([169.254.7.11]) by ORSMSX159.amr.corp.intel.com ([169.254.11.187]) with mapi id 14.03.0439.000; Tue, 3 Dec 2019 14:36:44 -0800 From: "Desimone, Ashley E" To: "Desimone, Nathaniel L" , "devel@edk2.groups.io" Subject: Re: [edk2-staging/EdkRepo] [PATCH] EdkRepo: Add list-repos command Thread-Topic: [edk2-staging/EdkRepo] [PATCH] EdkRepo: Add list-repos command Thread-Index: AQHVpNxkl0vmXFl1xU+qrpcMwhobO6epCbuQ Date: Tue, 3 Dec 2019 22:36:43 +0000 Message-ID: <4CF3A9EB60ABDA47BE7821A4DA3A0A3353CB0E41@ORSMSX116.amr.corp.intel.com> References: <20191127043707.923-1-nathaniel.l.desimone@intel.com> In-Reply-To: <20191127043707.923-1-nathaniel.l.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjViMTYyOWMtOWRhMS00N2JlLWIwNmItYmVhNDc2NmQwOTgwIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoibFBMMHk3MVdHa0F6YzNGZ04xaTErV0kxTWZTYjA1XC84a0pjb2N3c3E3Y2pKWDZzbFltUWlCbXFwaElzekZ5R0IifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Return-Path: ashley.e.desimone@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Just a few comments: edkrepo.commands.humble needs to be added to setup.py otherwise the import= will not resolve. #length =3D len(max(branches, key=3Dlen)) .ljust(length) on line 101 of l= ist_repos_command.py should either be uncommented or removed. Thanks, Ashley -----Original Message----- From: Desimone, Nathaniel L=20 Sent: Tuesday, November 26, 2019 8:37 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E Subject: [edk2-staging/EdkRepo] [PATCH] EdkRepo: Add list-repos command list-repos lists the git repos used by all projects and which branches thos= e projects use. Cc: Ashley E Desimone Signed-off-by: Nate DeSimone --- edkrepo/commands/arguments/list_repos_args.py | 16 ++ edkrepo/commands/humble/__init__.py | 8 + edkrepo/commands/humble/list_repos_humble.py | 26 +++ edkrepo/commands/list_repos_command.py | 218 ++++++++++++++++++ 4 files changed, 268 insertions(+) create mode 100644 edkrepo/commands/arguments/list_repos_args.py create mode 100644 edkrepo/commands/humble/__init__.py create mode 100644 edkrepo/commands/humble/list_repos_humble.py create mode 100644 edkrepo/commands/list_repos_command.py diff --git a/edkrepo/commands/arguments/list_repos_args.py b/edkrepo/comman= ds/arguments/list_repos_args.py new file mode 100644 index 0000000..6ff9b9c --- /dev/null +++ b/edkrepo/commands/arguments/list_repos_args.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3+#+## @file+# list_repos_args.py+#+# Copyright=20 +(c) 2019, Intel Corporation. All rights reserved.
+#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent+#++''' Contains the help=20 +and description strings for arguments in the+list-repos command meta=20 +data.+'''++COMMAND_DESCRIPTION =3D 'Lists the git repos used by available= =20 +projects and the branches that are used.'+ARCHIVED_HELP =3D 'Include a=20 +listing of archived projects.'+REPOS_HELP =3D 'Only show the given subset= =20 +of repos instead of all repos.'diff --git=20 +a/edkrepo/commands/humble/__init__.py=20 +b/edkrepo/commands/humble/__init__.py new file mode 100644 index 0000000..dea6eb4 --- /dev/null +++ b/edkrepo/commands/humble/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3+#+## @file+# __init__.py+#+# Copyright (c) 2019,=20 +Intel Corporation. All rights reserved.
+# SPDX-License-Identifier:=20 +BSD-2-Clause-Patent+#diff --git=20 +a/edkrepo/commands/humble/list_repos_humble.py=20 +b/edkrepo/commands/humble/list_repos_humble.py new file mode 100644 index 0000000..bbb05a7 --- /dev/null +++ b/edkrepo/commands/humble/list_repos_humble.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3+#+## @file+# list_repos_humble.py+#+# Copyright (c)= 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifie= r: BSD-2-Clause-Patent+#++'''+Contains user visible strings printed by the = list-repos command.+'''++from colorama import Fore+from colorama import Sty= le++BRANCHES =3D 'Branches:'+BRANCH_FORMAT_STRING =3D ' {}{}{{}}{}'.format= (Fore.BLUE, Style.BRIGHT, Style.RESET_ALL)+COMBO_FORMAT_STRING =3D ' {{}= } {}{}({{}}){}'.format(Fore.CYAN, Style.BRIGHT, Style.RESET_ALL)+DEFAULT_C= OMBO_FORMAT_STRING =3D ' {{}} {}{}*({{}})*{}'.format(Fore.GREEN, Style.B= RIGHT, Style.RESET_ALL)+MANIFEST_DIRECTORY =3D 'Manifest directory:'+PROJEC= T_NAME_FORMAT_STRING =3D '{}{}{{}}{}:'.format(Fore.YELLOW, Style.BRIGHT, St= yle.RESET_ALL)+REPOSITORIES =3D 'Repositories:'+REPO_NAME_AND_URL =3D '{}{}= {{}}{} - [{}{}{{}}{}]'.format(Fore.MAGENTA, Style.BRIGHT, Style.RESET_ALL, = Fore.RED, Style.BRIGHT, Style.RESET_ALL)+REPO_NAME_NOT_FOUND =3D 'repo_name= not found'+REPO_NOT_FOUND_IN_MANIFEST =3D 'Repo {} not found in any manife= st file'diff --git a/edkrepo/commands/list_repos_command.py b/edkrepo/comma= nds/list_repos_command.py new file mode 100644 index 0000000..de2de03 --- /dev/null +++ b/edkrepo/commands/list_repos_command.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3+#+## @file+# list_repos_command.py+#+# Copyright (c= ) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifi= er: BSD-2-Clause-Patent+#++import collections+import os++#from git import R= epo+from colorama import Fore, Style++# Our modules+from edkrepo.commands.e= dkrepo_command import EdkrepoCommand+from edkrepo.commands.edkrepo_command = import ColorArgument+import edkrepo.commands.arguments.list_repos_args as a= rguments+import edkrepo.commands.humble.list_repos_humble as humble+from ed= krepo.common.common_repo_functions import pull_latest_manifest_repo+from ed= krepo.common.edkrepo_exception import EdkrepoInvalidParametersException, Ed= krepoManifestInvalidException+from edkrepo.common.ui_functions import init_= color_console+from edkrepo_manifest_parser.edk_manifest import CiIndexXml, = ManifestXml++class ListReposCommand(EdkrepoCommand):+ def __init__(self)= :+ super().__init__()+ self.repo_names =3D None++ def get_= metadata(self):+ metadata =3D {}+ metadata['name'] =3D 'list-= repos'+ metadata['help-text'] =3D arguments.COMMAND_DESCRIPTION+ = args =3D []+ metadata['arguments'] =3D args+ args.append(= {'name': 'repos',+ 'positional': False,+ = 'required': False,+ 'action': 'store',+ = 'nargs': '+',+ 'help-text': arguments.REPOS_= HELP})+ args.append({'name': 'archived',+ 'short= -name': 'a',+ 'positional': False,+ = 'required': False,+ 'help-text': arguments.ARCHIVED_HE= LP})+ args.append(ColorArgument)+ return metadata++ def ru= n_command(self, args, config):+ print()+ init_color_console(a= rgs.color)++ # Get path to global manifest file+ global_manif= est_directory =3D config['cfg_file'].manifest_repo_abs_local_path+ i= f args.verbose:+ print(humble.MANIFEST_DIRECTORY)+ pr= int(global_manifest_directory)+ print()+ index_path =3D o= s.path.join(global_manifest_directory, 'CiIndex.xml')++ pull_latest_= manifest_repo(args, config)+ print()++ ci_index_xml =3D CiInd= exXml(index_path)+ manifests =3D {}+ repo_urls =3D set()+ = project_list =3D list(ci_index_xml.project_list)+ if args.archiv= ed:+ project_list.extend(ci_index_xml.archived_project_list)+ = for project in project_list:+ xml_file =3D ci_index_xml.get= _project_xml(project)+ manifest =3D ManifestXml(os.path.normpath= (os.path.join(global_manifest_directory, xml_file)))+ manifests[= project] =3D manifest+ for combo in [c.name for c in manifest.co= mbinations]:+ sources =3D manifest.get_repo_sources(combo)+ = for source in sources:+ repo_urls.add(sel= f.get_repo_url(source.remote_url))+ manifests =3D collections.Ordere= dDict(sorted(manifests.items()))+ project_justify =3D len(max(manife= sts.keys(), key=3Dlen))+ self.generate_repo_names(repo_urls, manifes= ts)+ print(humble.REPOSITORIES)+ for repo_name in self.repo_n= ames:+ if args.repos and repo_name not in args.repos:+ = continue+ repo =3D self.repo_names[repo_name][0]+ = print(humble.REPO_NAME_AND_URL.format(repo_name, repo))+ prin= t(humble.BRANCHES)+ branches =3D set()+ for project_n= ame in manifests:+ for combo in [c.name for c in manifests[p= roject_name].combinations]:+ sources =3D manifests[proje= ct_name].get_repo_sources(combo)+ for source in sources:= + if self.get_repo_url(source.remote_url) =3D=3D rep= o:+ branches.add(source.branch)+ bran= ches =3D sorted(branches, key=3Dstr.casefold)+ if 'master' in br= anches:+ branches.remove('master')+ branches.= insert(0, 'master')+ #length =3D len(max(branches, key=3Dlen)) .= ljust(length)+ for branch in branches:+ print(hum= ble.BRANCH_FORMAT_STRING.format(branch))+ for project_name i= n manifests:+ combos =3D []+ for comb= o in [c.name for c in manifests[project_name].combinations]:+ = sources =3D manifests[project_name].get_repo_sources(combo)+ = for source in sources:+ if se= lf.get_repo_url(source.remote_url) =3D=3D repo and source.branch =3D=3D bra= nch:+ combos.append(combo)+ = break+ if len(combos) > 0:+ = combos =3D sorted(combos, key=3Dstr.casefold)+ = default_combo =3D manifests[project_name].general_config.default_combo= + if default_combo in combos:+ = combos.remove(default_combo)+ combos.inser= t(0, default_combo)+ first_combo =3D True+ = for combo in combos:+ if first_com= bo:+ project_name_print =3D humble.PROJECT_N= AME_FORMAT_STRING.format(project_name.ljust(project_justify))+ = first_combo =3D False+ else:+= project_name_print =3D '{} '.format((' ' * = len(project_name)).ljust(project_justify))+ if d= efault_combo =3D=3D combo:+ print(humble.DEF= AULT_COMBO_FORMAT_STRING.format(project_name_print, combo))+ = else:+ print(humble.COMBO_FORMA= T_STRING.format(project_name_print, combo))++ def get_repo_url(self, rep= o_url):+ if repo_url[-4:].lower() =3D=3D '.git':+ return = repo_url[:-4]+ return repo_url++ def get_repo_name(self, repo_url= , manifests):+ for name in self.repo_names:+ if self.repo= _names[name][0] =3D=3D repo_url:+ return name+ raise = EdkrepoInvalidParametersException(humble.REPO_NAME_NOT_FOUND)++ def gene= rate_repo_names(self, repo_urls, manifests):+ self.repo_names =3D co= llections.OrderedDict()+ for repo_url in repo_urls:+ self= .__repo_name_worker(repo_url, manifests)+ self.repo_names =3D collec= tions.OrderedDict(sorted(self.repo_names.items()))+ names_to_move = =3D []+ for repo_name in self.repo_names:+ if repo_name.l= ower().find('edk2') =3D=3D 0:+ names_to_move.append(repo_nam= e)+ names_to_move =3D sorted(names_to_move, reverse=3DTrue)+ = for name_to_move in names_to_move:+ self.repo_names.move_to_end(= name_to_move, False)+ names_to_move =3D []+ for repo_name in = self.repo_names:+ if repo_name.lower().find('intel') =3D=3D 0:+ = names_to_move.append(repo_name)+ names_to_move =3D so= rted(names_to_move, reverse=3DTrue)+ for name_to_move in names_to_mo= ve:+ self.repo_names.move_to_end(name_to_move, False)++ def _= _repo_name_worker(self, repo_url, manifests):+ #This is a heuristic = that guesses the "name" of a repository by looking+ #at the name giv= en to it by the most manifest files.+ names =3D collections.defaultd= ict(int)+ for project_name in manifests:+ for combo in [c= .name for c in manifests[project_name].combinations]:+ sourc= es =3D manifests[project_name].get_repo_sources(combo)+ for = source in sources:+ if self.get_repo_url(source.remote_u= rl) =3D=3D repo_url:+ names[source.root] +=3D 1+ = found_unique_name =3D False+ original_best_name =3D None+ = original_best_name_frequency =3D 0+ while not found_unique_name:+ = best_name =3D None+ best_name_frequency =3D 0+ = if len(names) <=3D 0:+ if original_best_name_frequency = =3D=3D 1:+ #If only 1 project uses this name, then appen= d the project+ #name to the directory name to create the= repo name+ for project_name in manifests:+ = for combo in [c.name for c in manifests[project_name].combinatio= ns]:+ sources =3D manifests[project_name].get_re= po_sources(combo)+ for source in sources:+ = if self.get_repo_url(source.remote_url) =3D=3D re= po_url and source.root =3D=3D original_best_name:+ = best_name =3D "{}-{}".format(original_best_name, project_name)+ = best_name_frequency =3D original_best_na= me_frequency+ else:+ best_name =3D repo_u= rl+ best_name_frequency =3D 0+ break+ = for name in names:+ if names[name] > best_name_frequ= ency:+ best_name =3D name+ best_name_= frequency =3D names[name]+ if best_name is None:+ = raise EdkrepoManifestInvalidException(humble.REPO_NOT_FOUND_IN_MANIFEST.fo= rmat(repo_url))+ if original_best_name is None:+ = original_best_name =3D best_name+ original_best_name_frequen= cy =3D best_name_frequency+ if best_name in self.repo_names:+ = if self.repo_names[best_name][0] =3D=3D repo_url:+ = found_unique_name =3D True+ else:+ = #If there is a name collision, then which repo has the most+ = #Usage of the name owns the name+ if best_name_fr= equency > self.repo_names[best_name][1]:+ old_repo_u= rl =3D self.repo_names[name][0]+ del self.repo_names= [best_name]+ found_unique_name =3D True+ = self.repo_names[best_name] =3D (repo_url, best_name_frequency)+= self.__repo_name_worker(old_repo_url, manifests)+ = else:+ #Use the name given by the = second most manifest files+ del names[best_name]+ = else:+ found_unique_name =3D True+ self.repo= _names[best_name] =3D (repo_url, best_name_frequency)--=20 2.24.0.windows.2