From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web11.6798.1589176720303817719 for ; Sun, 10 May 2020 22:58:40 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=iC2psxMK; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: nathaniel.l.desimone@intel.com) IronPort-SDR: g4EibBCg4wjYI1DL8U/kvwhRPlyYo2TlmbppgJFzlTv/rKA4AJYjVom0nx4Ob7A5amfZ9NjuRd eJwMIVzTDuXw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2020 22:58:39 -0700 IronPort-SDR: yWH/yGCfIyLEJG1KTDbWv9bUpWtng2XCi/5+oDptJpAuJRbLl4bRPxHhQe93N3IHhydiFTDOE9 fgGV2uyd8Xmg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,378,1583222400"; d="scan'208";a="297578802" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga008.jf.intel.com with ESMTP; 10 May 2020 22:58:38 -0700 Received: from fmsmsx122.amr.corp.intel.com (10.18.125.37) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 10 May 2020 22:58:38 -0700 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx122.amr.corp.intel.com (10.18.125.37) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 10 May 2020 22:58:38 -0700 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (104.47.37.50) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 10 May 2020 22:58:38 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UaouPAKXqcU1gOsWZqV+ZWmm6Tm9D3zp4E6AbfVRgP1w7/pRX+gv99tUvozSIxwNWYLMwKGy29EgVSNiRtLk2cEExK14r68oKdHlR0adXQPCxj10l/Qw3Zb2XkMt9bUzLcISOf1ysarUHzs552yJDSufiYE37PtOCoJUmteBbLQ2lyZjVgfq6AMGIaUWyfIvAhrPD/RO3gb+OySp2IP2tV/JPvApeRstqDorpqw+yG+Tg11RY5CLgB0wySuiRuNmmvm9Ncys9VDmgOoBcWdyiH/jf/pFVkjB2z1GVdKkdn4gR8U5RoOYkosyBTK30QEWV92a5GBiT+bYsKACwAfCLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s8fZBaVTLBJSQrtHaABKd7lEXm5rLgYprBFYym1uD7M=; b=Vx7MCFqXltQolB4e3pzebmYORB/ArcsgvyH9peJlV9i4hvELwt7Kp/ZgFKF78Zi7SarhILZCnxcPue68y8B+4aSJbNNkvb8ZcDI6Bz94CJHmCrSBwESRyoU67qCqc5X9hXm4x2TxC/1nVgJhGx5bMco2rL5YLYo743pPL1vGm5g8Q4rC7i48zISBhVudQegP7kqENVaC+8wo+2kfQEym6VHct0X7ThyRE8pmm9L1EZQEbxLAF0aYdwzi00NjmUw6S6+smCqJ7yrAqbIc+EUbl3o9iYRnPxkDxX0v4Y4yTskik04oi8dO2HYRdkz3t9iLdIWmvdqTVBx4pLk6vm2HqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s8fZBaVTLBJSQrtHaABKd7lEXm5rLgYprBFYym1uD7M=; b=iC2psxMKCASpg/qUils5HHa1mhEuRhClBO2NKZ3H1aezxmEEhqcG1tBVPrsADSST9e61cM6rNDMVfFYi98mCFi77tYOuNonoMwuXDJjxCKUdNBPdLHOKLRMmxSZ5AqZAjLMMIssd6EJ1GDEvkerStVQxCWmTzYr7QOF7qw7ihGc= Received: from BL0PR11MB3489.namprd11.prod.outlook.com (2603:10b6:208:75::29) by BL0PR11MB3204.namprd11.prod.outlook.com (2603:10b6:208:60::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2979.34; Mon, 11 May 2020 05:58:36 +0000 Received: from BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42]) by BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42%7]) with mapi id 15.20.2979.033; Mon, 11 May 2020 05:58:36 +0000 From: "Nate DeSimone" To: "devel@edk2.groups.io" , "Desimone, Ashley E" CC: "Pandya, Puja" , "Bjorge, Erik C" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-devel] [edk2-staging/EdkRepo] [PATCH] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Topic: [edk2-devel] [edk2-staging/EdkRepo] [PATCH] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Index: AQHWJuOJrl5yGxR4+keLyuJxeeU27KiiZEYg Date: Mon, 11 May 2020 05:58:36 +0000 Message-ID: References: <20200510155538.34956-1-ashley.e.desimone@intel.com> In-Reply-To: <20200510155538.34956-1-ashley.e.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNTZhMGYwYzAtMWUxYS00ZjhjLWEwMTYtN2U3NGQ3OWM4YWIxIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiUW83MkN4THFJcEoxMWtIOUI2V09DaEwxZkFwMTR0VEhlSTlTNURzMU5GVTdURHFtQmFwWjZpbmp4OUFLUWlhNiJ9 dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows x-ctpclassification: CTP_NT authentication-results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f928b2f1-45f7-4adc-64be-08d7f5705870 x-ms-traffictypediagnostic: BL0PR11MB3204: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2276; x-forefront-prvs: 04004D94E2 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cTyHKgAmUYs1j7dazodaPZBnoXSiHasVARYXmiW0/bUnwlJIyB+02G0B/QsAyTuOWj08vPQZAInsmBH9jU1OkUIPsu9wbymSeGo0g0WKlzfZnGncBbMqqVx8qNjhs+A4b8rz+sry9teGZVk26fU3ORlXEW6WF0IfZ3X289NfRAS7ac6EYbfAZIitU1Ec7r1M31MdmS8asR5FkXdPc4fwDzZSKf7HeBI9tboLlSbh3f0pkdQ4rOGACOKgvdxeWL8TH8UFRO/ZzFRNHOHkkAXkgmUQqNupndbP6MlXK1RyQ6lpMeO2RHzBILbL/euuPcBcZGvb8Pg6tr0mX5Ms208wG/opuM4mK/vqHyZ2BnOHj/V1zo/Ko7e8fnix8p6rpKG6h5X6S2eNT3JkMnz7fUlgGg4v7JpalP60WLayMlV5s4MAwb/4+d//v3O9pSjZ7Xuy/TeAKmkUjkV1M7Zf/Chljk6HuAfcMgLL/TQGckxkLO1qgknCr7Iz0nI37mL6O4QY3DeYs2F78TnFZxh82k6qgoEhoHJdcMp90T8iOLGqmvT+J4L/lPNA9DPb7Sk9ESi5fSzfJCuVcEJwzYH1rPmd4nZ0BVkvEKOTITvXeU4sMUw= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR11MB3489.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(39860400002)(136003)(366004)(346002)(376002)(396003)(33430700001)(52536014)(8676002)(15650500001)(186003)(5660300002)(26005)(9686003)(53546011)(6506007)(7696005)(86362001)(4326008)(8936002)(107886003)(55016002)(66556008)(6636002)(64756008)(66446008)(33656002)(66946007)(76116006)(110136005)(66476007)(2906002)(966005)(33440700001)(316002)(478600001)(54906003)(71200400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: rZGUtHH1CpYZmdGRgpyL8Af5bwb3w27h1oCgE3hlxlBLwfP5yy4MtTPXtb73g4shlM+YiqEphhsXssFfd6u6oUQ4lYFUrM02Pkoc1Xw/K4pLTna+dIvNtAtiYQg8ZIZFVjj03E4/p7WGkj0eu+8PZ3tYdIMhPbt2MPkNjLtUfsQsOCTC93F9KItSOJRU6OuwGXtDmHPlwPLk6kfbfnKqrynUlTkpSJRGgLLr7SCfJqT5HNfjK2bBhLUc4CGyBQKc3X1QG8WQXLVvMIFrmatknZ7YcYdw/h5muOCfYzDd2Y7S1bpscI5Q5+6WqKtUriJ1Xgk/RuQU0Vb02ApqWx9ezJm4gUfYd4l1i4mr9FvBlQlOZXmCt4awXHK7DXssftFb8S1DXIXYrOmGc3ToIVEuPOkKZnabZgcJxmG3gxTZWAqD5HlDQ8Rhxd+MVSRp0j2r+gDrUPQIWQv78OWycGX+5iPgpjnFyf7Za9JYwqVHQj4= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: f928b2f1-45f7-4adc-64be-08d7f5705870 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 May 2020 05:58:36.2958 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Yes99dHRL2l38PZwG1rIdfgPAK//mrFglJLdhWUW0Z7RVbYE7lP3iiTyc8ALeS80xzATJJ+34IhvmtRXGaNDOGXzySBhOqXbGhVFcG1D4ZE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3204 Return-Path: nathaniel.l.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Ashley, Feedback inline below. Thanks, Nate > -----Original Message----- > From: devel@edk2.groups.io On Behalf Of Ashley E > Desimone > Sent: Sunday, May 10, 2020 8:56 AM > To: devel@edk2.groups.io > Cc: Desimone, Nathaniel L ; Pandya, Puja > ; Bjorge, Erik C ; Bret > Barkelew ; Agyeman, Prince > > Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH] EdkRepo: Update > list-repos to support multiple manifest repositories >=20 > Update the list repos command to support calculating the list of consume= d > repositories from all manifests in all manifest repositories defined in = both the > edkrepo.cfg and the edkrepo.cfg >=20 > Signed-off-by: Ashley E Desimone > Cc: Nate DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Cc: Prince Agyeman > --- > edkrepo/commands/list_repos_command.py | 97 > +++++++++++++++++++++++++--------- > 1 file changed, 72 insertions(+), 25 deletions(-) >=20 > diff --git a/edkrepo/commands/list_repos_command.py > b/edkrepo/commands/list_repos_command.py > index b06a493..364a138 100644 > --- a/edkrepo/commands/list_repos_command.py > +++ b/edkrepo/commands/list_repos_command.py > @@ -18,9 +18,11 @@ from edkrepo.commands.edkrepo_command import > EdkrepoCommand from edkrepo.commands.edkrepo_command import > ColorArgument import edkrepo.commands.arguments.list_repos_args as > arguments import edkrepo.commands.humble.list_repos_humble as > humble -from edkrepo.common.common_repo_functions import > pull_latest_manifest_repo from edkrepo.common.edkrepo_exception > import EdkrepoInvalidParametersException, > EdkrepoManifestInvalidException from edkrepo.common.ui_functions > import init_color_console > +from > edkrepo.common.workspace_maintenance.manifest_repos_maintenance > +import pull_all_manifest_repos from > +edkrepo.common.workspace_maintenance.manifest_repos_maintenance > import > +list_available_manifest_repos from edkrepo.config.tool_config import > +CI_INDEX_FILE_NAME > from edkrepo_manifest_parser.edk_manifest import CiIndexXml, > ManifestXml >=20 > class ListReposCommand(EdkrepoCommand): > @@ -52,35 +54,80 @@ class ListReposCommand(EdkrepoCommand): > print() > init_color_console(args.color) >=20 > - # Get path to global manifest file > - global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_local_path > - if args.verbose: > - print(humble.MANIFEST_DIRECTORY) > - print(global_manifest_directory) > - print() > - index_path =3D os.path.join(global_manifest_directory, 'CiIndex= .xml') > - > - pull_latest_manifest_repo(args, config) > + pull_all_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > print() >=20 > - #Create a dictionary containing all the manifests listed in the= CiIndex.xml > file > - ci_index_xml =3D CiIndexXml(index_path) > + cfg_manifest_repos, user_config_manifest_repos, conflicts =3D > + list_available_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > + > + found_manifests =3D {} > manifests =3D {} > repo_urls =3D set() > - project_list =3D list(ci_index_xml.project_list) > - if args.archived: > - project_list.extend(ci_index_xml.archived_project_list) > - for project in project_list: > - xml_file =3D ci_index_xml.get_project_xml(project) > - manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > - manifests[project] =3D manifest > - combo_list =3D [c.name for c in manifest.combinations] > + config_manifest_repos_project_list =3D [] > + user_config_manifest_repos_project_list =3D [] > + > + for manifest_repo in cfg_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in= the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + print(index_path) > + ci_index_xml =3D CiIndexXml(index_path) > + config_manifest_repos_project_list =3D > + ci_index_xml.project_list > if args.archived: > - combo_list.extend([c.name for c in > manifest.archived_combinations]) > - for combo in combo_list: > - sources =3D manifest.get_repo_sources(combo) > - for source in sources: > - repo_urls.add(self.get_repo_url(source.remote_url)) > + > config_manifest_repos_project_list.extend(ci_index_xml.archived_project_ > list) > + for project in config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)]= =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + repo_urls.add(self.get_repo_url(source.remote_u= rl)) > + for manifest_repo in user_config_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['user_cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in= the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + ci_index_xml =3D CiIndexXml(index_path) > + user_config_manifest_repos_project_list =3D ci_index_xml.pr= oject_list > + if args.archived: > + > user_config_manifest_repos_project_list.extend(ci_index_xml.archived_pr > oject_list) > + for project in user_config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)]= =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + > + repo_urls.add(self.get_repo_url(source.remote_url)) > + > + #Remove the manifest repo portion of the key is there is not a > duplicate project name > + key_list =3D list(found_manifests) > + for entry in key_list: > + new_key =3D entry.split(':')[1] > + value =3D found_manifests[entry] > + del found_manifests[entry] > + for found_manifest in list(found_manifests): > + if found_manifest.split(':')[1] =3D=3D new_key: > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':')[0], entry.split(':')[1]) > + #break I don't think the break should be commented out. I think you want that inc= luded. > + if new_key in manifests.keys(): > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':'[0]), entry.split(':')[1]) > + manifests[new_key] =3D value >=20 > #Sort the manifests so projects will be displayed alphabeticall= y > manifests =3D collections.OrderedDict(sorted(manifests.items())= ) > -- > 2.16.2.windows.1 >=20 >=20 >=20