From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web10.3406.1589498025872072729 for ; Thu, 14 May 2020 16:13:46 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=UBZNPJs+; spf=pass (domain: intel.com, ip: 134.134.136.20, mailfrom: nathaniel.l.desimone@intel.com) IronPort-SDR: 734zex6EDIOdlMdxDxMQR8+TmCiajLHbrDvKthn9CmsQxmDxSu0PCMeRWJJ+6CTw+rv5SS5gJY +8JLhMCfxVOg== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2020 16:13:45 -0700 IronPort-SDR: 2viMzUTnBbn6wQ2zdTQNJYexyM7n96uWDTp4AhYjgpo5RgJ0jdfJvbM3i3ikY83hd22qyJ+ZhA pEEY9a8Pkk7w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,392,1583222400"; d="scan'208";a="341778530" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by orsmga001.jf.intel.com with ESMTP; 14 May 2020 16:13:44 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 16:13:44 -0700 Received: from fmsmsx604.amr.corp.intel.com (10.18.126.84) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 14 May 2020 16:13:42 -0700 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 14 May 2020 16:13:42 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.173) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 16:13:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Clj5H0OZx43RRMfSsP03J9DPlqoNyNQ70JlNeRaJzEkpvBykwhEOjmNV8EEjNZ232aF9C5MdIU0aE+49dnkyXlXCX42FAy3UEXowkfbME0YeQWr1Zcwx0TJyIqZQCf6ZiLvTbM6Ml2t+JfWl4wyoDA/x+gPdNazjKyP5S9eS+b6XKwtRQQVPFwUImEGiYjXFnSVL+qy+IbWe89PhO5eY6fYXMBAK2C6ar6IJKm+76ls24qXuYCeWgKqgPDmkgcE81pn/yZB6Bz9eGHiEK8GeLQz3vYDwWQAXNZ/VpTzMr+uYQLGj3PK+gJEsQ6c/BHNi3lFTSxUxA8Mut3nalT8bYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IGVDqv2rOzdHJC9vV+XQ06fpfu/pDdu2bxThtHGEVcU=; b=XCXwB584R3I7sxsXr7YLTQQr3ZE/tu+KxJmoqoEG21bTxp5tyGBR6nUxmV0O6epvRwUhodHcSttkHs/0yLRFAGjAEcnbcBdU+ILCpkVxuNfKXPHdMzb84bJ16u23ks/g56515yx2WwuL7DoOB6v4FtPfgTmzi4owFTVBktygJ2InpTn6GuXXRH/v7EPFzrBpQXtbgFGl1fNDsWNyiIN6i4ZkRVUDMq4F6NY2BcRg6ipY5dMwj3w1mZk0V39JvjvcSuaVvQkHXn8Ebwop8GSmSvMyQUnKtsPEUZSxpx/c24u/+gl2vktayGDg4ad/lYJvr5JG2mIE0oxAwH877tD/HQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IGVDqv2rOzdHJC9vV+XQ06fpfu/pDdu2bxThtHGEVcU=; b=UBZNPJs+NuwKHylWBp0sllNYPY7sTge5RLlbzKEBQZ2/MFetURqekEtQ1dW4MGSwwOrv84BlWdVVLV2wGYXq0m90quQS9cKsNsSpdL7V71rMOytNRPGT9AYTxmQrcW78WdFBgziThigpiFiLv0GFOhgwkFhdiY1zk1XfX7GLX3A= Received: from BL0PR11MB3489.namprd11.prod.outlook.com (2603:10b6:208:75::29) by BL0PR11MB2929.namprd11.prod.outlook.com (2603:10b6:208:75::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.25; Thu, 14 May 2020 23:13:28 +0000 Received: from BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42]) by BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42%7]) with mapi id 15.20.2979.033; Thu, 14 May 2020 23:13:28 +0000 From: "Nate DeSimone" To: "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Pandya, Puja" , "Bjorge, Erik C" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Topic: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Index: AQHWJ7AV7yRUXZBlnkO4cccfBRQT8aioOwrA Date: Thu, 14 May 2020 23:13:28 +0000 Message-ID: References: <20200511161954.13940-1-ashley.e.desimone@intel.com> In-Reply-To: <20200511161954.13940-1-ashley.e.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYmRiNTVkODYtOTE3ZS00N2NjLWJhOTQtNmExZjJmOGJhMzEyIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiaENFNFJGK0pvajRPRDZUbEQxRGNkUmpIckVDNG1jNFdnMUtDZWhCWkZCM0F1eXlKSkhZSW1hb1wvQ0c5RmJwY1MifQ== dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows x-ctpclassification: CTP_NT authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b819979a-98e6-4e6a-9795-08d7f85c698f x-ms-traffictypediagnostic: BL0PR11MB2929: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1443; x-forefront-prvs: 040359335D x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Lh0yAOtooay92IoaiWGawB0635d3Wt2oaMlHqBw8WCPIIUFjxmNi52as9o/HTa6tE1uVkz2U1nCHIGRIkdYTb8CSWQG9e/O7UIn8vsP185smLSQnZb8a/OOAb2RzkTEhorQ6zNcl9oTwiBrt7CziC8U5QIJV0GoYwO41lh3IhaWjVE3WZaAa9+K6Dem4dzibK6URp6myYCpC+4mULPFjpWVHmMEaNJ+EF80pSU8yhMS0+7iOtaoulYDKJgM/Mz7YAQqxGV63VcKNu2g5LtpMrd0POfAz4mmDURzf0NZ3v0QZfjftPnZT9y575wE7v8tNpurbNO5lpme7LdhU14t1lAHsQoVpbgiZb5TZSlEF9i2fgiB2MbRdAAzBGE9GViz7oVUdCyFNhMzo0xfrM5I7ZIxI7ZQTkaaeDHgkzidIYTMYzWNEdTjniWCxqHLmPo3C x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR11MB3489.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(396003)(39860400002)(366004)(346002)(376002)(136003)(55016002)(4326008)(9686003)(7696005)(107886003)(33656002)(86362001)(5660300002)(8676002)(8936002)(52536014)(54906003)(2906002)(6506007)(15650500001)(478600001)(53546011)(76116006)(66476007)(66446008)(66556008)(64756008)(71200400001)(26005)(110136005)(66946007)(316002)(186003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: uQgDgOTTghnwHU4cNPpcArhkwn6hM6mh13WS9cno/gJl6HIO2Sco0s9kFCMXZNOaCHlly+iJ+NoqFkhqb+6ueaxpOCVjLdoOQksN5N9/nXtglG+M0G9Jzfpv1RwVbwyPwNzkOhXiqv4dF1udl/r9Q5a5k8eqSb0h0vQNB/Y05ZNFBFnj2SDS0n2xEEyb1aFpFY51jTQgFbyyq/r7qjxC44ClaZz1xnz0ipSrHhWA8253zW2kX1zcBMphOpY26jok4Ps51GKW3TYGuM9Nw7V2OqrIUlOG6I5Hw/aTC4f6eikG53DG7FUBM72EVaMeIVliAqjxqLFobMNAsLGh/8T5J0dGdOxdPHqafcge8MPar7EPlh7njG1GYHp/pfTBOEY8H4ahnJ+fBcl5FtMLk+4+s7+E25WlAVSDUPIzBI2/2CMwkRbN1Pvh4Gb6zVOInCPS8PRJ0TzJeLLHvNdGZosO63Eso8H+ydxtbTRaA8bLYV/7lV5QPUhhr0evzFAY+aYi MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: b819979a-98e6-4e6a-9795-08d7f85c698f X-MS-Exchange-CrossTenant-originalarrivaltime: 14 May 2020 23:13:28.5483 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ayWyOI+yYz0D8KJfZv1SkjVCUKTChmCBryl3OloujvOxwa2FyMR8AnlJUsxLDtxtafy9pFpB81hjQM8kl49LSyAR+8JUbvqcRSVQhoY1eis= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB2929 Return-Path: nathaniel.l.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Nate DeSimone > -----Original Message----- > From: Desimone, Ashley E > Sent: Monday, May 11, 2020 9:20 AM > To: devel@edk2.groups.io > Cc: Desimone, Nathaniel L ; Pandya, Puja > ; Bjorge, Erik C ; Bret > Barkelew ; Agyeman, Prince > > Subject: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to > support multiple manifest repositories >=20 > Update the list repos command to support calculating the list of consumed > repositories from all manifests in all manifest repositories defined in b= oth the > edkrepo.cfg and the edkrepo.cfg >=20 > Signed-off-by: Ashley E Desimone > Cc: Nate DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Cc: Prince Agyeman > --- > edkrepo/commands/list_repos_command.py | 97 > +++++++++++++++++++++++++--------- > 1 file changed, 72 insertions(+), 25 deletions(-) >=20 > diff --git a/edkrepo/commands/list_repos_command.py > b/edkrepo/commands/list_repos_command.py > index b06a493..b2088c0 100644 > --- a/edkrepo/commands/list_repos_command.py > +++ b/edkrepo/commands/list_repos_command.py > @@ -18,9 +18,11 @@ from edkrepo.commands.edkrepo_command import > EdkrepoCommand from edkrepo.commands.edkrepo_command import > ColorArgument import edkrepo.commands.arguments.list_repos_args as > arguments import edkrepo.commands.humble.list_repos_humble as > humble -from edkrepo.common.common_repo_functions import > pull_latest_manifest_repo from edkrepo.common.edkrepo_exception > import EdkrepoInvalidParametersException, > EdkrepoManifestInvalidException from edkrepo.common.ui_functions > import init_color_console > +from > edkrepo.common.workspace_maintenance.manifest_repos_maintenance > +import pull_all_manifest_repos from > +edkrepo.common.workspace_maintenance.manifest_repos_maintenance > import > +list_available_manifest_repos from edkrepo.config.tool_config import > +CI_INDEX_FILE_NAME > from edkrepo_manifest_parser.edk_manifest import CiIndexXml, > ManifestXml >=20 > class ListReposCommand(EdkrepoCommand): > @@ -52,35 +54,80 @@ class ListReposCommand(EdkrepoCommand): > print() > init_color_console(args.color) >=20 > - # Get path to global manifest file > - global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_local_path > - if args.verbose: > - print(humble.MANIFEST_DIRECTORY) > - print(global_manifest_directory) > - print() > - index_path =3D os.path.join(global_manifest_directory, 'CiIndex.= xml') > - > - pull_latest_manifest_repo(args, config) > + pull_all_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > print() >=20 > - #Create a dictionary containing all the manifests listed in the = CiIndex.xml > file > - ci_index_xml =3D CiIndexXml(index_path) > + cfg_manifest_repos, user_config_manifest_repos, conflicts =3D > + list_available_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > + > + found_manifests =3D {} > manifests =3D {} > repo_urls =3D set() > - project_list =3D list(ci_index_xml.project_list) > - if args.archived: > - project_list.extend(ci_index_xml.archived_project_list) > - for project in project_list: > - xml_file =3D ci_index_xml.get_project_xml(project) > - manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > - manifests[project] =3D manifest > - combo_list =3D [c.name for c in manifest.combinations] > + config_manifest_repos_project_list =3D [] > + user_config_manifest_repos_project_list =3D [] > + > + for manifest_repo in cfg_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in = the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + print(index_path) > + ci_index_xml =3D CiIndexXml(index_path) > + config_manifest_repos_project_list =3D > + ci_index_xml.project_list > if args.archived: > - combo_list.extend([c.name for c in > manifest.archived_combinations]) > - for combo in combo_list: > - sources =3D manifest.get_repo_sources(combo) > - for source in sources: > - repo_urls.add(self.get_repo_url(source.remote_url)) > + > config_manifest_repos_project_list.extend(ci_index_xml.archived_project_ > list) > + for project in config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + repo_urls.add(self.get_repo_url(source.remote_ur= l)) > + for manifest_repo in user_config_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['user_cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in = the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + ci_index_xml =3D CiIndexXml(index_path) > + user_config_manifest_repos_project_list =3D ci_index_xml.pro= ject_list > + if args.archived: > + > user_config_manifest_repos_project_list.extend(ci_index_xml.archived_pr > oject_list) > + for project in user_config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + > + repo_urls.add(self.get_repo_url(source.remote_url)) > + > + #Remove the manifest repo portion of the key is there is not a > duplicate project name > + key_list =3D list(found_manifests) > + for entry in key_list: > + new_key =3D entry.split(':')[1] > + value =3D found_manifests[entry] > + del found_manifests[entry] > + for found_manifest in list(found_manifests): > + if found_manifest.split(':')[1] =3D=3D new_key: > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':')[0], entry.split(':')[1]) > + break > + if new_key in manifests.keys(): > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':'[0]), entry.split(':')[1]) > + manifests[new_key] =3D value >=20 > #Sort the manifests so projects will be displayed alphabetically > manifests =3D collections.OrderedDict(sorted(manifests.items())) > -- > 2.16.2.windows.1