From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mx.groups.io with SMTP id smtpd.web11.3411.1589498171551144775 for ; Thu, 14 May 2020 16:16:11 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=A5LxD+Gr; spf=pass (domain: intel.com, ip: 134.134.136.126, mailfrom: nathaniel.l.desimone@intel.com) IronPort-SDR: BisnUq9DML6XjHu59g0WfKwAmsZLkxI4iAYMaziWD/iKnlrOVxdD5vai4c0aYiRFq8+7ZVDUNU vVhdrZgsSVIQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2020 16:16:10 -0700 IronPort-SDR: JQOlVW806mkJ7KWQYyEIF6uSNES+y5zoa3uGCiYj+QgwgxSgXXO/qPADa7prKGEbWS62gmBkRo pVzwTHY3imxg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,392,1583222400"; d="scan'208";a="266409622" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga006.jf.intel.com with ESMTP; 14 May 2020 16:16:10 -0700 Received: from fmsmsx607.amr.corp.intel.com (10.18.126.87) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 16:16:10 -0700 Received: from fmsmsx607.amr.corp.intel.com (10.18.126.87) by fmsmsx607.amr.corp.intel.com (10.18.126.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 14 May 2020 16:16:09 -0700 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx607.amr.corp.intel.com (10.18.126.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 14 May 2020 16:16:09 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.103) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 16:15:53 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gnFSmYoFx/ZMe8Cx6cKVmdDvql9OTkAAD+y+btDNgSf1Dopm+xFc5O8aCrUz70S3E9Fk+sb08C9Qpu+PwDmRe9jiRayLGSE88Kwi0CwRsHroRYNP/0DMYQPLmxcfbT6KeeaC+8pwdL2m7yVa32l+m/9leip5Yzt1BLEWyuZfrj4bEfJ6WYxdP7tflU8VtjilL/YRHYinFeJlJ3CDIPioXHGOfhqVUJLGBX0jSVfbyR1SFwmubREztPBNEPB8ShiuBUwGwsCowqFjyxKXvP6OGNyB4RVW/L1tdi/LsCrWPvLwHzstk94L7ppjPEuZrToxTjvCSEn5r28Ic+sM77ykPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LoTA5Ag05rVgS3IRECbxEDQ9APba8UWnPBWFWv9rhss=; b=lV/ur1U9RBnxb/5E5NrbfTHaH9Puze5rtxor2n4rd850hjxrJMDJUaFhGx2Wf8BrFpWWMr5K4nhtna3d3shHSVR6TI4psn7o9rtxLcW8WUUe+ngwzAetTq6Wn1vbWXJgOARWhtFfebvSyVf1aRopBiU117OWiLNFffXQ4QymYNbvADVKHAmUF5/3jimS6cx1GaIr38nnZyyrixohJ3F3me3R4yRZ1zRtOdOk3ySDwODZPhUeWmxr6JnNOMv8Zkjb+gOfJl8h5x0w1PhjPhccK4z5H1iMXvIEi4iEWxGKzHgSRDQ/BnHtpfud2OK3nUGLtsZdm918PLAriEP6tChOVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LoTA5Ag05rVgS3IRECbxEDQ9APba8UWnPBWFWv9rhss=; b=A5LxD+GriBr7p1Fv9r/94h3Z5Fv6wzj1eHHzC1KBcks0am6N+mrE5LkpX6clIonujW74BJ9lKDpksF7EpqZ7GNf5pFLS5L1HVtGvWouS5kTbW0RdGQZhW0Zsx9OchFZmdRxlzbFdJCsvG+cklRkPbHL4DVWffYWgfKb7w/9x1qw= Received: from BL0PR11MB3489.namprd11.prod.outlook.com (2603:10b6:208:75::29) by BL0PR11MB2929.namprd11.prod.outlook.com (2603:10b6:208:75::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3000.25; Thu, 14 May 2020 23:15:52 +0000 Received: from BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42]) by BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42%7]) with mapi id 15.20.2979.033; Thu, 14 May 2020 23:15:52 +0000 From: "Nate DeSimone" To: "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Pandya, Puja" , "Bjorge, Erik C" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Topic: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Index: AQHWJ7AV7yRUXZBlnkO4cccfBRQT8aioO7zQ Date: Thu, 14 May 2020 23:15:51 +0000 Message-ID: References: <20200511161954.13940-1-ashley.e.desimone@intel.com> In-Reply-To: <20200511161954.13940-1-ashley.e.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiY2Q1MzJhMDMtMGJkNS00OTc1LWIzZjQtNGVmZjdmNDU0ZmJiIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoibjA4R0tjUERYOU5PWkRWbWVrZzM4Ulk4NU5hWERmU1ZuaE1BMVJDMEJONU5SVXhHbTlkRGl0bUVOSWdQajF3RCJ9 dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows x-ctpclassification: CTP_NT authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3da15f74-d704-43e8-f561-08d7f85cbf78 x-ms-traffictypediagnostic: BL0PR11MB2929: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1443; x-forefront-prvs: 040359335D x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8sJ1D0QbI7pJWGbiLTR3MHxnugU3i97E4qctKoG6QJ+orl/DgKNQbd+F0Oc6Nuduf6GZE1QX/NUr2fuBTRsDOvWcT4NtvcI6a31367pRx465dy+gARkVIrmlPUQpVNUFxyC8jOS30BjOHWD1lKUlt7wjLPw08QhWz4+m8u2GNO9LAHi3z4+s0OTjWRd0Gfj6Yn1mKDpudL+6+ZjQxiAM8RfjghaQFlTiWspWN4U+bN9WU1IM/oSUIes87OIbJbMgISSIWYR7iYg4YTehxzotQYlr5OMDxfD3NP6uM5gCd4lk/Bq0j8IK107Sxa6fgOeXHaLSKPXCBQ95Bjh1u/pqjujb0U63bN4tsGgHIoT9PS+4KJ5QuWJzdHnIWrkqIaKp9apO8Kkg/PhjDvjS7slkbC2Z3vL285GvqfCLlTQAEPkFl514F6HpF0YZRSwZpoUe x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR11MB3489.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(396003)(39860400002)(366004)(346002)(376002)(136003)(55016002)(4326008)(9686003)(7696005)(107886003)(33656002)(86362001)(5660300002)(8676002)(8936002)(52536014)(54906003)(2906002)(6506007)(15650500001)(478600001)(53546011)(76116006)(66476007)(66446008)(66556008)(64756008)(71200400001)(26005)(110136005)(66946007)(316002)(186003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: FHt/+vmWT4VG9gHGfsa8zlffyTtBxTOCD9WKXkX6zZlK7YmRgvW0bpojPrFaOBAdnYQZ6D7N+NVWQYbgOOTewE4fZ9R8yb2NQerfjKQ/y4cjKCuClvMYmiWCiY5OocMoJsQENo0n2dpctoYUcmdEb+SQU2TosjBXK72KTFF291S3XsAyTMwtcs8Kkobkbn23ePf2uDrdJVRn6kyQsk6B7luILTOjTZgNgTPH6AjpgyNxGhD9kOse5xKHCKzdCHhtwMH2rTnxIQfuEVhNwlZ4NK4E1TQCE/hssiZzojdqv29E7cpvQFjyYfpP/WPmB15YlRBSs0MfA0ba81SQAEFciYpyd3ljLZrc95BqQYItYfvmowntxZtuqS9Odx4EexZAl9XCHFegxJqNwfzsNOdzCmJtV3sBq8BdHZdaTtvKkMasxhTq8UyYzAjSFxk4oLjs5oN7E73O4VS3l6oWotPYPfTwDxXfPoIOIHaEu0XgMu01T/JX4oGqSjaqtXSRKUO7 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 3da15f74-d704-43e8-f561-08d7f85cbf78 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 May 2020 23:15:52.4322 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: MFJW+xkoLOpBXCH/jEQxRuFlt6A1Nk3unPiQPcjgfUwQJuK2wuF/YNoYJSxAm4iPfTzFP/Dy8b4r3L6Dr9ExbAvgZyaOaseGhw7GSIoU3UA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB2929 Return-Path: nathaniel.l.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Pushed: 4ae181e1 > -----Original Message----- > From: Desimone, Ashley E > Sent: Monday, May 11, 2020 9:20 AM > To: devel@edk2.groups.io > Cc: Desimone, Nathaniel L ; Pandya, Puja > ; Bjorge, Erik C ; Bret > Barkelew ; Agyeman, Prince > > Subject: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to > support multiple manifest repositories >=20 > Update the list repos command to support calculating the list of consumed > repositories from all manifests in all manifest repositories defined in b= oth the > edkrepo.cfg and the edkrepo.cfg >=20 > Signed-off-by: Ashley E Desimone > Cc: Nate DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Cc: Prince Agyeman > --- > edkrepo/commands/list_repos_command.py | 97 > +++++++++++++++++++++++++--------- > 1 file changed, 72 insertions(+), 25 deletions(-) >=20 > diff --git a/edkrepo/commands/list_repos_command.py > b/edkrepo/commands/list_repos_command.py > index b06a493..b2088c0 100644 > --- a/edkrepo/commands/list_repos_command.py > +++ b/edkrepo/commands/list_repos_command.py > @@ -18,9 +18,11 @@ from edkrepo.commands.edkrepo_command import > EdkrepoCommand from edkrepo.commands.edkrepo_command import > ColorArgument import edkrepo.commands.arguments.list_repos_args as > arguments import edkrepo.commands.humble.list_repos_humble as > humble -from edkrepo.common.common_repo_functions import > pull_latest_manifest_repo from edkrepo.common.edkrepo_exception > import EdkrepoInvalidParametersException, > EdkrepoManifestInvalidException from edkrepo.common.ui_functions > import init_color_console > +from > edkrepo.common.workspace_maintenance.manifest_repos_maintenance > +import pull_all_manifest_repos from > +edkrepo.common.workspace_maintenance.manifest_repos_maintenance > import > +list_available_manifest_repos from edkrepo.config.tool_config import > +CI_INDEX_FILE_NAME > from edkrepo_manifest_parser.edk_manifest import CiIndexXml, > ManifestXml >=20 > class ListReposCommand(EdkrepoCommand): > @@ -52,35 +54,80 @@ class ListReposCommand(EdkrepoCommand): > print() > init_color_console(args.color) >=20 > - # Get path to global manifest file > - global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_local_path > - if args.verbose: > - print(humble.MANIFEST_DIRECTORY) > - print(global_manifest_directory) > - print() > - index_path =3D os.path.join(global_manifest_directory, 'CiIndex.= xml') > - > - pull_latest_manifest_repo(args, config) > + pull_all_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > print() >=20 > - #Create a dictionary containing all the manifests listed in the = CiIndex.xml > file > - ci_index_xml =3D CiIndexXml(index_path) > + cfg_manifest_repos, user_config_manifest_repos, conflicts =3D > + list_available_manifest_repos(config['cfg_file'], > + config['user_cfg_file']) > + > + found_manifests =3D {} > manifests =3D {} > repo_urls =3D set() > - project_list =3D list(ci_index_xml.project_list) > - if args.archived: > - project_list.extend(ci_index_xml.archived_project_list) > - for project in project_list: > - xml_file =3D ci_index_xml.get_project_xml(project) > - manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > - manifests[project] =3D manifest > - combo_list =3D [c.name for c in manifest.combinations] > + config_manifest_repos_project_list =3D [] > + user_config_manifest_repos_project_list =3D [] > + > + for manifest_repo in cfg_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in = the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + print(index_path) > + ci_index_xml =3D CiIndexXml(index_path) > + config_manifest_repos_project_list =3D > + ci_index_xml.project_list > if args.archived: > - combo_list.extend([c.name for c in > manifest.archived_combinations]) > - for combo in combo_list: > - sources =3D manifest.get_repo_sources(combo) > - for source in sources: > - repo_urls.add(self.get_repo_url(source.remote_url)) > + > config_manifest_repos_project_list.extend(ci_index_xml.archived_project_ > list) > + for project in config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + repo_urls.add(self.get_repo_url(source.remote_ur= l)) > + for manifest_repo in user_config_manifest_repos: > + # Get path to global manifest file > + global_manifest_directory =3D > config['user_cfg_file'].manifest_repo_abs_path(manifest_repo) > + if args.verbose: > + print(humble.MANIFEST_DIRECTORY) > + print(global_manifest_directory) > + print() > + #Create a dictionary containing all the manifests listed in = the > CiIndex.xml file > + index_path =3D os.path.join(global_manifest_directory, > CI_INDEX_FILE_NAME) > + ci_index_xml =3D CiIndexXml(index_path) > + user_config_manifest_repos_project_list =3D ci_index_xml.pro= ject_list > + if args.archived: > + > user_config_manifest_repos_project_list.extend(ci_index_xml.archived_pr > oject_list) > + for project in user_config_manifest_repos_project_list: > + xml_file =3D ci_index_xml.get_project_xml(project) > + manifest =3D > ManifestXml(os.path.normpath(os.path.join(global_manifest_directory, > xml_file))) > + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest > + combo_list =3D [c.name for c in manifest.combinations] > + if args.archived: > + combo_list.extend([c.name for c in > manifest.archived_combinations]) > + for combo in combo_list: > + sources =3D manifest.get_repo_sources(combo) > + for source in sources: > + > + repo_urls.add(self.get_repo_url(source.remote_url)) > + > + #Remove the manifest repo portion of the key is there is not a > duplicate project name > + key_list =3D list(found_manifests) > + for entry in key_list: > + new_key =3D entry.split(':')[1] > + value =3D found_manifests[entry] > + del found_manifests[entry] > + for found_manifest in list(found_manifests): > + if found_manifest.split(':')[1] =3D=3D new_key: > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':')[0], entry.split(':')[1]) > + break > + if new_key in manifests.keys(): > + new_key =3D 'Manifest Repository: {} Project: > {}'.format(entry.split(':'[0]), entry.split(':')[1]) > + manifests[new_key] =3D value >=20 > #Sort the manifests so projects will be displayed alphabetically > manifests =3D collections.OrderedDict(sorted(manifests.items())) > -- > 2.16.2.windows.1