From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mx.groups.io with SMTP id smtpd.web11.1435.1589491476856309830 for ; Thu, 14 May 2020 14:24:37 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=vkvzrCDS; spf=pass (domain: intel.com, ip: 134.134.136.126, mailfrom: erik.c.bjorge@intel.com) IronPort-SDR: ydOTx1/V+oMDuoK3CSC5QPEih5YE15mvFtmApyxTMfaIX8Yb/lP/a6m1S4X8pITgx6alZVRJxl AH4S0pQbL+Fw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2020 14:24:36 -0700 IronPort-SDR: fQ/QQtiVRWnHth8GSVDNu7smzRLuE31B1RBYy0F8ESV+VPcwVyfiz7QQUgrEC1h+Ynm4EBMCK8 18VHpVSaDllQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,392,1583222400"; d="scan'208";a="251811828" Received: from orsmsx105.amr.corp.intel.com ([10.22.225.132]) by orsmga007.jf.intel.com with ESMTP; 14 May 2020 14:24:36 -0700 Received: from orsmsx157.amr.corp.intel.com (10.22.240.23) by ORSMSX105.amr.corp.intel.com (10.22.225.132) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 14:24:36 -0700 Received: from ORSEDG001.ED.cps.intel.com (10.7.248.4) by ORSMSX157.amr.corp.intel.com (10.22.240.23) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 14:24:35 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.100) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 14:24:35 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A4Y24hcoyCvvafkIW2z1d5ffOHEKhpNumNWyG2EsshkF2xCXZ5V25GTUeLHPc9eTNiQX+oiI2xgS7B29CAI2Ay8SsLP3uelKOl+JNcxLfbfNHmPsBNUYotXPflIXXYHcQS/6Vb+ivOxTjI+y+ixWdhYkICWhVgxSIVz8IGdBs7J7nSu7eOtZNmkoqwt+nGsAg/ti/jkd3gQKWTZn+MMtVr6F3Z9GaEzUDve1Oq6Z0vaYfyUVXlDyQpfsWIKsb1Ecuhg5c0/JvKbOXfAJC0mf2e8qv2O099dDBtEcBbg+74rD5DimFxPOIvfNoadA/3jJ3+cl7O0Y2+PcOXTbXkI+Qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sdYcj9Ym+lPJHDQ76aKxP2BBFK71NP4Z7MPG40sOUHk=; b=ULm6VVm8z/bxxPNS1pGoPw5eG+MsxfSADMLMuSoQ33ArmRY23Pm76gHqF1ZTXinI5WJKWv2A4WnL/LdI0C7QT788Av4L1MC27tnW+9W5zk//LfhV+VaWTfqbEeRlrOQnMhK4iYM/lJF/w2ximZslZG6lk0R10jf0NslF6nXKSiQKT1kvpOXwpcX4MPOoMdDQSEzXtHVrkP1q3c7h3KTJ4YmK9Od8pGVtBkg4znTkFz1y44SLNHVwD+OE8GZGkW+1i1a3hggM7cRS3toahEIUupdQwWrpaxYejDlr+WmZ2v3eGsTI1jnBZKKz9vVykUkjvGNSiftc842rLuD0cvtgqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sdYcj9Ym+lPJHDQ76aKxP2BBFK71NP4Z7MPG40sOUHk=; b=vkvzrCDSASAFJ5+sBMOCR9LGzPQ+DnveeHqYPgIJk4TLdh6VFdFACYPZ//ZnwaTgvbwfkgNdpcZUanTZTyV+tgmMoLefvBHE6jWv6ipnUOZLlJc/v397YG/aXJq9Dv4inDzBldT4nJ7vvTjDDMYAFc46YfE0LPMDptvXo6jxw0c= Received: from MW3PR11MB4554.namprd11.prod.outlook.com (2603:10b6:303:5d::7) by MW3PR11MB4729.namprd11.prod.outlook.com (2603:10b6:303:5d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2979.28; Thu, 14 May 2020 21:24:34 +0000 Received: from MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281]) by MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281%6]) with mapi id 15.20.3000.022; Thu, 14 May 2020 21:24:34 +0000 From: "Bjorge, Erik C" To: "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Topic: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to support multiple manifest repositories Thread-Index: AQHWJ7AVMUsC4FRBiUy/yvp2JRgozKioHLGg Date: Thu, 14 May 2020 21:24:34 +0000 Message-ID: References: <20200511161954.13940-1-ashley.e.desimone@intel.com> In-Reply-To: <20200511161954.13940-1-ashley.e.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.52.205] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 41358767-fbbf-4d0d-d7af-08d7f84d3308 x-ms-traffictypediagnostic: MW3PR11MB4729: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1443; x-forefront-prvs: 040359335D x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yUSsJl9rTCTjGwaoTAGhljz4/nKJqvv+IoN8OHIX9X3mPoljpntDbIgwylIboQwHnXaVEfAHP0ruJZhNjSnCZLmsa4vIwGP0p8N54OTVg4CjIK62jAbChOGVQ/ho28Pl7+o1lR7V6KzgtVf3lc+VVr5CTlWL5Auwj3HyofoHlRvfQJjpUEOApzTduYT5qxqhlyPCRY4MevJiFvh85b1IE6K3BdZh861bRmWjZbHuMtcNwmrr3Tcafe2ny2J7wOqp/y7szGCfNpgatdLtXATcN22IQSnaPEEZHKugS7+2aSZxpSdZtbvCzWHJ6XWZjQd98MN3RiKZAkj7A/uu9ao25yNNbb+0L2rQmqnxEKgt74MxzsBCR9H1g672EehRMxk49kVyWGUPsSoKkeYdSiVOHUYdq8cUKRgEaVnJsiNxlURF+k39yO8uqJg6C6Yl/5xg x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW3PR11MB4554.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(366004)(376002)(396003)(136003)(346002)(39860400002)(66946007)(86362001)(66446008)(316002)(64756008)(7696005)(66556008)(52536014)(53546011)(6506007)(76116006)(110136005)(15650500001)(26005)(186003)(66476007)(2906002)(4326008)(9686003)(54906003)(107886003)(478600001)(33656002)(71200400001)(5660300002)(8936002)(55016002)(8676002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: u97QtPsejxUZJf+5Uz9MXJTVGlNZzA8i2cBCBOFTrjsIob3B8IxXHza9zzIiR7nAR1aj+he5/QLC+mmoad8v9dM1ovTkZzcYmmx6RN82rrHnGy4QDJPrP7yr5amKzu+9TWgP2r3ZLrU9s/aq4dbf8u89KvZk1oid8e6zj6N5x9ks1NwRi9JWn53NBB0ND1fCdgndd6ulFKhFu1g5D7PmKzZtomrjl6NQ0PMFqsc5inZvTPoNRi6LhzabN1E3xZP+eM734ZwdVi6FGnwGtCnX/FOlJjQZx57g5SaY5I/S398LDYjW4Zw4tA02s7F9dY3eU3rzS9PnKpd/mzbCSq4W97LmUeyb8+3++NQfqEnc5irXOkPBUSO+pNGgVHCSyIUaQy5QlkKyNeqnka6I7Oa8LqJ8x9Uw7Gvp3iuQRgrss/en+AZPfgbXa16mWEPzO5pbtIE6oKcxNzmnGwp+vcJaye0ZUgL5NUna7Ssv2dK8Kam0GfB7xnd7gzAl3Lx+KTCl MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 41358767-fbbf-4d0d-d7af-08d7f84d3308 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 May 2020 21:24:34.5894 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /yJSHTfdcquC+1Y1swcO5UQkKXw2JnNwWKKZjKcUP6QF/gNIP5TlL2DVOnWbGhpnczp7WZv24sqR84gDPlDRVQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4729 Return-Path: erik.c.bjorge@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Erik Bjorge -----Original Message----- From: Desimone, Ashley E =20 Sent: Monday, May 11, 2020 9:20 AM To: devel@edk2.groups.io Cc: Desimone, Nathaniel L ; Pandya, Puja ; Bjorge, Erik C ; Bret Barke= lew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Update list-repos to su= pport multiple manifest repositories Update the list repos command to support calculating the list of consumed r= epositories from all manifests in all manifest repositories defined in both= the edkrepo.cfg and the edkrepo.cfg Signed-off-by: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Erik Bjorge Cc: Bret Barkelew Cc: Prince Agyeman --- edkrepo/commands/list_repos_command.py | 97 +++++++++++++++++++++++++-----= ---- 1 file changed, 72 insertions(+), 25 deletions(-) diff --git a/edkrepo/commands/list_repos_command.py b/edkrepo/commands/list= _repos_command.py index b06a493..b2088c0 100644 --- a/edkrepo/commands/list_repos_command.py +++ b/edkrepo/commands/list_repos_command.py @@ -18,9 +18,11 @@ from edkrepo.commands.edkrepo_command import EdkrepoComm= and from edkrepo.commands.edkrepo_command import ColorArgument import edk= repo.commands.arguments.list_repos_args as arguments import edkrepo.comman= ds.humble.list_repos_humble as humble -from edkrepo.common.common_repo_func= tions import pull_latest_manifest_repo from edkrepo.common.edkrepo_excepti= on import EdkrepoInvalidParametersException, EdkrepoManifestInvalidExceptio= n from edkrepo.common.ui_functions import init_color_console +from edkrepo.common.workspace_maintenance.manifest_repos_maintenance=20 +import pull_all_manifest_repos from=20 +edkrepo.common.workspace_maintenance.manifest_repos_maintenance import=20 +list_available_manifest_repos from edkrepo.config.tool_config import=20 +CI_INDEX_FILE_NAME from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml =20 class ListReposCommand(EdkrepoCommand): @@ -52,35 +54,80 @@ class ListReposCommand(EdkrepoCommand): print() init_color_console(args.color) =20 - # Get path to global manifest file - global_manifest_directory =3D config['cfg_file'].manifest_repo_abs= _local_path - if args.verbose: - print(humble.MANIFEST_DIRECTORY) - print(global_manifest_directory) - print() - index_path =3D os.path.join(global_manifest_directory, 'CiIndex.xm= l') - - pull_latest_manifest_repo(args, config) + pull_all_manifest_repos(config['cfg_file'],=20 + config['user_cfg_file']) print() =20 - #Create a dictionary containing all the manifests listed in the Ci= Index.xml file - ci_index_xml =3D CiIndexXml(index_path) + cfg_manifest_repos, user_config_manifest_repos, conflicts =3D=20 + list_available_manifest_repos(config['cfg_file'],=20 + config['user_cfg_file']) + + found_manifests =3D {} manifests =3D {} repo_urls =3D set() - project_list =3D list(ci_index_xml.project_list) - if args.archived: - project_list.extend(ci_index_xml.archived_project_list) - for project in project_list: - xml_file =3D ci_index_xml.get_project_xml(project) - manifest =3D ManifestXml(os.path.normpath(os.path.join(global_= manifest_directory, xml_file))) - manifests[project] =3D manifest - combo_list =3D [c.name for c in manifest.combinations] + config_manifest_repos_project_list =3D [] + user_config_manifest_repos_project_list =3D [] + + for manifest_repo in cfg_manifest_repos: + # Get path to global manifest file + global_manifest_directory =3D config['cfg_file'].manifest_repo= _abs_path(manifest_repo) + if args.verbose: + print(humble.MANIFEST_DIRECTORY) + print(global_manifest_directory) + print() + #Create a dictionary containing all the manifests listed in th= e CiIndex.xml file + index_path =3D os.path.join(global_manifest_directory, CI_INDE= X_FILE_NAME) + print(index_path) + ci_index_xml =3D CiIndexXml(index_path) + config_manifest_repos_project_list =3D=20 + ci_index_xml.project_list if args.archived: - combo_list.extend([c.name for c in manifest.archived_combi= nations]) - for combo in combo_list: - sources =3D manifest.get_repo_sources(combo) - for source in sources: - repo_urls.add(self.get_repo_url(source.remote_url)) + config_manifest_repos_project_list.extend(ci_index_xml.arc= hived_project_list) + for project in config_manifest_repos_project_list: + xml_file =3D ci_index_xml.get_project_xml(project) + manifest =3D ManifestXml(os.path.normpath(os.path.join(glo= bal_manifest_directory, xml_file))) + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest + combo_list =3D [c.name for c in manifest.combinations] + if args.archived: + combo_list.extend([c.name for c in manifest.archived_c= ombinations]) + for combo in combo_list: + sources =3D manifest.get_repo_sources(combo) + for source in sources: + repo_urls.add(self.get_repo_url(source.remote_url)= ) + for manifest_repo in user_config_manifest_repos: + # Get path to global manifest file + global_manifest_directory =3D config['user_cfg_file'].manifest= _repo_abs_path(manifest_repo) + if args.verbose: + print(humble.MANIFEST_DIRECTORY) + print(global_manifest_directory) + print() + #Create a dictionary containing all the manifests listed in th= e CiIndex.xml file + index_path =3D os.path.join(global_manifest_directory, CI_INDE= X_FILE_NAME) + ci_index_xml =3D CiIndexXml(index_path) + user_config_manifest_repos_project_list =3D ci_index_xml.proje= ct_list + if args.archived: + user_config_manifest_repos_project_list.extend(ci_index_xm= l.archived_project_list) + for project in user_config_manifest_repos_project_list: + xml_file =3D ci_index_xml.get_project_xml(project) + manifest =3D ManifestXml(os.path.normpath(os.path.join(glo= bal_manifest_directory, xml_file))) + found_manifests['{}:{}'.format(manifest_repo, project)] = =3D manifest + combo_list =3D [c.name for c in manifest.combinations] + if args.archived: + combo_list.extend([c.name for c in manifest.archived_c= ombinations]) + for combo in combo_list: + sources =3D manifest.get_repo_sources(combo) + for source in sources: + =20 + repo_urls.add(self.get_repo_url(source.remote_url)) + + #Remove the manifest repo portion of the key is there is not a dup= licate project name + key_list =3D list(found_manifests) + for entry in key_list: + new_key =3D entry.split(':')[1] + value =3D found_manifests[entry] + del found_manifests[entry] + for found_manifest in list(found_manifests): + if found_manifest.split(':')[1] =3D=3D new_key: + new_key =3D 'Manifest Repository: {} Project: {}'.form= at(entry.split(':')[0], entry.split(':')[1]) + break + if new_key in manifests.keys(): + new_key =3D 'Manifest Repository: {} Project: {}'.format(e= ntry.split(':'[0]), entry.split(':')[1]) + manifests[new_key] =3D value =20 #Sort the manifests so projects will be displayed alphabetically manifests =3D collections.OrderedDict(sorted(manifests.items())) -- 2.16.2.windows.1