From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx.groups.io with SMTP id smtpd.web11.1072.1588282136694096616 for ; Thu, 30 Apr 2020 14:28:57 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=ZGv4P5FP; spf=pass (domain: intel.com, ip: 192.55.52.88, mailfrom: nathaniel.l.desimone@intel.com) IronPort-SDR: oCnfwy9LJjM80/rVg1/KeEt+6H46aVUdF/O3PQ8MbQno9WN3j85LUGytLXBbfq+gDSzBV5upLn bf4Ix7IWwLWw== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Apr 2020 14:28:56 -0700 IronPort-SDR: TGYM1lYR7SmlKHe/Id1bubYGrSr76BmavENouAi12CsSfEAya4iMSYcMk6dZM/q0ckb3dY0W63 9ymGNkkCYGYQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,337,1583222400"; d="scan'208";a="293700913" Received: from orsmsx102.amr.corp.intel.com ([10.22.225.129]) by fmsmga002.fm.intel.com with ESMTP; 30 Apr 2020 14:28:55 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX102.amr.corp.intel.com (10.22.225.129) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 30 Apr 2020 14:28:55 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 30 Apr 2020 14:28:54 -0700 Received: from ORSEDG001.ED.cps.intel.com (10.7.248.4) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 30 Apr 2020 14:28:54 -0700 Received: from NAM04-BN3-obe.outbound.protection.outlook.com (104.47.46.50) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 30 Apr 2020 14:28:53 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H1BYm7jS32OqKSIZkXZ5B1V7Ob7N2VB9gosI00dbC2JUFveSraY4y9kE2gidfHlsJQWOvMcUCJQc2jZmQooAOnmtZWo45Vz/WTCi/1K8AhJg3S67L3EpY4y/RXLz4lDalI3N6M6Cf0bscu2uQsINOdgs9ee5PDQtpnXoI9GHJZbfQkEulQZAn9nCnuptuocKf4J8v4Leuy7SfN0/J6daXuhR2w83nNRlwYhcnuv4waGNFjBvxUQJ3c+u7nsD+pJ8yaR0mU5LV9Y3JDwmcHbfj02ZUnl/j9tjCH9mWrzK6SF2070bkV6KeK9XPN+NkC6WGu1HSMROV2IwwtGeUmjU5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u1xOUfm6toBIdCMMF7vcxXGfB6seUU0IjR50mvwLpQI=; b=NKiL7IwYSSN4fGr6ywFOjfWPaz6P/dDIlgUQmRKuWWO5Qn5waoAsJMZuXDjbbkAk65eV5V4g7zNi6QAgEsiCMrrTyl8DgVlcs8abMonnswz6X7YxkRHhn1BC8sivTuyJg9xQS5YKeCGn2uolShRDBZBvxPmiqPR5kxKVDS/8YWph3NdxUrwQXCeWKBK9c5CC3iZoANbE7qGnFpcEfBmAfYojRQRLEqnqYJlo9/XdV87kfnO6+Et4TGh25/5yGNn+nNqM0oU8UifOMjDPkK/QETHkY51JJhVkwvmsDmfpEEszRwXvKOFfSwhstmHF6a/JynGbNRAQXA+UphiVMeb05w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u1xOUfm6toBIdCMMF7vcxXGfB6seUU0IjR50mvwLpQI=; b=ZGv4P5FP376Vffi1KwmeZUn7z4vCNEVjaQXIRWzVexoG6SsY3mJ/K0TlYgm8p1l6On56RfIbWyNIFabxrRM/X9UxVucRZZFPzXqk6kBI9b0LKQH50spVHU8F3ZDrbJdFVPCIdvFBWFye510Nm/VJm4+pRmuVXgTpqzjafk6grqc= Received: from BL0PR11MB3489.namprd11.prod.outlook.com (2603:10b6:208:75::29) by BL0PR11MB3075.namprd11.prod.outlook.com (2603:10b6:208:7a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.20; Thu, 30 Apr 2020 21:28:51 +0000 Received: from BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42]) by BL0PR11MB3489.namprd11.prod.outlook.com ([fe80::d0f8:bfd7:bae2:3e42%7]) with mapi id 15.20.2937.026; Thu, 30 Apr 2020 21:28:51 +0000 From: "Nate DeSimone" To: "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Pandya, Puja" , "Bjorge, Erik C" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to find projects across all manifest repositories Thread-Topic: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to find projects across all manifest repositories Thread-Index: AQHWHagVxJ91fJTjIEm7RoOnaGSrlqiSLlHA Date: Thu, 30 Apr 2020 21:28:51 +0000 Message-ID: References: <20200428215710.45504-1-ashley.e.desimone@intel.com> <20200428215710.45504-6-ashley.e.desimone@intel.com> In-Reply-To: <20200428215710.45504-6-ashley.e.desimone@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMGUyNjE3NjgtNzNlYi00NjQ5LWE4OTItZjlhNGQyY2Y1MDMxIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiSUg5WlJWaVVobXVmYU1MK25QMmR6Y3UyemtJTlQ4cXhOZmNpcjhyR1p2SG5oMzRyQVRJXC9jNnQ0SUJpMUpFVDkifQ== dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows x-ctpclassification: CTP_NT authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [134.134.136.198] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 65adf5ab-4d59-4aee-18f0-08d7ed4d7a70 x-ms-traffictypediagnostic: BL0PR11MB3075: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0389EDA07F x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: THS/FfGLz7WA29SVm6QJKz2mI1Llimo9+wSL7VbVLMalyiPKjKuynk/oV4wn2rs5U61bhJhBhc4XrHXaknQkdT9NJ3beFSYy5GdVnggrM2+ks5NJNbC/T3s9MwDNc3NWDCqOwtayQiQb63YY4aj6obV79O8W303g2rDLcMH3+vKDUsQqEUIXHMmOWqa2SKnhMaDVUPuzftSBOuMaSTld2ewVLXLtmPwQ1yyvTMpO2ToEfl8OB6utq/JdchwdTowGVQTqj2GjMIAkjBopxmFfxhCNogs5K2ByBRn8jFvPnPXivw14nCWyj27nBs7UnKaf48zbGoC1ai2+sYP4Ezbrja66Y/EAnqcZzhX+PW61jNhgbapMd9152fxIbC1EK3jsgzoYuGWbI4dy9l6k10BuOJTL5HRVTe0XfXbeniAHQJjwUdhpXJvkOfAiWw7EgzLmwkTsWrq/kjc/+oK74OP+aTFlBoo0G4z/9mtdXOLEULzzcS60AehOJrP9qvcsZyHM x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BL0PR11MB3489.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(136003)(376002)(366004)(346002)(39860400002)(396003)(478600001)(86362001)(6506007)(54906003)(71200400001)(26005)(53546011)(316002)(186003)(107886003)(5660300002)(110136005)(7696005)(4326008)(8936002)(66946007)(2906002)(66476007)(55016002)(64756008)(66446008)(66556008)(52536014)(33656002)(9686003)(76116006)(8676002)(37363001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: 7qGH8GnRu1VJcMARnFuf+arfuYlATalgA4kcmSBnnNngppIZ54eRIqhTnC+sNkwNqaZ5u8E8oAuPkKrADJiFhyAzwU8EGegRIjDQzEhxxR+j2or1WGqOH8H+jugp/jPCRBIxDBZK3Y5g2r4iv4ilBQCx7VVl+B70IFWPLEFBJG0ebO6BqlpzS6mu42X6ZfBG6n8FTpl/nPyC3gkTfIjAfCsqTEeVF/A6yTXyuMPeuT9ocQfq8YAF1oien3Ljt7j4cX9/tH2RtlkN9xiNj4dOFmEDfha9mDUBfIgtp54hv50r7ypbiSbTbOBsj6N8QTmFRI8a1jsO0DtmNvjlvinsgLwu34FxAMBNl1LVKlwfDwJDPhKRZmLa4AtIFwBmuzUXBikO28XFXZ4Scl/DxJtPoyjVrDh4fMSdUO4PvLc6OEKk5wV+v/Oe+ITOCpti0ZRG2OcNfPQAnPP7HE2uVnrHLRNzPgiGx8XFsfi8C0vyMwDHlQkTI37WsmUKCjZXPVX1rexalFgGlz7+pCRd9+Ck+bbvjaiPlETOMN/lcQmJibF3HAVI/QuvlGblNlYwBNT/08EdFFaix9hmvgx726RUrALlGkZuxFGgXBbdnULtSv47w8NZplZiBf0+vABDNAbxE3BGNUr7p9Ln/VWoR5ZzayiTEoCz3Zz+ReivJHHpieXy8seWO7jHtTK28zfxHFMa7apOvRcnnqNSAy2FJw1ZTHkhMmhtHMUWhtLu3GGURpOXwijg0+wVa0134TpgRXGmC6IYBpVmWumcJEqsAz7XVbariiQwS5m+g8/PbyMXSi1UaiV0ACSRd+k52HkOSaaD MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 65adf5ab-4d59-4aee-18f0-08d7ed4d7a70 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2020 21:28:51.7756 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: XERYa/56SCuyBtYj/d2wN4OllHG+il2NPuArJPjFJkOH+1Hb5k67+iE8bSBN82F/qFkNDGhRi2Wp+bwjHxI0QeTw3MGHdYRMBk9ZPrfxMzg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3075 Return-Path: nathaniel.l.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Ashley, Please see comments inline. Thanks, Nate > -----Original Message----- > From: Desimone, Ashley E > Sent: Tuesday, April 28, 2020 2:57 PM > To: devel@edk2.groups.io > Cc: Desimone, Nathaniel L ; Pandya, Puja > ; Bjorge, Erik C ; Bret > Barkelew ; Agyeman, Prince > > Subject: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to find > projects across all manifest repositories >=20 > Add find_project_in_all_indicies() to search for and return a tuple (sour= ce > repo, source config, path to manifest) if a matching project is found. >=20 > Add find_project_in_single_index() to find the path to a project within a > single manifest repositories index file and return a whether the specifie= d > project was found and if so its path within the manifest repository. >=20 > Signed-off-by: Ashley E Desimone > Cc: Nate DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Cc: Prince Agyeman > --- > .../manifest_repos_maintenance.py | 70 > +++++++++++++++++++++- > .../workspace_maintenance/workspace_maintenance.py | 17 +++++- > 2 files changed, 84 insertions(+), 3 deletions(-) >=20 > diff --git > a/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > b/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > index 4bded46..9b441ac 100644 > --- > a/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > +++ > b/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > @@ -15,11 +15,12 @@ import git > from git import Repo >=20 > import edkrepo.config.config_factory as cfg -from > edkrepo.common.edkrepo_exception import > EdkrepoUncommitedChangesException > +from edkrepo.common.edkrepo_exception import > +EdkrepoUncommitedChangesException, > EdkrepoInvalidParametersException > from edkrepo.common.progress_handler import GitProgressHandler > import > edkrepo.common.workspace_maintenance.humble.manifest_repos_maint > enance_humble as humble from > edkrepo.common.workspace_maintenance.workspace_maintenance import > generate_name_for_obsolete_backup > - > +from edkrepo.common.workspace_maintenance.workspace_maintenance > import > +case_insensitive_single_match from > edkrepo_manifest_parser.edk_manifest > +import CiIndexXml, ManifestXml >=20 > def pull_single_manifest_repo(url, branch, local_path, reset_hard=3DFals= e): > ''' > @@ -135,3 +136,68 @@ def list_available_man_repos(edkrepo_cfg, > edkrepo_user_cfg): > return cfg_man_repos, user_cfg_man_repos, conflicts >=20 >=20 > +def find_project_in_single_index (project, index_file, manifest_dir): > + ''' > + Finds a project in a single global manifest repositories index file.= If found > + returns (True, path to file) if not returns (False, None) > + ''' > + global_manifest_path =3D None > + try: > + proj_name =3D case_insensitive_single_match(project, > index_file.project_list) > + except: > + proj_name =3D None > + if proj_name: > + ci_index_xml_rel_path =3D > os.path.normpath(index_file.get_project_xml(proj_name)) > + global_manifest_path =3D os.path.join(manifest_dir, > ci_index_xml_rel_path) > + return True, global_manifest_path > + else: > + return False, global_manifest_path > + > + > +def find_project_in_all_indices (project, edkrepo_cfg, edkrepo_user_cfg, > except_msg_man_repo, except_msg_not_found, man_repo=3DNone): > + ''' > + Finds the project in all manifest repositories listed in the edkrepo= .efg and > + edkrepo_user.cfg. If a project with the same name is found uses > man_repo to select > + the correct entry > + ''' > + cfg_man_repos, user_cfg_man_repos, conflicts =3D > list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg) > + projects =3D {} > + for repo in cfg_man_repos: > + manifest_dir =3D edkrepo_cfg.manifest_repo_abs_path(repo) > + index_file =3D CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xm= l')) The magic string 'CiIndex.xml' is repeated several times. It would be good = to have something like humble.py that stores the magic string, so that if w= e ever change it that can be done quickly. > + found, man_path =3D find_project_in_single_index(project, index_= file, > manifest_dir) > + if found: > + projects[repo] =3D ('edkrepo_cfg', man_path) > + for repo in user_cfg_man_repos: > + manifest_dir =3D edkrepo_user_cfg.manifest_repo_abs_path(repo) > + index_file =3D CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xm= l')) > + found, man_path =3D find_project_in_single_index(project, index_= file, > manifest_dir) > + if found: > + projects[repo] =3D ('edkrepo_user_cfg', man_path) > + if len(projects.keys()) =3D=3D 1: > + repo =3D list(projects.keys())[0] > + return repo, projects[repo][0], projects[repo][1] > + elif len(projects.keys()) > 1 and man_repo: > + try: > + return man_repo, projects[man_repo][0], projects[man_repo][1= ] > + except KeyError: > + raise EdkrepoInvalidParametersException(except_msg_man_repo) > + elif os.path.isabs(project): > + return None, None, project > + elif os.path.isfile(os.path.join(os.getcwd(), project)): > + return None, None, os.path.join(os.getcwd(), project) > + elif not os.path.dirname(project): > + for repo in cfg_man_repos: > + if (man_repo and (repo =3D=3D man_repo)) or not man_repo: > + for dirpath, dirname, filenames in > os.walk(edkrepo_cfg.manifest_repo_abs_path(repo)): > + if project in filenames: > + return repo, 'edkrepo_cfg', os.path.join(dirpath= , project) > + for repo in user_cfg_man_repos: > + if (man_repo and (repo =3D=3D man_repo)) or not man_repo: > + for dirpath, dirname, filenames in > os.walk(edkrepo_user_cfg.manifest_repo_abs_path(repo)): > + if project in filenames: > + return repo, 'edkrepo_user_cfg', > +os.path.join(dirpath, project) If man_repo is None, and the project name exists in more than 1 manifest re= pository, then you will reach the end of this function and return None. It = seems like that should be a raised exception instead, telling the user that= they need to specific a specific manifest repository to disambiguate. > + > + > + > + > diff --git > a/edkrepo/common/workspace_maintenance/workspace_maintenance.py > b/edkrepo/common/workspace_maintenance/workspace_maintenance.py > index 6e20d43..ba62f6d 100644 > --- > a/edkrepo/common/workspace_maintenance/workspace_maintenance.py > +++ > b/edkrepo/common/workspace_maintenance/workspace_maintenance.py > @@ -10,6 +10,10 @@ > ''' Contains shared workspace maintenance functions. ''' >=20 > import os > +import unicodedata > + > +from edkrepo.common.edkrepo_exception import > +EdkrepoFoundMultipleException, EdkrepoNotFoundException from > +edkrepo.common.humble import GEN_A_NOT_IN_B, > GEN_FOUND_MULT_A_IN_B >=20 > def generate_name_for_obsolete_backup(absolute_path): > if not os.path.exists(absolute_path): > @@ -27,4 +31,15 @@ def > generate_name_for_obsolete_backup(absolute_path): > if not os.path.exists(os.path.join(dir_name, unique_name)): > unique_name_found =3D True > index +=3D 1 > - return unique_name > \ No newline at end of file > + return unique_name > + > +def case_insensitive_equal(str1, str2): > + return unicodedata.normalize("NFKD", str1.casefold()) =3D=3D > +unicodedata.normalize("NFKD", str2.casefold()) > + > +def case_insensitive_single_match(str1, str_list): > + matches =3D [x for x in str_list if case_insensitive_equal(str1, x)] > + if len(matches) =3D=3D 0: > + raise EdkrepoNotFoundException(GEN_A_NOT_IN_B.format(str1, > str_list)) > + elif len(matches) > 1: > + raise > EdkrepoFoundMultipleException(GEN_FOUND_MULT_A_IN_B.format(str1, > str_list)) > + return matches[0] > -- > 2.16.2.windows.1