From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mx.groups.io with SMTP id smtpd.web12.1273.1588282839070360381 for ; Thu, 30 Apr 2020 14:40:39 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=pMmb62ZI; spf=pass (domain: intel.com, ip: 192.55.52.120, mailfrom: erik.c.bjorge@intel.com) IronPort-SDR: MY9c+jXpjmZYNFsZX1oZRUfz4s0SfYq0ObPJbFTfYieA5bufKHberQ1SNBYt7Aw/LbzqrIEC4F QO/+2mzJGgEA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Apr 2020 14:40:37 -0700 IronPort-SDR: +i7m7+0/1gv0xgeq2B/EMl3Y6h6qdNQfZ3GxaBuKC1TEL8qt+BcQk79mztW5bTDv3TdDPCzkzM 9+FWHkovkavw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,337,1583222400"; d="scan'208";a="405556730" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by orsmga004.jf.intel.com with ESMTP; 30 Apr 2020 14:40:37 -0700 Received: from fmsmsx120.amr.corp.intel.com (10.18.124.208) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 30 Apr 2020 14:40:37 -0700 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by fmsmsx120.amr.corp.intel.com (10.18.124.208) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 30 Apr 2020 14:40:36 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.107) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 30 Apr 2020 14:40:36 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RL9jn04rSfLNIbhcEd7bPnkJMyNOlm6OQRxJ9fVbDuj4bONvAun5iQmt2wSfMpvdxgZukKTbeUFimxqXlNdTh2GwYmcCm1e0siyWlEhNANJRZlZqkCOCmtUed1VJfedW7nJVVso/gi2XcloNIGzzgR5sUDDoGmLzAG+MSpEnWgvPYkyxLr+p2+t2Xp97YyxZB5rjDDvosllQQVgTw5QxYTF7sl4qsNmdAPZdUg0YLKPvRSILNmTQP9gowPMR6w/hWmSDixd8saGral0IdOH+r22DbSy5SY6SmCAFUf12X/UZ+74Dcot0QK6HhoWtzCfL76f6V4HjIjgv3KR3o1EUHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QmWmiPnVLliyOSoLtbOAX6qgPk6IsBB6kwre+Y4ExD8=; b=VbdIF195syr3GLAFEDrgP8mJCgaiOERjgBHC7fxwDP7MjoFh/8T4085vw9TeVWhvzIz+CQPFDALsUHuL8cyfVmHMXqTpp5HYwv1Ma5nBX5ZcA7I0agZU7xEalH2AAloBxeL1vz3Ynl+jpV1sw8z46q+JjJzAzyGEszv05WI1KeqAEP8cfCaOHfZLFAKl6c6P+IW95IsU+mHsQVorZmQChhitv+Gm3r9KCEJ2WL72Q5Vk/0IFuYk3mH+enPQZcft6e4UeA0I68l6jKqnFEo7kYIAhMohvFvfLmM4jhM8VyDgTCV1tSQsi/txYNK61132WEQFroRKFqgrXOFa//HnRng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QmWmiPnVLliyOSoLtbOAX6qgPk6IsBB6kwre+Y4ExD8=; b=pMmb62ZIjd4uK/Mc053fdTyNXcBGNd3ixAb2R3eDhq6CDO1QWY0jM6gk7heI8utBAb1HMGBw0v8LZWfZ4SIv5JmYM5Pm9ucW5cKrx2N/JyF8yj4JjPVZKv2F1t/zZxJpi8XAlwJty8HopH1LrfBJvT3dVVsO+U1CoplorJTibo4= Received: from MW3PR11MB4554.namprd11.prod.outlook.com (2603:10b6:303:5d::7) by MW3PR11MB4571.namprd11.prod.outlook.com (2603:10b6:303:59::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.20; Thu, 30 Apr 2020 21:40:35 +0000 Received: from MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281]) by MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281%5]) with mapi id 15.20.2958.020; Thu, 30 Apr 2020 21:40:35 +0000 From: "Bjorge, Erik C" To: "Desimone, Nathaniel L" , "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to find projects across all manifest repositories Thread-Topic: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to find projects across all manifest repositories Thread-Index: AQHWHagVhN5VrU94BEi34XOYxV0mN6iSMV6AgAAB7lA= Date: Thu, 30 Apr 2020 21:40:35 +0000 Message-ID: References: <20200428215710.45504-1-ashley.e.desimone@intel.com> <20200428215710.45504-6-ashley.e.desimone@intel.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [134.134.136.198] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b6f45fee-c35f-4289-3f20-08d7ed4f1e0c x-ms-traffictypediagnostic: MW3PR11MB4571: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0389EDA07F x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: uMiGf90ku0KxP/DbDSk8XSSYibxDFSkeeSc56Ll4pUQie2maFBx1bszOaj6oMcEsA+r59ClxjOZxYCaurVkG9I8YdWdM2LLTLt4iGDSK7EjirlQTklCOu64BZVFHUV67E6I/p4yL9ForrLXDM9cS6hgcd7j28STjeU2lrrQ20w0Q0g8RemGhGKbhboVP+9HTYHPIsc004f7YDbRlBlNkphlYqG6TI/SMH470hfH27uFXdMa5VLmBHOtyesxH/RZrIbk+RQ2cLUTgh4XyyX0fge9eaoKw+DRz/7AnCTfqUE//F2fFfUuu8j76eYS/KNnnVydr2xspyyn7gbyBtH7IJcM8LINBjYLoK6XEAQddU4unZ4e84cw6a5WB+kcnhf8XecrMptX/9nFbMhOHRKVUM/Wl8qYhyNZckOhOhTwbERCmexXTEBBQrOAyEd/orS6DIyPkfg1q0mYNAkPLJvz7dtaGdCk7ddD8uE7KPkB0jAkD5JVokvYx1LRVXYN7ppMj x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW3PR11MB4554.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(396003)(366004)(136003)(346002)(39860400002)(376002)(66946007)(66476007)(66556008)(66446008)(64756008)(478600001)(76116006)(8936002)(71200400001)(2906002)(52536014)(5660300002)(8676002)(107886003)(4326008)(7696005)(110136005)(54906003)(316002)(53546011)(26005)(186003)(33656002)(9686003)(55016002)(86362001)(6506007)(37363001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: dUGuTk01+ea5H8hO8LXuV5bnpwu17ywxjeN/4WpCArbkqlnGMJuYYkVpFpCyTYzUncGr02H8bolDCvgDJjkEp5R4fzthygOYoE8E1x6YyX05TOA+TrnbR8i/A40DfXlyroKDk2DgBVlCn78ua3ZkPOzxyTJ6+8thl61Ra8KwK3VT8ASQNLmJ/V8bOfq+gLTEonZAgjy+JCWvkRJF9t4So6PJH/EV8iXN+1w+w6aIh5QnVWi/ER3E5SFM0TGXpZT+aRW8Gjn27ib0ejCZ6shZ4MesZXIrnbsnsBCUBO7DD+mlzA09wKCTgJFKXnhNkGytqx26hSS4/gCnEnKuwCKAwY1kayqk3c/yGETB+5NtIP7SLU+UHACIb3ra2H3poTn4UH4y1IsHLn0f6a1aXjZpfornFtJ433tTc3Xh0Z5A4y7/pAe097qf/mXN/0il30ryXU/Dgw1amcD4RPF9aQMPSNt5zGBfmQ3KIyguEn+/bENF7oUy/u3Yx+02gpxJLrLB6MCH0cEoiVOImcxfswAR313k9arR7cOVO1jCL6qfupa81erFomo6OEqEsdsNIBMNebqVr9myTdcTp3XmN+ZSB0FXNVhlHJQTn6XEx6DIBkPnspVRuiGML+9eDSfmFQxD57N7ZpiQdral/SEnIcnuvfGyDG3sa4JN5tcK71YSa34u9ek9Ul4eFYmKwiAo+I796/b8m0AyH4clF8KHNjDa3fghnpyBw/9vq1FyMVYndl8qq8PO+2CqHxH1ZzAA+W8Kag1wxjKNbf+c/frJ+U4Qq59ralK9I1mGLz9dfEs+Q2WCz5BzmGR/GxPFuHNgM1IW MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: b6f45fee-c35f-4289-3f20-08d7ed4f1e0c X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2020 21:40:35.6850 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: b/mCSK9Ul3Cf9i8XvA3ffT1AojeVT8/2VbVDnLRqkQmKwS/brRVkt7CDsF7hqQGx0lQWEiDaJODtYdvhmnULfA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4571 Return-Path: erik.c.bjorge@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I agree with your comment about moving magic strings to a common location. = I would suggest that we put it someplace other than humble.py. I think we= should add a new file for configuration settings like this. I think somet= hing like tool_config.py would be good. Thanks, -Erik -----Original Message----- From: Desimone, Nathaniel L =20 Sent: Thursday, April 30, 2020 2:29 PM To: Desimone, Ashley E ; devel@edk2.groups.io Cc: Pandya, Puja ; Bjorge, Erik C ; Bret Barkelew ; Agyeman, Prince Subject: RE: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to fin= d projects across all manifest repositories Hi Ashley, Please see comments inline. Thanks, Nate > -----Original Message----- > From: Desimone, Ashley E > Sent: Tuesday, April 28, 2020 2:57 PM > To: devel@edk2.groups.io > Cc: Desimone, Nathaniel L ; Pandya,=20 > Puja ; Bjorge, Erik C=20 > ; Bret Barkelew=20 > ; Agyeman, Prince=20 > > Subject: [edk2-staging/EdkRepo] [PATCH 5/7] EdkRepo: Add ability to=20 > find projects across all manifest repositories >=20 > Add find_project_in_all_indicies() to search for and return a tuple=20 > (source repo, source config, path to manifest) if a matching project is f= ound. >=20 > Add find_project_in_single_index() to find the path to a project=20 > within a single manifest repositories index file and return a whether=20 > the specified project was found and if so its path within the manifest re= pository. >=20 > Signed-off-by: Ashley E Desimone > Cc: Nate DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Cc: Prince Agyeman > --- > .../manifest_repos_maintenance.py | 70 > +++++++++++++++++++++- > .../workspace_maintenance/workspace_maintenance.py | 17 +++++- > 2 files changed, 84 insertions(+), 3 deletions(-) >=20 > diff --git > a/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > b/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > index 4bded46..9b441ac 100644 > --- > a/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > +++ > b/edkrepo/common/workspace_maintenance/manifest_repos_maintenanc > e.py > @@ -15,11 +15,12 @@ import git > from git import Repo >=20 > import edkrepo.config.config_factory as cfg -from=20 > edkrepo.common.edkrepo_exception import=20 > EdkrepoUncommitedChangesException > +from edkrepo.common.edkrepo_exception import=20 > +EdkrepoUncommitedChangesException, > EdkrepoInvalidParametersException > from edkrepo.common.progress_handler import GitProgressHandler import=20 > edkrepo.common.workspace_maintenance.humble.manifest_repos_maint > enance_humble as humble from > edkrepo.common.workspace_maintenance.workspace_maintenance import=20 > generate_name_for_obsolete_backup > - > +from edkrepo.common.workspace_maintenance.workspace_maintenance > import > +case_insensitive_single_match from > edkrepo_manifest_parser.edk_manifest > +import CiIndexXml, ManifestXml >=20 > def pull_single_manifest_repo(url, branch, local_path, reset_hard=3DFals= e): > ''' > @@ -135,3 +136,68 @@ def list_available_man_repos(edkrepo_cfg, > edkrepo_user_cfg): > return cfg_man_repos, user_cfg_man_repos, conflicts >=20 >=20 > +def find_project_in_single_index (project, index_file, manifest_dir): > + ''' > + Finds a project in a single global manifest repositories index file.= If found > + returns (True, path to file) if not returns (False, None) > + ''' > + global_manifest_path =3D None > + try: > + proj_name =3D case_insensitive_single_match(project, > index_file.project_list) > + except: > + proj_name =3D None > + if proj_name: > + ci_index_xml_rel_path =3D > os.path.normpath(index_file.get_project_xml(proj_name)) > + global_manifest_path =3D os.path.join(manifest_dir, > ci_index_xml_rel_path) > + return True, global_manifest_path > + else: > + return False, global_manifest_path > + > + > +def find_project_in_all_indices (project, edkrepo_cfg,=20 > +edkrepo_user_cfg, > except_msg_man_repo, except_msg_not_found, man_repo=3DNone): > + ''' > + Finds the project in all manifest repositories listed in the edkrepo= .efg and > + edkrepo_user.cfg. If a project with the same name is found uses > man_repo to select > + the correct entry > + ''' > + cfg_man_repos, user_cfg_man_repos, conflicts =3D > list_available_man_repos(edkrepo_cfg, edkrepo_user_cfg) > + projects =3D {} > + for repo in cfg_man_repos: > + manifest_dir =3D edkrepo_cfg.manifest_repo_abs_path(repo) > + index_file =3D CiIndexXml(os.path.join(manifest_dir,=20 > + 'CiIndex.xml')) The magic string 'CiIndex.xml' is repeated several times. It would be good = to have something like humble.py that stores the magic string, so that if w= e ever change it that can be done quickly. > + found, man_path =3D find_project_in_single_index(project,=20 > + index_file, > manifest_dir) > + if found: > + projects[repo] =3D ('edkrepo_cfg', man_path) > + for repo in user_cfg_man_repos: > + manifest_dir =3D edkrepo_user_cfg.manifest_repo_abs_path(repo) > + index_file =3D CiIndexXml(os.path.join(manifest_dir, 'CiIndex.xm= l')) > + found, man_path =3D find_project_in_single_index(project,=20 > + index_file, > manifest_dir) > + if found: > + projects[repo] =3D ('edkrepo_user_cfg', man_path) > + if len(projects.keys()) =3D=3D 1: > + repo =3D list(projects.keys())[0] > + return repo, projects[repo][0], projects[repo][1] > + elif len(projects.keys()) > 1 and man_repo: > + try: > + return man_repo, projects[man_repo][0], projects[man_repo][1= ] > + except KeyError: > + raise EdkrepoInvalidParametersException(except_msg_man_repo) > + elif os.path.isabs(project): > + return None, None, project > + elif os.path.isfile(os.path.join(os.getcwd(), project)): > + return None, None, os.path.join(os.getcwd(), project) > + elif not os.path.dirname(project): > + for repo in cfg_man_repos: > + if (man_repo and (repo =3D=3D man_repo)) or not man_repo: > + for dirpath, dirname, filenames in > os.walk(edkrepo_cfg.manifest_repo_abs_path(repo)): > + if project in filenames: > + return repo, 'edkrepo_cfg', os.path.join(dirpath= , project) > + for repo in user_cfg_man_repos: > + if (man_repo and (repo =3D=3D man_repo)) or not man_repo: > + for dirpath, dirname, filenames in > os.walk(edkrepo_user_cfg.manifest_repo_abs_path(repo)): > + if project in filenames: > + return repo, 'edkrepo_user_cfg',=20 > +os.path.join(dirpath, project) If man_repo is None, and the project name exists in more than 1 manifest re= pository, then you will reach the end of this function and return None. It = seems like that should be a raised exception instead, telling the user that= they need to specific a specific manifest repository to disambiguate. > + > + > + > + > diff --git > a/edkrepo/common/workspace_maintenance/workspace_maintenance.py > b/edkrepo/common/workspace_maintenance/workspace_maintenance.py > index 6e20d43..ba62f6d 100644 > --- > a/edkrepo/common/workspace_maintenance/workspace_maintenance.py > +++ > b/edkrepo/common/workspace_maintenance/workspace_maintenance.py > @@ -10,6 +10,10 @@ > ''' Contains shared workspace maintenance functions. ''' >=20 > import os > +import unicodedata > + > +from edkrepo.common.edkrepo_exception import=20 > +EdkrepoFoundMultipleException, EdkrepoNotFoundException from=20 > +edkrepo.common.humble import GEN_A_NOT_IN_B, > GEN_FOUND_MULT_A_IN_B >=20 > def generate_name_for_obsolete_backup(absolute_path): > if not os.path.exists(absolute_path): > @@ -27,4 +31,15 @@ def > generate_name_for_obsolete_backup(absolute_path): > if not os.path.exists(os.path.join(dir_name, unique_name)): > unique_name_found =3D True > index +=3D 1 > - return unique_name > \ No newline at end of file > + return unique_name > + > +def case_insensitive_equal(str1, str2): > + return unicodedata.normalize("NFKD", str1.casefold()) =3D=3D=20 > +unicodedata.normalize("NFKD", str2.casefold()) > + > +def case_insensitive_single_match(str1, str_list): > + matches =3D [x for x in str_list if case_insensitive_equal(str1, x)] > + if len(matches) =3D=3D 0: > + raise EdkrepoNotFoundException(GEN_A_NOT_IN_B.format(str1, > str_list)) > + elif len(matches) > 1: > + raise > EdkrepoFoundMultipleException(GEN_FOUND_MULT_A_IN_B.format(str1, > str_list)) > + return matches[0] > -- > 2.16.2.windows.1