From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mx.groups.io with SMTP id smtpd.web09.1369.1607021488156736420 for ; Thu, 03 Dec 2020 10:51:28 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=pvX4QiVw; spf=pass (domain: intel.com, ip: 134.134.136.126, mailfrom: ashley.e.desimone@intel.com) IronPort-SDR: BUKk6k6v6SjgAA2H9tiZ7yB4lx2/chwiFUgJJAzvcr53nF+aiz6RHamX7W+cQwowCfnu/K3WGB YJ9a1I/a+IFA== X-IronPort-AV: E=McAfee;i="6000,8403,9824"; a="161021948" X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="161021948" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2020 10:51:26 -0800 IronPort-SDR: Paei/We9LmiGmS66F+VAnhFHw+rFyG9E8HUPUh/XD203elWmkapYcw6/J3DXxTaFpzthi5HtU2 4oQJFSWtW/GQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="538494452" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga006.fm.intel.com with ESMTP; 03 Dec 2020 10:51:26 -0800 Received: from fmsmsx606.amr.corp.intel.com (10.18.126.86) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 10:51:25 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx606.amr.corp.intel.com (10.18.126.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 3 Dec 2020 10:51:25 -0800 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.43) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 3 Dec 2020 10:51:25 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WZWG5B4TQMPFySRJjg0y8Ohj5iLYf9E8t0DaxVPwXzaWikbdDRdN+S68Z6cGp73Dund4kHwZH0EkOR2jwe99u8Hb2tqDSIRn9daLEEGKFtHjR8uTAhH/KzvhZMVnvwavXBx8YboNio3E9ebQjMTGUukQKDBF3RlUwqnBBdlhCIvWBoUf6U5Xa4YwSzB7CWwhLm0R/3YZ7fTbovd4PNJllwXNIXWapHzFKdKuMh5MwDGrFkQAnKWNEgLHlvXsLqYKYVa2VZMnJRIJNI0VWiWY6rp8l+m3J/avmJbWBRU3Jnu9fHuIh4lPdej8ZzYU31M6W/iaTT4+Fb7doyjVtnUwoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nxU/QvxC37TpMDq/u5c7YLZo98RAxpYMElbTfltpplk=; b=n7w1hEz6uCWYoG2A2+6zmSb/eSIa7Tdw+V74UD1tBn0eqTRtkfXLzJjuwsRg6zj8ZOhduKdDa4YTTmUafDkXOEqhlfveCw71Srx6tinZKZQXW19uxnwMk5Qcwcydbja914XNMKQFGqOA2GUVkPqJ4+sBTrHL9hCXgXX9Zf7o9kNPnWVC5HXSYL/7vRSiEUL3SYlPb6fA7U1Zq8CU9bxwgNhDGUBxLd191ZSMgBYeNx79/fEunKVN04trNjmzdQ+dymdNfx2LRpWiNYeL7viAfKZBWewqkVGtGmRdeYHirdzZXwg0kq/t+LAh79Aj05cTl2LfITSvme3HI4NIXp8izg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nxU/QvxC37TpMDq/u5c7YLZo98RAxpYMElbTfltpplk=; b=pvX4QiVwAP4a5xPQWWV+lbFt88E3Qvw4glZAHMaAuq+gWJf6seQ7To+bvsyem0M7c4ov04E+llnt5ZWmqLmp1tS1rxnFT5GvzJ44Zb0PRjxqRHeEWzawx5Q6NZqxD4TYNBqY/gN0z3rny2WsVyRtwlzfdT4k7mgnf/pz8gsFeDU= Received: from BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) by BYAPR11MB3447.namprd11.prod.outlook.com (2603:10b6:a03:1d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.31; Thu, 3 Dec 2020 18:51:24 +0000 Received: from BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5]) by BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5%6]) with mapi id 15.20.3611.034; Thu, 3 Dec 2020 18:51:24 +0000 From: "Ashley E Desimone" To: "Bjorge, Erik C" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Thread-Topic: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Thread-Index: AQHWvJ0pg8AioRin4UWHvaJJ3YpFCKnl0VPQ Date: Thu, 3 Dec 2020 18:51:23 +0000 Message-ID: References: <737e00b25148bcf2faede39d38b352dd41843090.1605588263.git.erik.c.bjorge@intel.com> In-Reply-To: <737e00b25148bcf2faede39d38b352dd41843090.1605588263.git.erik.c.bjorge@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a639e2b2-31b0-43a3-28d5-08d897bc6ef0 x-ms-traffictypediagnostic: BYAPR11MB3447: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: j8OsBecT+sPSvwN2uUom+Eg9pQ1JI5vKz3eGLfPODNwGfz1Jjl3MQqgrkKSKucsUghzwMOx8BVveiwj7FgznqWmzTLze6To72YqYI1SYAS4vU9plJq/gNN2MJQ0h4NYYqHJpJGNqKYPyyZGDQ8I44lnPiqE7NCiz3sAe4PCIfM9ryszkIjX/scgVLaq49cFdSVwct+Y54zlorglGCfOcfLomaO4SnqCtgyZekP8HJyM7q422PESApZrTg4khz0doOVPM93d+ThTlJ6gVBXuO5ItfqeHFY5zeo0jx31YGPfDWeUqGtqhlHMzxReDNZUTnbnBlDjloo3+3+dMoAdwoOQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3973.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(376002)(396003)(366004)(346002)(39860400002)(136003)(186003)(110136005)(478600001)(64756008)(107886003)(8936002)(76116006)(8676002)(66556008)(54906003)(83380400001)(66476007)(9686003)(66446008)(52536014)(26005)(5660300002)(66946007)(86362001)(2906002)(30864003)(55016002)(53546011)(316002)(4326008)(6506007)(33656002)(71200400001)(7696005)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?F/th3ytbSkH4qsmi0s8g0QXAznhPMo/CTZOlvxJd5Qn4G7sKvr6Us0NM68EC?= =?us-ascii?Q?JLmVRSOptiD029bArAEA3oAnJD2jAX4kRnt2ec8ZmFGeDxAPhE28IUBqxwaO?= =?us-ascii?Q?9QYTyUy3rYOFSSaiUL1fqa29QgC3iuqUyUG5jyYQsgIIkOHsBRQmEIztA/IW?= =?us-ascii?Q?etXcEkH3F855LBYA589ZoPsDjCGNY+dZk9TXa0DLH3xydZlNSFySNOCM4dKn?= =?us-ascii?Q?849PY565iphDVBn1vb/VPrHR6XAZ65AxfY9lvf2uFF46449M4f42tQPAGPPV?= =?us-ascii?Q?Qxv3uLMlIPq6wNtqcy+IT1HYHCZA1s7BaoleEWzkFvXKj/Sr6eg6gCFR78lk?= =?us-ascii?Q?gj1VngRV1K8LROaI2JyJ2hxiGvjGV8FexIW8MGyFFCC1xFq8FAw9J7oT5G9D?= =?us-ascii?Q?1AVvClAWZonpWY6KusVg7VNhSf4cNGt6x2HUMSgk6AQxJ0eCTpL7UXL4VxpP?= =?us-ascii?Q?0xUAzpgDNaBdVZyD+9ZLQmgsjQROny4/QqRAzAoLnJYxuoyADiN0Gcuxvnkn?= =?us-ascii?Q?Q5uQ06bnpJEI2F/ZtapdtyfmVoeeUaZTcDdEf1+H7nGIbKTm263z/YR6bP8X?= =?us-ascii?Q?40xRRvinfvuzuauiWsTkFubFK06VcP29YVpU9oJ/UE+ylJg3azyzXWC+xBL4?= =?us-ascii?Q?7HAgk3aBOjj39qzI3gxPDh6E6jaP0SEClRPqUHbGyfNiW4jsTSw6okMhALbq?= =?us-ascii?Q?AcKlPzcDyp1sgUnh/ztFWlIXmahhwMExPCsFXxtkt+bae7hzn9rM3NQgzCsw?= =?us-ascii?Q?E7J+OGIZdwdO6qG4OX2MJNQOjDyz6B4fwc4I0dCmka8QAXpkxFk3BmRNZB5G?= =?us-ascii?Q?HS5E1wqXvxg2N7gcX6gHRQ+1BaifffwzXte0QI4ibZTecq2d2LurzqzWIyLj?= =?us-ascii?Q?YaZ9b5TF6NLpB68ZSrub91x+Y+IZoSj4XhubiLLEhXZehfJrp/tQIgJfVKKf?= =?us-ascii?Q?3MH6Lo3YLkQrUjH1KNC32P9qDX502wsDCDtzLtpsXVQ=3D?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB3973.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a639e2b2-31b0-43a3-28d5-08d897bc6ef0 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 18:51:23.8836 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: JNeeG7E1HTas5lK0p8SW6YIK/SUuZBg+hnkzARH1yKJtq3RUHUd6vK73Z4Jzr5u/h8N7e3jCWLAOifGe/dMXI2VBImk668hCxmvg+6OvEYk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3447 Return-Path: ashley.e.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Ashley DeSimone -----Original Message----- From: Erik Bjorge =20 Sent: Monday, November 16, 2020 8:50 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Adds a module to add a repo cache and mange it. Also adds a command to man= age the repo cache from EdkRepo. No other commands use the functionality a= t this point. Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge Signed-off-by: Erik Bjorge --- edkrepo/commands/arguments/cache_args.py | 19 ++ edkrepo/commands/cache_command.py | 118 ++++++++++++ edkrepo/commands/humble/cache_humble.py | 17 ++ edkrepo/common/common_c= ache_functions.py | 41 +++++ edkrepo/common/edkrepo_exception.py | 3 + edkrepo/config/config_factory.py | 14 +- edkrepo/config/tool_config.py | 5 +- project_utils/cache.py | 224 +++++++++++++++++++++++ project_utils/project_utils_strings.py | 11 ++ 9 files changed, 448 insertions(+), 4 deletions(-) create mode 100644 edk= repo/commands/arguments/cache_args.py create mode 100644 edkrepo/commands/cache_command.py create mode 100644 e= dkrepo/commands/humble/cache_humble.py create mode 100644 edkrepo/common/common_cache_functions.py create mode 100644 project_utils/cache.py diff --git a/edkrepo/commands/arguments/cache_args.py b/edkrepo/commands/ar= guments/cache_args.py new file mode 100644 index 0000000..0080536 --- /dev/null +++ b/edkrepo/commands/arguments/cache_args.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +# +## @file +# cache_args.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +''' Contains the help and description strings for arguments in the=20 +cache command meta data. +''' +COMMAND_DESCRIPTION =3D ('Manages local caching support for project repos.= The goal of this feature ' + 'is to improve clone performance')=20 +COMMAND_ENABLE_HELP =3D 'Enables caching support on the system.' +COMMAND_DISABLE_HELP =3D 'Disables caching support on the system.' +COMMAND_UPDATE_HELP =3D 'Update the repo cache for all cached projects.' +COMMAND_INFO_HELP =3D 'Display the current cache information.' +COMMAND_PROJECT_HELP =3D 'Project to add to the cache.' diff --git a/edkrepo/commands/cache_command.py b/edkrepo/commands/cache_com= mand.py new file mode 100644 index 0000000..9f0d6e9 --- /dev/null +++ b/edkrepo/commands/cache_command.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +# +## @file +# cache_command.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +import edkrepo.commands.arguments.cache_args as arguments from=20 +edkrepo.commands.edkrepo_command import EdkrepoCommand from=20 +edkrepo.commands.edkrepo_command import SourceManifestRepoArgument from=20 +edkrepo.commands.humble.cache_humble import CACHE_ENABLED, CACHE_FETCH,=20 +CACHE_INFO from edkrepo.commands.humble.cache_humble import=20 +CACHE_INFO_LINE, PROJECT_NOT_FOUND, NO_INSTANCE from=20 +edkrepo.commands.humble.cache_humble import UNABLE_TO_LOAD_MANIFEST,=20 +UNABLE_TO_PARSE_MANIFEST from edkrepo.common.common_cache_functions=20 +import add_missing_cache_repos from=20 +edkrepo.common.common_cache_functions import get_repo_cache_obj from=20 +edkrepo.common.edkrepo_exception import EdkrepoCacheException from=20 +edkrepo.common.workspace_maintenance.manifest_repos_maintenance import=20 +find_project_in_all_indices from edkrepo.config.config_factory import=20 +get_workspace_manifest from edkrepo_manifest_parser.edk_manifest import=20 +ManifestXml + + +class CacheCommand(EdkrepoCommand): + def __init__(self): + super().__init__() + + def get_metadata(self): + metadata =3D {} + metadata['name'] =3D 'cache' + metadata['help-text'] =3D arguments.COMMAND_DESCRIPTION + args =3D [] + metadata['arguments'] =3D args + args.append({'name': 'enable', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_ENABLE_HELP}) + args.append({'name': 'disable', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_DISABLE_HELP}) + args.append({'name': 'update', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_UPDATE_HELP}) + args.append({'name': 'info', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_INFO_HELP}) + args.append({'name': 'project', + 'positional': True, + 'required': False, + 'help-text': arguments.COMMAND_PROJECT_HELP}) + args.append(SourceManifestRepoArgument) + return metadata + + def run_command(self, args, config): + # Process enable disable requests + if args.disable: + config['user_cfg_file'].set_caching_state(False) + elif args.enable: + config['user_cfg_file'].set_caching_state(True) + + # Get the current state now that we have processed enable/disable + cache_state =3D config['user_cfg_file'].caching_state + print(CACHE_ENABLED.format(cache_state)) + if not cache_state: + return + + # State is enabled so make sure cache directory exists + cache_obj =3D get_repo_cache_obj(config) + + # Check to see if a manifest was provided and add any missing remo= tes + manifest =3D None + if args.project is not None: + manifest =3D _get_manifest(args.project, config, args.source_m= anifest_repo) + else: + try: + manifest =3D get_workspace_manifest() + except Exception: + pass + + # If manifest is provided attempt to add any remotes that do not e= xist + if manifest is not None: + add_missing_cache_repos(cache_obj, manifest, True) + + # Display all the cache information + if args.info: + print(CACHE_INFO) + info =3D cache_obj.get_cache_info(args.verbose) + for item in info: + print(CACHE_INFO_LINE.format(item.path, item.remote,=20 + item.url)) + + # Do an update if requested + if args.update: + print(CACHE_FETCH) + cache_obj.update_cache(verbose=3DTrue) + + # Close the cache repos + cache_obj.close(args.verbose) + + +def _get_manifest(project, config, source_manifest_repo=3DNone): + try: + manifest_repo, source_cfg, manifest_path =3D find_project_in_all_i= ndices( + project, + config['cfg_file'], + config['user_cfg_file'], + PROJECT_NOT_FOUND.format(project), + NO_INSTANCE.format(project), + source_manifest_repo) + except Exception: + raise EdkrepoCacheException(UNABLE_TO_LOAD_MANIFEST) + try: + manifest =3D ManifestXml(manifest_path) + except Exception: + raise EdkrepoCacheException(UNABLE_TO_PARSE_MANIFEST) + return manifest diff --git a/edkrepo/commands/humble/cache_humble.py b/edkrepo/commands/hum= ble/cache_humble.py new file mode 100644 index 0000000..4f318ac --- /dev/null +++ b/edkrepo/commands/humble/cache_humble.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +# +## @file +# cache_humble.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +CACHE_ENABLED =3D 'Caching Enabled: {}' +CACHE_INFO =3D 'Cache Information:' +CACHE_INFO_LINE =3D '+ {}\n {} ({})' +CACHE_FETCH =3D 'Fetching all remotes... (this could take a while)' +PROJECT_NOT_FOUND =3D 'Project {} does not exist' +NO_INSTANCE =3D 'Unable to determine instance to use for {}' +UNABLE_TO_LOAD_MANIFEST =3D 'Unable to load manifest file.' +UNABLE_TO_PARSE_MANIFEST =3D 'Failed to parse manifest file.' diff --git a/edkrepo/common/common_cache_functions.py b/edkrepo/common/comm= on_cache_functions.py new file mode 100644 index 0000000..84bd3ed --- /dev/null +++ b/edkrepo/common/common_cache_functions.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +# +## @file +# common_cache_functions.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +import os + +from edkrepo.config.config_factory import=20 +get_edkrepo_global_data_directory from edkrepo.config.tool_config=20 +import SUBMODULE_CACHE_REPO_NAME from project_utils.cache import=20 +RepoCache + + +def get_global_cache_directory(config): + if config['user_cfg_file'].caching_state: + return os.path.join(get_edkrepo_global_data_directory(), '.cache') + return None + + +def get_repo_cache_obj(config): + cache_obj =3D None + cache_directory =3D get_global_cache_directory(config) + if cache_directory is not None: + cache_obj =3D RepoCache(cache_directory) + cache_obj.open() + return cache_obj + + +def add_missing_cache_repos(cache_obj, manifest, verbose=3DFalse): + print('Adding and fetching new remotes... (this could take a while)') + for remote in manifest.remotes: + cache_obj.add_repo(url=3Dremote.url, verbose=3Dverbose) + alt_submodules =3D manifest.submodule_alternate_remotes + if alt_submodules: + print('Adding and fetching new submodule remotes... (this could al= so take a while)') + cache_obj.add_repo(name=3DSUBMODULE_CACHE_REPO_NAME, verbose=3Dver= bose) + for alt in alt_submodules: + cache_obj.add_remote(alt.alternate_url,=20 +SUBMODULE_CACHE_REPO_NAME, verbose) diff --git a/edkrepo/common/edkrepo_exception.py b/edkrepo/common/edkrepo_e= xception.py index a56e709..b3f2300 100644 --- a/edkrepo/common/edkrepo_exception.py +++ b/edkrepo/common/edkrepo_exception.py @@ -98,3 +98,6 @@ class EdkrepoGitConfigSetupException(EdkrepoException): def __init__(self, message): super().__init__(message, 131) =20 +class EdkrepoCacheException(EdkrepoException): + def __init__(self, message): + super().__init__(message, 132) diff --git a/edkrepo/config/config_factory.py b/edkrepo/config/config_facto= ry.py index fe69460..3680c0b 100644 --- a/edkrepo/config/config_factory.py +++ b/edkrepo/config/config_factory.py @@ -225,10 +225,20 @@ class GlobalUserConfig(BaseConfig): self.filename =3D os.path.join(get_edkrepo_global_data_directory()= , "edkrepo_user.cfg") self.prop_list =3D [ CfgProp('scm', 'mirror_geo', 'geo', 'none', False), - CfgProp('send-review', 'max-patch-set', 'max_patch_set', '10',= False) - ] + CfgProp('send-review', 'max-patch-set', 'max_patch_set', '10',= False), + CfgProp('caching', 'enable-caching', 'enable_caching_text',=20 + 'false', False)] super().__init__(self.filename, get_edkrepo_global_data_directory(= ), False) =20 + @property + def caching_state(self): + return self.enable_caching_text.lower() =3D=3D 'true' + + def set_caching_state(self, enable): + if enable: + self.enable_caching_text =3D 'true' + else: + self.enable_caching_text =3D 'false' + @property def max_patch_set_int(self): try: diff --git a/edkrepo/config/tool_config.py b/edkrepo/config/tool_config.py = index eee1326..81f4ddf 100644 --- a/edkrepo/config/tool_config.py +++ b/edkrepo/config/tool_config.py @@ -1,10 +1,11 @@ #!/usr/bin/env python3 # ## @file -# tool)config.py +# tool_config.py # # Copyright (c) 2020, Intel Corporation. All rights reserved.
# SPDX-= License-Identifier: BSD-2-Clause-Patent # =20 -CI_INDEX_FILE_NAME =3D 'CiIndex.xml' \ No newline at end of file +CI_INDEX_FILE_NAME =3D 'CiIndex.xml' +SUBMODULE_CACHE_REPO_NAME =3D 'submodule-cache' diff --git a/project_utils/cache.py b/project_utils/cache.py new file mode = 100644 index 0000000..8efd411 --- /dev/null +++ b/project_utils/cache.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +# +## @file +# cache.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # from collections import=20 +namedtuple import os import shutil + +from git import Repo + +from edkrepo.common.progress_handler import GitProgressHandler from=20 +project_utils.project_utils_strings import CACHE_ADD_REMOTE,=20 +CACHE_ADDING_REPO, CACHE_CHECK_ROOT_DIR from=20 +project_utils.project_utils_strings import CACHE_FAILED_TO_CLOSE,=20 +CACHE_FAILED_TO_OPEN, CACHE_FETCH_REMOTE from=20 +project_utils.project_utils_strings import CACHE_REMOTE_EXISTS,=20 +CACHE_REMOVE_REPO, CACHE_REPO_EXISTS + +CacheInfo =3D namedtuple('CacheInfo', ['path', 'remote', 'url']) + + +class RepoCache(object): + """ + Provides basic management of a cache repo. + """ + def __init__(self, path): + self._cache_root_path =3D path + self._repos =3D {} + + def _create_name(self, url_or_name): + """ + Used to create consistent repo and remote names + """ + dir_name =3D url_or_name.split('/')[-1] + if not dir_name.endswith('.git'): + dir_name +=3D '.git' + return dir_name + + def _get_repo_path(self, dir_name): + return os.path.join(self._cache_root_path, dir_name) + + def _get_repo(self, dir_name): + """ + Returns the git repo object for the cache repo. + + Raises FileNotFoundError if the cache directory does not exist. + Raises IOError if the repo cannot be opened + """ + repo_path =3D self._get_repo_path(dir_name) + if not os.path.isdir(repo_path): + raise FileNotFoundError + try: + repo =3D Repo(repo_path) + except Exception: + raise IOError + return repo + + def _get_cache_dirs(self): + if not os.path.isdir(self._cache_root_path): + raise FileNotFoundError + return [x for x in os.listdir(self._cache_root_path) if=20 + os.path.isdir(self._get_repo_path(x))] + + def _add_and_fetch_remote(self, repo, remote_name, url, verbose=3DFals= e): + if verbose: + print(CACHE_ADD_REMOTE.format(remote_name, url)) + repo.create_remote(remote_name, url) + if verbose: + print(CACHE_FETCH_REMOTE.format(remote_name, url)) + repo.remotes[remote_name].fetch(progress=3DGitProgressHandler()) + + def open(self, verbose=3DFalse): + """ + Opens all cache repos. + + Raises FileNotFoundError if the cache directory does not exist. + """ + if not self._repos: + if not os.path.isdir(self._cache_root_path): + if verbose: + print(CACHE_CHECK_ROOT_DIR.format(self._cache_root_pat= h)) + os.makedirs(self._cache_root_path) + + for dir_name in self._get_cache_dirs(): + try: + self._repos[dir_name] =3D self._get_repo(dir_name) + except Exception: + if verbose: + print(CACHE_FAILED_TO_OPEN.format(dir_name)) + + def close(self, verbose=3DFalse): + """ + Closes all cache repos. + """ + for dir_name in self._repos: + try: + self._repos[dir_name].close() + except Exception: + if verbose: + print(CACHE_FAILED_TO_CLOSE.format(dir_name)) + self._repos =3D {} + + def get_cache_path(self, url_or_name): + dir_name =3D self._create_name(url_or_name) + if dir_name not in self._repos: + return None + return self._get_repo_path(dir_name) + + def get_cache_info(self, verbose=3DFalse): + """ + Returns a list of remotes currently configured in the cache. + + Raises FileNotFoundError if the cache repo is not open. + """ + ret_val =3D [] + for dir_name in self._repos: + for remote in self._repos[dir_name].remotes: + ret_val.append(CacheInfo(self._get_repo_path(dir_name), re= mote.name, remote.url)) + return ret_val + + def delete_cache_root(self, verbose=3DFalse): + """ + Deletes the cache root directory and all caches. + """ + if os.path.isdir(self._cache_root_path): + if self._repos: + self.close() + shutil.rmtree(self._cache_root_path, ignore_errors=3DTrue) + + def add_repo(self, url=3DNone, name=3DNone, verbose=3DFalse): + """ + Adds a repo to the cache if it does not already exist. + + """ + remote_name =3D None + if url is None and name is None: + raise ValueError + elif name is not None: + dir_name =3D self._create_name(name) + else: + dir_name =3D self._create_name(url) + if url is not None: + remote_name =3D self._create_name(url) + repo_path =3D self._get_repo_path(dir_name) + + if dir_name in self._repos: + if verbose: + print(CACHE_REPO_EXISTS.format(dir_name)) + else: + if verbose: + print(CACHE_ADDING_REPO.format(dir_name)) + os.makedirs(repo_path) + self._repos[dir_name] =3D Repo.init(repo_path, bare=3DTrue) + + if remote_name is not None and remote_name not in self._repos[dir_= name].remotes: + self._add_and_fetch_remote(self._get_repo(dir_name), remote_na= me, url) + return dir_name + + def remove_repo(self, url=3DNone, name=3DNone, verbose=3DFalse): + """ + Removes a remote from the cache repo if it exists + + Raises FileNotFoundError if the cache repo is not open. + """ + if url is None and name is None: + raise ValueError + elif name is not None: + dir_name =3D self._create_name(name) + else: + dir_name =3D self._create_name(url) + if dir_name not in self._repos: + return + if verbose: + print(CACHE_REMOVE_REPO.format(dir_name)) + self._repos.pop(dir_name).close() + shutil.rmtree(os.path.join(self._cache_root_path, dir_name),=20 + ignore_errors=3DTrue) + + def add_remote(self, url, name, verbose=3DFalse): + remote_name =3D self._create_name(url) + dir_name =3D self._create_name(name) + if dir_name not in self._repos: + raise ValueError + repo =3D self._get_repo(dir_name) + if remote_name in repo.remotes: + if verbose: + print(CACHE_REMOTE_EXISTS.format(remote_name)) + return + self._add_and_fetch_remote(repo, remote_name, url, verbose) + + def remove_remote(self, url, name, verbose=3DFalse): + remote_name =3D self._create_name(url) + dir_name =3D self._create_name(name) + if dir_name not in self._repos: + raise ValueError + repo =3D self._get_repo(dir_name) + if remote_name not in repo.remotes: + raise IndexError + repo.remove_remote(repo.remotes[remote_name]) + + def update_cache(self, url_or_name=3DNone, verbose=3DFalse): + if not self._repos: + raise FileNotFoundError + repo_dirs =3D self._repos.keys() + + if url_or_name is not None: + dir_name =3D self._create_name(url_or_name) + if dir_name in self._repos: + repo_dirs =3D [dir_name] + else: + return + + for dir_name in repo_dirs: + try: + repo =3D self._get_repo(dir_name) + except Exception: + print(CACHE_FAILED_TO_OPEN.format(dir_name)) + continue + for remote in repo.remotes: + if verbose: + print(CACHE_FETCH_REMOTE.format(dir_name, remote.url)) + remote.fetch(progress=3DGitProgressHandler()) + + def clean_cache(self, verbose=3DFalse): + raise NotImplementedError diff --git a/project_utils/project_utils_strings.py b/project_utils/project= _utils_strings.py index 33c22d2..1547978 100644 --- a/project_utils/project_utils_strings.py +++ b/project_utils/project_utils_strings.py @@ -22,3 +22,14 @@ SUBMOD_DEINIT_PATH =3D 'Submodule deinit: {}' SUBMOD_SYNC_PATH =3D 'Submodule sync: {}' SUBMOD_UPDATE_PATH =3D 'Submodule update: {}' SUBMOD_EXCEPTION =3D '- Exception: {}' + +# Caching support strings +CACHE_ADD_REMOTE =3D '+ Adding remote {} ({})' +CACHE_FETCH_REMOTE =3D '+ Fetching data for {} ({})' +CACHE_CHECK_ROOT_DIR =3D '+ Creating cache root directory: {}' +CACHE_FAILED_TO_OPEN =3D '- Failed to open cache: {}' +CACHE_FAILED_TO_CLOSE =3D '- Failed to close cache: {}' +CACHE_REPO_EXISTS =3D '- Repo {} already exists.' +CACHE_ADDING_REPO =3D '+ Adding cache repo {}' +CACHE_REMOVE_REPO =3D '- Removing cache repo: {}' +CACHE_REMOTE_EXISTS =3D '- Remote {} already exists.' -- 2.21.0.windows.1