From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mx.groups.io with SMTP id smtpd.web08.1760.1607022743175244184 for ; Thu, 03 Dec 2020 11:12:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=EaJLht9Z; spf=pass (domain: intel.com, ip: 192.55.52.43, mailfrom: ashley.e.desimone@intel.com) IronPort-SDR: /2IhpIM008M+PZM3k9Q8r+SXQHufBHAD+C9zr44dPU9i2BIJvNzpQrjQwQJcDYCgoA4Chz8EUl I69Jf2EqUN2w== X-IronPort-AV: E=McAfee;i="6000,8403,9824"; a="257967348" X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="257967348" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2020 11:12:22 -0800 IronPort-SDR: +9G6PfTUohqJm4b6dMluuep5I2nVkB0pAAefYkLayHZKi+KIDxLQ4GnctrJiRumXd15L/ZwdF/ QF3sHStD6tlg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="330939300" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga003.jf.intel.com with ESMTP; 03 Dec 2020 11:12:22 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 11:12:21 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 11:12:21 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 3 Dec 2020 11:12:21 -0800 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.100) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 3 Dec 2020 11:11:15 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Zb44KLV/qa4YKPZvOjzJt/eCvkwBnwU73dPdv/oQ2biYpW1ZMLcLBPsnS2Nsp6pdLfEyugtUPt5VhwHjBT8fy4D5Bk/dk/XLRpuj1Mdiii+RTS5y8LTD3wijlIQj0hNdGTFGwlZYf5ar0dz3FzQZMTqmC7CMHjSgZNvnUYdgyj4OATUXV/krJCyTGb3YGCet9ZBn8KZUm9oa8RcH/w/AGpm1HYz8lhjdW1k9mX83bExAn/Vg7sWpuLs3UG521gep8SmHGm0Q8laSZU9WXQCFcY/07qxLLw52sGyxiAJHom283e1PE1DgGFPY4Lv8mR3D3MHJjRIAagNkJokLsV2ZHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JpRNBDN8C79irNTZRB8qLzIDy2fz43Ih/3A6X1qlDtc=; b=jhyhIdBKXHNvJXTxP3v4NXhMLmc/atejLXNaPWoN7ldcspdwD0xiDwejWkSp8aQ8VvDAaGk2T/yqQPIaG0Cbo/Ddyo665/TRFeM+VMZH0UldT0kd2GZEuaQGYCNuAW9wfxqX6qFElOS8is+bY8amoM+XAeqxBjk0aIOZjd1Al7zYrKp4cMd3M2aIhfpyifGV3FT0yPVMd4AUeFWK31RV7fe+U+O/mFSo30ihocaFd1QJlYdqLr01sKIy4Vfq8/fMwBbuGMG9IkppQMIzMKR1xVZ2+xfq+lU/WibF7Z885ga2G3xsC90pOsBxK3JOIU3tnzD7xfzw+/pKqFAKhTlklg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JpRNBDN8C79irNTZRB8qLzIDy2fz43Ih/3A6X1qlDtc=; b=EaJLht9ZZG2A2XR1Fo42g1kvzVXqEutbPXuSiEbntHRmALDLVYpnNQDuaKwvWdURgdq+fM5srTpJuWE7t0SaT7sBLw6PYg3Gw+IspxFKj6R57oTuiI0bu7OJOhBn/Xw8MVl3QiGmDkZZtYrH2f6sWsU0UTgy29N4zgFoD+DtEMk= Received: from BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) by SJ0PR11MB5134.namprd11.prod.outlook.com (2603:10b6:a03:2de::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.21; Thu, 3 Dec 2020 19:11:14 +0000 Received: from BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5]) by BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5%6]) with mapi id 15.20.3611.034; Thu, 3 Dec 2020 19:11:14 +0000 From: "Ashley E Desimone" To: "Bjorge, Erik C" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Thread-Topic: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Thread-Index: AQHWvJ0pg8AioRin4UWHvaJJ3YpFCKnl1taQ Date: Thu, 3 Dec 2020 19:11:14 +0000 Message-ID: References: <737e00b25148bcf2faede39d38b352dd41843090.1605588263.git.erik.c.bjorge@intel.com> In-Reply-To: <737e00b25148bcf2faede39d38b352dd41843090.1605588263.git.erik.c.bjorge@intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7af98883-f5f7-4afe-0d4d-08d897bf3448 x-ms-traffictypediagnostic: SJ0PR11MB5134: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UXj6Nli78W0JfRh/1KbEXq0J+TKPOouHnON6Xn9GvHI2Kjd4bU7GdRvM5B1U2zInaA4/lkoxAfikUx/42Hxt1cexiELwLHbqLHkrxKTON4fbYMt9m5TXSYC1roxqQQk3sYuTbjWdAggNkxRPF4958pJclKP3zYoPfgBpCTkoiG930fdJNFmLEk/LV6CD4/k2xr4xNc0ZPfuAnPXjA99I9EARPnH4rbDdZxQpWzOJ29Sn6QRdyiKOuCS0eDdxSNFdefOOGuutbF+BvdoMKbHAS4QfS33dliSxqMcgWFsLjpzPSiadkh5Ka82OmjkRGgcaPJ796ofinXB5/p1erMLr6w== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3973.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(366004)(346002)(396003)(376002)(39850400004)(8936002)(7696005)(478600001)(71200400001)(9686003)(4326008)(52536014)(53546011)(30864003)(86362001)(5660300002)(66446008)(6506007)(66556008)(2906002)(8676002)(186003)(66946007)(64756008)(66476007)(26005)(110136005)(55016002)(33656002)(83380400001)(76116006)(54906003)(316002)(107886003)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?vky4rWawySfT3w8piH/NEyR8nMucYSDOqcJT7NHEJbQdtd89zsrsR6rZ29Z0?= =?us-ascii?Q?IEqCTDifYb1OG4gEHXmkD0lIWjRBTP0/Qqca0COMQb4Wrn+Lc1m6BEWTtx05?= =?us-ascii?Q?IsKKRD0cuUrA3EPDFIXOARPNIag30qJy/p/qsJ+1u+5AhQ0wkyf0gnHAICvG?= =?us-ascii?Q?rSnhlChNINtpVOLXCiN8K6dNj52ngqDflVrT0g2CPLK9CbWhEyz9Qr8KbX5Q?= =?us-ascii?Q?lM6WdDgnoXUjf10R8X2xS5eQwbo/iaXpM0hNnbRoosuWwGftKvwjZjaCq6YM?= =?us-ascii?Q?07Non0FUrvEoVarfH4LFJ9M5oNRP0f+05epWaIA/q2p4z3cp2cFn3dWZ4MbF?= =?us-ascii?Q?20RiIaS3WnSJPOB8dBA+DkWbX7JCCicVKCYqW9YEwjD0f+fCBOtRUYQpVrN+?= =?us-ascii?Q?cZlYu3pj5Z9La2UDWTvHbQkp+Qyp5alDh7h+9u5MBtLpZi7grXCLYvvNk4mz?= =?us-ascii?Q?Wtz7qX/Sii89B+YH3Neql3V3b/NRVOz4vQhYa7eN3dBdYoQBWbDLaap4Q6my?= =?us-ascii?Q?4tu3s+2+tfdZSDFstrEjeVB8UujoCtBvFfhCdsoCFXkflJRlF2dmFsB7iVgq?= =?us-ascii?Q?e9kmqdoOAHL9OBbI972mx5TV8ty/Zt5F19vBtsNcoTMiag7m6fK7Ajz9ivlO?= =?us-ascii?Q?nNq9UGEaEXgJkhGZ+OuavvEhjUWk9CvF+Dl1PIQG/jbwba9K7sgq0ZbXhSq6?= =?us-ascii?Q?D/wEV9olMbu/OANBzl4pNng2FX5tWWo3rYg1ktE0/Sjp149o0IDcEnvT+zfv?= =?us-ascii?Q?JIQom3/xp52IgPivehvVAKgHUSd5c97Jhsvm5dQDhelBTmhcigjtTZ8u656f?= =?us-ascii?Q?lzUaiCwLTKukrK2f7BY7KaBmNOCdr6+6KOoKlfrEUIA2XxUd7y/YsF6RK489?= =?us-ascii?Q?EkyOJUJbYayCEGno40bs7w4G5SlQJPepngezpo6WOpg5saeRni7Emu7w727t?= =?us-ascii?Q?b8Phy/h9FaFLtTseFjxHYfVUbiuOsnNvv0rB5yJQHH4=3D?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB3973.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7af98883-f5f7-4afe-0d4d-08d897bf3448 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 19:11:14.3470 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: s4KfPl45E1ldK06l1gqebbj6ui++xnGVz664Q2pjt7E8IvZQI86kaGEu5iIJ5O+rJHkpQW+QvBHqdNfRXSzXW6Bpq5r9WRxq/y++gzvNlW0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5134 Return-Path: ashley.e.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Pushed as bbab07f4917f386adc30d65895a82e1a96bb0318 -----Original Message----- From: Erik Bjorge =20 Sent: Monday, November 16, 2020 8:50 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Add cache command Adds a module to add a repo cache and mange it. Also adds a command to man= age the repo cache from EdkRepo. No other commands use the functionality a= t this point. Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge Signed-off-by: Erik Bjorge --- edkrepo/commands/arguments/cache_args.py | 19 ++ edkrepo/commands/cache_command.py | 118 ++++++++++++ edkrepo/commands/humble/cache_humble.py | 17 ++ edkrepo/common/common_c= ache_functions.py | 41 +++++ edkrepo/common/edkrepo_exception.py | 3 + edkrepo/config/config_factory.py | 14 +- edkrepo/config/tool_config.py | 5 +- project_utils/cache.py | 224 +++++++++++++++++++++++ project_utils/project_utils_strings.py | 11 ++ 9 files changed, 448 insertions(+), 4 deletions(-) create mode 100644 edk= repo/commands/arguments/cache_args.py create mode 100644 edkrepo/commands/cache_command.py create mode 100644 e= dkrepo/commands/humble/cache_humble.py create mode 100644 edkrepo/common/common_cache_functions.py create mode 100644 project_utils/cache.py diff --git a/edkrepo/commands/arguments/cache_args.py b/edkrepo/commands/ar= guments/cache_args.py new file mode 100644 index 0000000..0080536 --- /dev/null +++ b/edkrepo/commands/arguments/cache_args.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +# +## @file +# cache_args.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +''' Contains the help and description strings for arguments in the=20 +cache command meta data. +''' +COMMAND_DESCRIPTION =3D ('Manages local caching support for project repos.= The goal of this feature ' + 'is to improve clone performance')=20 +COMMAND_ENABLE_HELP =3D 'Enables caching support on the system.' +COMMAND_DISABLE_HELP =3D 'Disables caching support on the system.' +COMMAND_UPDATE_HELP =3D 'Update the repo cache for all cached projects.' +COMMAND_INFO_HELP =3D 'Display the current cache information.' +COMMAND_PROJECT_HELP =3D 'Project to add to the cache.' diff --git a/edkrepo/commands/cache_command.py b/edkrepo/commands/cache_com= mand.py new file mode 100644 index 0000000..9f0d6e9 --- /dev/null +++ b/edkrepo/commands/cache_command.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +# +## @file +# cache_command.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +import edkrepo.commands.arguments.cache_args as arguments from=20 +edkrepo.commands.edkrepo_command import EdkrepoCommand from=20 +edkrepo.commands.edkrepo_command import SourceManifestRepoArgument from=20 +edkrepo.commands.humble.cache_humble import CACHE_ENABLED, CACHE_FETCH,=20 +CACHE_INFO from edkrepo.commands.humble.cache_humble import=20 +CACHE_INFO_LINE, PROJECT_NOT_FOUND, NO_INSTANCE from=20 +edkrepo.commands.humble.cache_humble import UNABLE_TO_LOAD_MANIFEST,=20 +UNABLE_TO_PARSE_MANIFEST from edkrepo.common.common_cache_functions=20 +import add_missing_cache_repos from=20 +edkrepo.common.common_cache_functions import get_repo_cache_obj from=20 +edkrepo.common.edkrepo_exception import EdkrepoCacheException from=20 +edkrepo.common.workspace_maintenance.manifest_repos_maintenance import=20 +find_project_in_all_indices from edkrepo.config.config_factory import=20 +get_workspace_manifest from edkrepo_manifest_parser.edk_manifest import=20 +ManifestXml + + +class CacheCommand(EdkrepoCommand): + def __init__(self): + super().__init__() + + def get_metadata(self): + metadata =3D {} + metadata['name'] =3D 'cache' + metadata['help-text'] =3D arguments.COMMAND_DESCRIPTION + args =3D [] + metadata['arguments'] =3D args + args.append({'name': 'enable', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_ENABLE_HELP}) + args.append({'name': 'disable', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_DISABLE_HELP}) + args.append({'name': 'update', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_UPDATE_HELP}) + args.append({'name': 'info', + 'positional': False, + 'required': False, + 'help-text': arguments.COMMAND_INFO_HELP}) + args.append({'name': 'project', + 'positional': True, + 'required': False, + 'help-text': arguments.COMMAND_PROJECT_HELP}) + args.append(SourceManifestRepoArgument) + return metadata + + def run_command(self, args, config): + # Process enable disable requests + if args.disable: + config['user_cfg_file'].set_caching_state(False) + elif args.enable: + config['user_cfg_file'].set_caching_state(True) + + # Get the current state now that we have processed enable/disable + cache_state =3D config['user_cfg_file'].caching_state + print(CACHE_ENABLED.format(cache_state)) + if not cache_state: + return + + # State is enabled so make sure cache directory exists + cache_obj =3D get_repo_cache_obj(config) + + # Check to see if a manifest was provided and add any missing remo= tes + manifest =3D None + if args.project is not None: + manifest =3D _get_manifest(args.project, config, args.source_m= anifest_repo) + else: + try: + manifest =3D get_workspace_manifest() + except Exception: + pass + + # If manifest is provided attempt to add any remotes that do not e= xist + if manifest is not None: + add_missing_cache_repos(cache_obj, manifest, True) + + # Display all the cache information + if args.info: + print(CACHE_INFO) + info =3D cache_obj.get_cache_info(args.verbose) + for item in info: + print(CACHE_INFO_LINE.format(item.path, item.remote,=20 + item.url)) + + # Do an update if requested + if args.update: + print(CACHE_FETCH) + cache_obj.update_cache(verbose=3DTrue) + + # Close the cache repos + cache_obj.close(args.verbose) + + +def _get_manifest(project, config, source_manifest_repo=3DNone): + try: + manifest_repo, source_cfg, manifest_path =3D find_project_in_all_i= ndices( + project, + config['cfg_file'], + config['user_cfg_file'], + PROJECT_NOT_FOUND.format(project), + NO_INSTANCE.format(project), + source_manifest_repo) + except Exception: + raise EdkrepoCacheException(UNABLE_TO_LOAD_MANIFEST) + try: + manifest =3D ManifestXml(manifest_path) + except Exception: + raise EdkrepoCacheException(UNABLE_TO_PARSE_MANIFEST) + return manifest diff --git a/edkrepo/commands/humble/cache_humble.py b/edkrepo/commands/hum= ble/cache_humble.py new file mode 100644 index 0000000..4f318ac --- /dev/null +++ b/edkrepo/commands/humble/cache_humble.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +# +## @file +# cache_humble.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +CACHE_ENABLED =3D 'Caching Enabled: {}' +CACHE_INFO =3D 'Cache Information:' +CACHE_INFO_LINE =3D '+ {}\n {} ({})' +CACHE_FETCH =3D 'Fetching all remotes... (this could take a while)' +PROJECT_NOT_FOUND =3D 'Project {} does not exist' +NO_INSTANCE =3D 'Unable to determine instance to use for {}' +UNABLE_TO_LOAD_MANIFEST =3D 'Unable to load manifest file.' +UNABLE_TO_PARSE_MANIFEST =3D 'Failed to parse manifest file.' diff --git a/edkrepo/common/common_cache_functions.py b/edkrepo/common/comm= on_cache_functions.py new file mode 100644 index 0000000..84bd3ed --- /dev/null +++ b/edkrepo/common/common_cache_functions.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +# +## @file +# common_cache_functions.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +import os + +from edkrepo.config.config_factory import=20 +get_edkrepo_global_data_directory from edkrepo.config.tool_config=20 +import SUBMODULE_CACHE_REPO_NAME from project_utils.cache import=20 +RepoCache + + +def get_global_cache_directory(config): + if config['user_cfg_file'].caching_state: + return os.path.join(get_edkrepo_global_data_directory(), '.cache') + return None + + +def get_repo_cache_obj(config): + cache_obj =3D None + cache_directory =3D get_global_cache_directory(config) + if cache_directory is not None: + cache_obj =3D RepoCache(cache_directory) + cache_obj.open() + return cache_obj + + +def add_missing_cache_repos(cache_obj, manifest, verbose=3DFalse): + print('Adding and fetching new remotes... (this could take a while)') + for remote in manifest.remotes: + cache_obj.add_repo(url=3Dremote.url, verbose=3Dverbose) + alt_submodules =3D manifest.submodule_alternate_remotes + if alt_submodules: + print('Adding and fetching new submodule remotes... (this could al= so take a while)') + cache_obj.add_repo(name=3DSUBMODULE_CACHE_REPO_NAME, verbose=3Dver= bose) + for alt in alt_submodules: + cache_obj.add_remote(alt.alternate_url,=20 +SUBMODULE_CACHE_REPO_NAME, verbose) diff --git a/edkrepo/common/edkrepo_exception.py b/edkrepo/common/edkrepo_e= xception.py index a56e709..b3f2300 100644 --- a/edkrepo/common/edkrepo_exception.py +++ b/edkrepo/common/edkrepo_exception.py @@ -98,3 +98,6 @@ class EdkrepoGitConfigSetupException(EdkrepoException): def __init__(self, message): super().__init__(message, 131) =20 +class EdkrepoCacheException(EdkrepoException): + def __init__(self, message): + super().__init__(message, 132) diff --git a/edkrepo/config/config_factory.py b/edkrepo/config/config_facto= ry.py index fe69460..3680c0b 100644 --- a/edkrepo/config/config_factory.py +++ b/edkrepo/config/config_factory.py @@ -225,10 +225,20 @@ class GlobalUserConfig(BaseConfig): self.filename =3D os.path.join(get_edkrepo_global_data_directory()= , "edkrepo_user.cfg") self.prop_list =3D [ CfgProp('scm', 'mirror_geo', 'geo', 'none', False), - CfgProp('send-review', 'max-patch-set', 'max_patch_set', '10',= False) - ] + CfgProp('send-review', 'max-patch-set', 'max_patch_set', '10',= False), + CfgProp('caching', 'enable-caching', 'enable_caching_text',=20 + 'false', False)] super().__init__(self.filename, get_edkrepo_global_data_directory(= ), False) =20 + @property + def caching_state(self): + return self.enable_caching_text.lower() =3D=3D 'true' + + def set_caching_state(self, enable): + if enable: + self.enable_caching_text =3D 'true' + else: + self.enable_caching_text =3D 'false' + @property def max_patch_set_int(self): try: diff --git a/edkrepo/config/tool_config.py b/edkrepo/config/tool_config.py = index eee1326..81f4ddf 100644 --- a/edkrepo/config/tool_config.py +++ b/edkrepo/config/tool_config.py @@ -1,10 +1,11 @@ #!/usr/bin/env python3 # ## @file -# tool)config.py +# tool_config.py # # Copyright (c) 2020, Intel Corporation. All rights reserved.
# SPDX-= License-Identifier: BSD-2-Clause-Patent # =20 -CI_INDEX_FILE_NAME =3D 'CiIndex.xml' \ No newline at end of file +CI_INDEX_FILE_NAME =3D 'CiIndex.xml' +SUBMODULE_CACHE_REPO_NAME =3D 'submodule-cache' diff --git a/project_utils/cache.py b/project_utils/cache.py new file mode = 100644 index 0000000..8efd411 --- /dev/null +++ b/project_utils/cache.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +# +## @file +# cache.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # from collections import=20 +namedtuple import os import shutil + +from git import Repo + +from edkrepo.common.progress_handler import GitProgressHandler from=20 +project_utils.project_utils_strings import CACHE_ADD_REMOTE,=20 +CACHE_ADDING_REPO, CACHE_CHECK_ROOT_DIR from=20 +project_utils.project_utils_strings import CACHE_FAILED_TO_CLOSE,=20 +CACHE_FAILED_TO_OPEN, CACHE_FETCH_REMOTE from=20 +project_utils.project_utils_strings import CACHE_REMOTE_EXISTS,=20 +CACHE_REMOVE_REPO, CACHE_REPO_EXISTS + +CacheInfo =3D namedtuple('CacheInfo', ['path', 'remote', 'url']) + + +class RepoCache(object): + """ + Provides basic management of a cache repo. + """ + def __init__(self, path): + self._cache_root_path =3D path + self._repos =3D {} + + def _create_name(self, url_or_name): + """ + Used to create consistent repo and remote names + """ + dir_name =3D url_or_name.split('/')[-1] + if not dir_name.endswith('.git'): + dir_name +=3D '.git' + return dir_name + + def _get_repo_path(self, dir_name): + return os.path.join(self._cache_root_path, dir_name) + + def _get_repo(self, dir_name): + """ + Returns the git repo object for the cache repo. + + Raises FileNotFoundError if the cache directory does not exist. + Raises IOError if the repo cannot be opened + """ + repo_path =3D self._get_repo_path(dir_name) + if not os.path.isdir(repo_path): + raise FileNotFoundError + try: + repo =3D Repo(repo_path) + except Exception: + raise IOError + return repo + + def _get_cache_dirs(self): + if not os.path.isdir(self._cache_root_path): + raise FileNotFoundError + return [x for x in os.listdir(self._cache_root_path) if=20 + os.path.isdir(self._get_repo_path(x))] + + def _add_and_fetch_remote(self, repo, remote_name, url, verbose=3DFals= e): + if verbose: + print(CACHE_ADD_REMOTE.format(remote_name, url)) + repo.create_remote(remote_name, url) + if verbose: + print(CACHE_FETCH_REMOTE.format(remote_name, url)) + repo.remotes[remote_name].fetch(progress=3DGitProgressHandler()) + + def open(self, verbose=3DFalse): + """ + Opens all cache repos. + + Raises FileNotFoundError if the cache directory does not exist. + """ + if not self._repos: + if not os.path.isdir(self._cache_root_path): + if verbose: + print(CACHE_CHECK_ROOT_DIR.format(self._cache_root_pat= h)) + os.makedirs(self._cache_root_path) + + for dir_name in self._get_cache_dirs(): + try: + self._repos[dir_name] =3D self._get_repo(dir_name) + except Exception: + if verbose: + print(CACHE_FAILED_TO_OPEN.format(dir_name)) + + def close(self, verbose=3DFalse): + """ + Closes all cache repos. + """ + for dir_name in self._repos: + try: + self._repos[dir_name].close() + except Exception: + if verbose: + print(CACHE_FAILED_TO_CLOSE.format(dir_name)) + self._repos =3D {} + + def get_cache_path(self, url_or_name): + dir_name =3D self._create_name(url_or_name) + if dir_name not in self._repos: + return None + return self._get_repo_path(dir_name) + + def get_cache_info(self, verbose=3DFalse): + """ + Returns a list of remotes currently configured in the cache. + + Raises FileNotFoundError if the cache repo is not open. + """ + ret_val =3D [] + for dir_name in self._repos: + for remote in self._repos[dir_name].remotes: + ret_val.append(CacheInfo(self._get_repo_path(dir_name), re= mote.name, remote.url)) + return ret_val + + def delete_cache_root(self, verbose=3DFalse): + """ + Deletes the cache root directory and all caches. + """ + if os.path.isdir(self._cache_root_path): + if self._repos: + self.close() + shutil.rmtree(self._cache_root_path, ignore_errors=3DTrue) + + def add_repo(self, url=3DNone, name=3DNone, verbose=3DFalse): + """ + Adds a repo to the cache if it does not already exist. + + """ + remote_name =3D None + if url is None and name is None: + raise ValueError + elif name is not None: + dir_name =3D self._create_name(name) + else: + dir_name =3D self._create_name(url) + if url is not None: + remote_name =3D self._create_name(url) + repo_path =3D self._get_repo_path(dir_name) + + if dir_name in self._repos: + if verbose: + print(CACHE_REPO_EXISTS.format(dir_name)) + else: + if verbose: + print(CACHE_ADDING_REPO.format(dir_name)) + os.makedirs(repo_path) + self._repos[dir_name] =3D Repo.init(repo_path, bare=3DTrue) + + if remote_name is not None and remote_name not in self._repos[dir_= name].remotes: + self._add_and_fetch_remote(self._get_repo(dir_name), remote_na= me, url) + return dir_name + + def remove_repo(self, url=3DNone, name=3DNone, verbose=3DFalse): + """ + Removes a remote from the cache repo if it exists + + Raises FileNotFoundError if the cache repo is not open. + """ + if url is None and name is None: + raise ValueError + elif name is not None: + dir_name =3D self._create_name(name) + else: + dir_name =3D self._create_name(url) + if dir_name not in self._repos: + return + if verbose: + print(CACHE_REMOVE_REPO.format(dir_name)) + self._repos.pop(dir_name).close() + shutil.rmtree(os.path.join(self._cache_root_path, dir_name),=20 + ignore_errors=3DTrue) + + def add_remote(self, url, name, verbose=3DFalse): + remote_name =3D self._create_name(url) + dir_name =3D self._create_name(name) + if dir_name not in self._repos: + raise ValueError + repo =3D self._get_repo(dir_name) + if remote_name in repo.remotes: + if verbose: + print(CACHE_REMOTE_EXISTS.format(remote_name)) + return + self._add_and_fetch_remote(repo, remote_name, url, verbose) + + def remove_remote(self, url, name, verbose=3DFalse): + remote_name =3D self._create_name(url) + dir_name =3D self._create_name(name) + if dir_name not in self._repos: + raise ValueError + repo =3D self._get_repo(dir_name) + if remote_name not in repo.remotes: + raise IndexError + repo.remove_remote(repo.remotes[remote_name]) + + def update_cache(self, url_or_name=3DNone, verbose=3DFalse): + if not self._repos: + raise FileNotFoundError + repo_dirs =3D self._repos.keys() + + if url_or_name is not None: + dir_name =3D self._create_name(url_or_name) + if dir_name in self._repos: + repo_dirs =3D [dir_name] + else: + return + + for dir_name in repo_dirs: + try: + repo =3D self._get_repo(dir_name) + except Exception: + print(CACHE_FAILED_TO_OPEN.format(dir_name)) + continue + for remote in repo.remotes: + if verbose: + print(CACHE_FETCH_REMOTE.format(dir_name, remote.url)) + remote.fetch(progress=3DGitProgressHandler()) + + def clean_cache(self, verbose=3DFalse): + raise NotImplementedError diff --git a/project_utils/project_utils_strings.py b/project_utils/project= _utils_strings.py index 33c22d2..1547978 100644 --- a/project_utils/project_utils_strings.py +++ b/project_utils/project_utils_strings.py @@ -22,3 +22,14 @@ SUBMOD_DEINIT_PATH =3D 'Submodule deinit: {}' SUBMOD_SYNC_PATH =3D 'Submodule sync: {}' SUBMOD_UPDATE_PATH =3D 'Submodule update: {}' SUBMOD_EXCEPTION =3D '- Exception: {}' + +# Caching support strings +CACHE_ADD_REMOTE =3D '+ Adding remote {} ({})' +CACHE_FETCH_REMOTE =3D '+ Fetching data for {} ({})' +CACHE_CHECK_ROOT_DIR =3D '+ Creating cache root directory: {}' +CACHE_FAILED_TO_OPEN =3D '- Failed to open cache: {}' +CACHE_FAILED_TO_CLOSE =3D '- Failed to close cache: {}' +CACHE_REPO_EXISTS =3D '- Repo {} already exists.' +CACHE_ADDING_REPO =3D '+ Adding cache repo {}' +CACHE_REMOVE_REPO =3D '- Removing cache repo: {}' +CACHE_REMOTE_EXISTS =3D '- Remote {} already exists.' -- 2.21.0.windows.1