From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mx.groups.io with SMTP id smtpd.web09.1641.1607022431145084156 for ; Thu, 03 Dec 2020 11:07:11 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=yPHVpjzz; spf=pass (domain: intel.com, ip: 134.134.136.24, mailfrom: ashley.e.desimone@intel.com) IronPort-SDR: JJmKVeOXSHOG1ru0v7Ch1OgPpDKOElgjJN71WIYmGUA8fnk+pdTrasyBwRLGxiLKTqJxJGc2+/ sy0wgvIyW8/A== X-IronPort-AV: E=McAfee;i="6000,8403,9824"; a="173415140" X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="173415140" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2020 11:07:10 -0800 IronPort-SDR: aKmtSI8yRwcjbaXTRrKgmpR+Cux17mHjb7GQ8By1ava/TmVU+BFnk+CFYQmNLHY3f22Nr77LGH CM/8H70mBuXA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="550626591" Received: from fmsmsx606.amr.corp.intel.com ([10.18.126.86]) by orsmga005.jf.intel.com with ESMTP; 03 Dec 2020 11:07:09 -0800 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx606.amr.corp.intel.com (10.18.126.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 11:07:09 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 3 Dec 2020 11:07:09 -0800 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.173) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 3 Dec 2020 11:07:09 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BgF8Ahss5vmtldAqgzJTx81sphB6mGAi3YyeSYx+lNSpou+vcObBR33rR+fKbqd0xCuOn4+dyQZSPjnoE10HBEoWZzSRHu5byJlL0fHS8vS4nSNAGGd951JXi1Hzwhq5a5VOXAbdfeSkoKy4LX/CgNtKru04OyOSZ8fmkLDpP/s9EqfN7SoZmPHAp0jrf03rR+NWgAf540z7szUO2jjrc+dGt48Vt8s4G1QPZrg4nRJfIig1xA/3tOa+RENTGrUHgGlKb6eW5X+ih8qx5s345g5RYwqIKC0F8jSeGUrMvxbAH3MgILip6bCWIMewZO6lMRMzwUCKirXiV16qiBnJCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5euH3ac20GY7IlYyz4PnxAAgt9TS0QvAlE57wFKbiXA=; b=SHcbbo/Ust8BQEJ+VBE1UgrTIDyDLU67VDWmUPVFyNUqFF4+FjsnauQ+DeE7or40ErzRcfJTQV+XaFrNR18dpFPEOqBdSeIN/9PMlovLAX/yB21ckDcULSvDrlH03dcnjIO2+45KTiQBKfD60mmG2NMCh9Xg/esaqLoZYJ/sHeC1oA8ULsvHSPvuy2cZ3Og5DkJE+clJgVmllNRnsHxF1Bjy1gZb3LYVBmFeSDwhAExyJPPtU3LlBtNfGbPafPlZBmNP3kDOGv6jgQyVagvr1/ccdEAfBjnvsQ5QaSY1lLJmzdU1Z+BlaOhtcZZNwZ0KICBPR0JB36X/I59WXB5+MA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5euH3ac20GY7IlYyz4PnxAAgt9TS0QvAlE57wFKbiXA=; b=yPHVpjzzwUo/LfeCd58PtYSGJQ7GNRRe1W2VD1tsRRlm7bBiqSlPfdQxnzIV2A/bmMKrgq/NhnL/kpquJs9xJFCfynYR9i94q0i1gwFVntiMIfE7eRyK/3WYhfb6Ra+aGPGb1sguo0sLpoQdEuxjDF0tRdx4A25JYf7+uKZA+L4= Received: from BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) by BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.31; Thu, 3 Dec 2020 19:07:07 +0000 Received: from BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5]) by BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5%6]) with mapi id 15.20.3611.034; Thu, 3 Dec 2020 19:07:07 +0000 From: "Ashley E Desimone" To: "Bjorge, Erik C" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo cache support. Thread-Topic: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo cache support. Thread-Index: AQHWvJ0m7C8nCn72hkucu4vaWYeZFanl1bxg Date: Thu, 3 Dec 2020 19:07:07 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 05901ce4-b2d8-4007-f195-08d897bea101 x-ms-traffictypediagnostic: BY5PR11MB3973: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: AVPmtcVgajOl08zA4bMBanOLDL93A4LZMii/bYyzeDB+a7abMziFYLW+8kUfdu+iW/ACwy4H1I7GVK9eOckgQkfldf/7XgbrYnT2nQS3cH95YEQ3ORWAjDYKZqmwGgjmFWBt3jEappxXIVe7YeL6gBNnOpyTn5K3u8YAp+T90gXWEA4Jv6g14Fk8P3m1YQq4XEQOlZCLmkcw9Ag02hE/syKnSr16iHrin9LmcxaZMdpL9HqLtkSr6KqXooU2jjBIgaUedHvnCZQSuk54IGmGimyLs2tn0YXdYOe9zrw0qwg+HlbXn1d97uWCk3AYQX3vcxV5HQNqKj6tAz/MEKn4iA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3973.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(346002)(376002)(136003)(366004)(396003)(64756008)(53546011)(2906002)(7696005)(5660300002)(30864003)(66946007)(52536014)(26005)(66446008)(186003)(316002)(71200400001)(86362001)(54906003)(6506007)(110136005)(33656002)(66476007)(66556008)(478600001)(83380400001)(8936002)(8676002)(4326008)(76116006)(9686003)(107886003)(55016002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?VTYfV6SI1rlS13MYLmMpqLB+3XgpKiIxdHRF87uizh8fx8vEe398M0mi1HZv?= =?us-ascii?Q?AsxNqzsqHLwLk4LMTUmBtWIWZlJNXl26Sm3ItBcHHQEjzmM6+bFFHd8R5nuA?= =?us-ascii?Q?RbN7VT4ifrGiVzpZVY/2L0qRr/knXPWcvA0WlwBV9XpLjGEZ35Gp8VxeunH5?= =?us-ascii?Q?aNnoB4VwCl0s6JspiPylhqw6jKE4XR9jVce2N98ePMc6olFej+N5uHTjnCGC?= =?us-ascii?Q?9R23eFxNSKi47a62njiljcEWGv149TxYs/W163eGEQddPRNXMX8ZxPi7VBvS?= =?us-ascii?Q?USR+kSPbfe8yGL4EF1mTKnD7N9W3yd6Aa8Dtgc/SIJp9+wOoH6Hfs/70KPSx?= =?us-ascii?Q?gDTkDkulZBYvIELO1fCtla82uNo011J8p353cw9StDukbc3Z9uSxNVprjYMc?= =?us-ascii?Q?/QJ1QRWbVVMCwBqK+ipdzfOgVfR3WndW5xTttOQTgbPKUimRRUoayJ8xsPCL?= =?us-ascii?Q?fUianEPWCMx+QnXRHHwWbCLN5IRGWqirafw57QZyKOKqqCAOIW7R8/E7MJ7n?= =?us-ascii?Q?4TD54BUYmyPizwjhZmcs+ZPAnwb2K7+oBpfls4oLQXEGRgs8S0Mdw7rKmfjS?= =?us-ascii?Q?311bENKHreCdLEOpdvk9KNh7bFdoAzQ5Fzb8GI8IzHKLtBBvjcetJQar8gKK?= =?us-ascii?Q?feZJPKlmnufoYQXrjByMwXECjHZAHkmj1StdEZiCrFP7+8kcNbGkcV5RNWqg?= =?us-ascii?Q?fflOuVV6YJASA/8ijEfCBaXgbEvHRh/T4a6GtfeQLJJ7pMM+L00qekseKUZI?= =?us-ascii?Q?r+qPYMR8ZmO7xiPUqZd4m5mlknZ7T/WRxie36z3FQEfyWmlRQerZtZrpqWSj?= =?us-ascii?Q?1poDrFu5L1uhVyJxKBU8l3z0IgFWCaXUdS8FrVr5INYPG8f3zBmNDXsZakyd?= =?us-ascii?Q?IqEcaDAkSG7eDXz3pgeZRUG3k9QGgXEKjIrRWho4UKaVUEidwM4dyt2Rmt23?= =?us-ascii?Q?nDfUYCZaeRh8JWg9IIv/hVJVXMgy9/DX7b7CrP3c9LI=3D?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB3973.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 05901ce4-b2d8-4007-f195-08d897bea101 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 19:07:07.2705 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: C2L1GyryY8Zt/M9knikhWVyyJk/cYRgBv5YxKDu61uxge1YGdmjkUdvDTci1EobdBheUXs4jX8ezKJe8soM8IKatlZVXalEWqICpieIN8IY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB3973 Return-Path: ashley.e.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Ashley DeSimone -----Original Message----- From: Erik Bjorge =20 Sent: Monday, November 16, 2020 8:50 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo = cache support. This changes enables the local repo cache to be used when cloning and synci= ng changes. The repo cache applies to submodules as well. Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge Signed-off-by: Erik Bjorge --- edkrepo/commands/checkout_command.py | 3 ++- edkrepo/commands/checkout_pin_command.py | 8 +++++++- edkrepo/commands/clone_command.py | 15 ++++++++++++-- edkrepo/commands/sync_command.py | 12 +++++++++--- edkrepo/common/common_repo_functions.py | 25 ++++++++++++++++++++---- project_utils/submodule.py | 13 ++++++++---- 6 files changed, 61 insertions(+), 15 deletions(-) diff --git a/edkrepo/commands/checkout_command.py b/edkrepo/commands/checko= ut_command.py index 0169f30..2ce26c0 100644 --- a/edkrepo/commands/checkout_command.py +++ b/edkrepo/commands/checkout_command.py @@ -16,6 +16,7 @@ import os from edkrepo.commands.edkrepo_command import EdkrepoCommand, OverrideArgum= ent import edkrepo.commands.arguments.checkout_args as arguments import e= dkrepo.commands.humble.checkout_humble as humble +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import checkout, combination_is_= in_manifest from edkrepo.common.edkrepo_exception import EdkrepoInvalidPar= ametersException from edkrepo.config.config_factory import get_workspace_m= anifest @@ -42,6 +43,6 @@ class CheckoutCommand(EdkrepoCommand): =20 def run_command(self, args, config): if combination_is_in_manifest(args.Combination, get_workspace_mani= fest()): - checkout(args.Combination, args.verbose, args.override) + checkout(args.Combination, args.verbose, args.override,=20 + get_repo_cache_obj(config)) else: raise EdkrepoInvalidParametersException(humble.NO_COMBO.format= (args.Combination)) diff --git a/edkrepo/commands/checkout_pin_command.py b/edkrepo/commands/ch= eckout_pin_command.py index 1c58113..0ad1b48 100644 --- a/edkrepo/commands/checkout_pin_command.py +++ b/edkrepo/commands/checkout_pin_command.py @@ -14,6 +14,7 @@ from git import Repo from edkrepo.commands.edkrepo_command import EdkrepoCommand, OverrideArgum= ent, SourceManifestRepoArgument import edkrepo.commands.arguments.checkout= _pin_args as arguments import edkrepo.commands.humble.checkout_pin_humble = as humble +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import sparse_checkout_enabled, = reset_sparse_checkout, sparse_checkout from edkrepo.common.common_repo_fun= ctions import check_dirty_repos, checkout_repos, combinations_in_manifest = from edkrepo.common.humble import SPARSE_CHECKOUT, SPARSE_RESET, SUBMODULE_= DEINIT_FAILED @@ -21,6 +22,7 @@ from edkrepo.common.edkrepo_exception impor= t EdkrepoInvalidParametersException, from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impor= t list_available_manifest_repos from edkrepo.common.workspace_maintenance.= manifest_repos_maintenance import find_source_manifest_repo from edkrepo.c= onfig.config_factory import get_workspace_path, get_workspace_manifest +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo_manifest_parser.edk_manifest import ManifestXml from project= _utils.submodule import deinit_full, maintain_submodules =20 @@ -82,7 +84,11 @@ class CheckoutPinCommand(EdkrepoCommand): checkout_repos(args.verbose, args.override, pin_repo_sources, = workspace_path, manifest) manifest.write_current_combo(humble.PIN_COMBO.format(args.pinf= ile)) finally: - maintain_submodules(workspace_path, pin, submodule_combo, args= .verbose) + cache_path =3D None + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_path, pin, submodule_combo,=20 + args.verbose, cache_path) if sparse_enabled: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, pin_repo_sources, manifest= ) diff --git a/edkrepo/commands/clone_command.py b/edkrepo/commands/clone_c= ommand.py index 8769102..56c15c9 100644 --- a/edkrepo/commands/clone_command.py +++ b/edkrepo/commands/clone_command.py @@ -14,6 +14,8 @@ import sys from edkrepo.commands.edkrepo_command import EdkrepoCommand from edkrepo.= commands.edkrepo_command import SubmoduleSkipArgument, SourceManifestRepoAr= gument import edkrepo.commands.arguments.clone_args as arguments +from edkrepo.common.common_cache_functions import get_repo_cache_obj=20 +from edkrepo.common.common_cache_functions import=20 +add_missing_cache_repos from edkrepo.common.common_repo_functions import clone_repos, sparse_check= out, verify_single_manifest from edkrepo.common.common_repo_functions impo= rt update_editor_config, combinations_in_manifest from edkrepo.common.comm= on_repo_functions import write_included_config, write_conditional_include @= @ -28,6 +30,7 @@ from edkrepo.common.workspace_maintenance.manifest_repos_m= aintenance import list from edkrepo.common.workspace_maintenance.humble.ma= nifest_repos_maintenance_humble import PROJ_NOT_IN_REPO, SOURCE_MANIFEST_RE= PO_NOT_FOUND from edkrepo_manifest_parser.edk_manifest import CiIndexXml, = ManifestXml from project_utils.submodule import maintain_submodules +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME =20 =20 class CloneCommand(EdkrepoCommand): @@ -151,11 +154,19 @@ class CloneCommand(EdkrepoCommand): # Set up submodule alt url config settings prior to cloning any re= pos submodule_included_configs =3D write_included_config(manifest.remo= tes, manifest.submodule_alternate_remotes, local_manifest_dir) write_conditional_include(workspace_dir, repo_sources_to_clone, su= bmodule_included_configs) - clone_repos(args, workspace_dir, repo_sources_to_clone, project_cl= ient_side_hooks, config, manifest) + + # Determine if caching is going to be used and then clone + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + add_missing_cache_repos(cache_obj, manifest, args.verbose) + clone_repos(args, workspace_dir, repo_sources_to_clone,=20 + project_client_side_hooks, config, manifest, cache_obj) =20 # Init submodules if not args.skip_submodule: - maintain_submodules(workspace_dir, manifest, combo_name, args.= verbose) + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_dir, manifest, combo_name,=20 + args.verbose, cache_path) =20 # Perform a sparse checkout if requested. use_sparse =3D args.sparse diff --git a/edkrepo/commands/sync_command.py b/edkrepo/commands/sync_comma= nd.py index c4ee330..ff48f50 100644 --- a/edkrepo/commands/sync_command.py +++ b/edkrepo/commands/sync_command.py @@ -32,6 +32,7 @@ from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_R= EPO, SYNC_NEEDS_REBASE, from edkrepo.common.humble import SYNC_BRANCH_CHAN= GE_ON_LOCAL, SYNC_INCOMPATIBLE_COMBO from edkrepo.common.humble import SYN= C_REBASE_CALC_FAIL from edkrepo.common.pathfix import get_actual_path, exp= anduser +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import clone_repos, sparse_check= out_enabled from edkrepo.common.common_repo_functions import reset_sparse_= checkout, sparse_checkout, verify_single_manifest from edkrepo.common.comm= on_repo_functions import checkout_repos, check_dirty_repos @@ -47,6 +48,7 @= @ from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impo= rt list from edkrepo.common.ui_functions import init_color_console from e= dkrepo.config.config_factory import get_workspace_path, get_workspace_manif= est, get_edkrepo_global_data_directory from edkrepo.config.config_factory = import get_workspace_manifest_file +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml = from project_utils.submodule import deinit_submodules, maintain_submodules =20 @@ -102,7 +104,7 @@ class SyncCommand(EdkrepoCommand): if not args.update_local_manifest: self.__check_for_new_manifest(args, config, initial_manifest, = workspace_path, global_manifest_directory) check_dirty_repos(initial_manifest, workspace_path) - + # Determine if sparse checkout needs to be disabled for this opera= tion sparse_settings =3D initial_manifest.sparse_settings sparse_enabled =3D sparse_checkout_enabled(workspace_path, initial= _sources) @@ -116,7 +118,7 @@ class SyncCommand(EdkrepoCommand): reset_sparse_checkout(workspace_path, initial_sources) =20 # Get the latest manifest if requested - if args.update_local_manifest: # NOTE: hyphens in arg name replac= ed with underscores due to argparse + if args.update_local_manifest: # NOTE: hyphens in arg name=20 + replaced with underscores due to argparse self.__update_local_manifest(args, config, initial_manifest, w= orkspace_path, global_manifest_directory) manifest =3D get_workspace_manifest() if args.update_local_manifest: @@ -212,7 +214,11 @@ class SyncCommand(EdkrepoCommand): =20 # Initialize submodules if not args.skip_submodule: - maintain_submodules(workspace_path, manifest, current_combo, a= rgs.verbose) + cache_path =3D None + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_path, manifest,=20 + current_combo, args.verbose, cache_path) =20 # Restore sparse checkout state if sparse_enabled: diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/commo= n_repo_functions.py index 2277c1e..336661a 100644 --- a/edkrepo/common/common_repo_functions.py +++ b/edkrepo/common/common_repo_functions.py @@ -56,6 +56,7 @@ from project_utils.sparse import BuildInfo, process_spars= e_checkout from edkrepo.config.config_factory import get_workspace_path f= rom edkrepo.config.config_factory import get_workspace_manifest from edkre= po.config.tool_config import CI_INDEX_FILE_NAME +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo.common.edkrepo_exception import EdkrepoInvalidParametersExcep= tion from edkrepo_manifest_parser.edk_manifest import CiIndexXml, Manifest= Xml from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException,= EdkrepoGitException, EdkrepoWarningException @@ -75,12 +76,25 @@ CLEAR_LIN= E =3D '\x1b[K' DEFAULT_REMOTE_NAME =3D 'origin' PRIMARY_REMOTE_NAME =3D 'primary' =20 -def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, manifest): + +def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, manifest, cache_obj=3DNone): for repo_to_clone in repos_to_clone: local_repo_path =3D os.path.join(workspace_dir, repo_to_clone.root= ) local_repo_url =3D repo_to_clone.remote_url + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(local_repo_url) print("Cloning from: " + str(local_repo_url)) - repo =3D Repo.clone_from(local_repo_url, local_repo_path, progress= =3DGitProgressHandler(), no_checkout=3DTrue) + if cache_path is not None: + print('+ Using cache at {}'.format(cache_path)) + repo =3D Repo.clone_from(local_repo_url, local_repo_path, + progress=3DGitProgressHandler(), + reference_if_able=3Dcache_path, + no_checkout=3DTrue) + else: + repo =3D Repo.clone_from(local_repo_url, local_repo_path, + progress=3DGitProgressHandler(), + no_checkout=3DTrue) # Fetch notes repo.remotes.origin.fetch("refs/notes/*:refs/notes/*") =20 @@ -427,7 +441,7 @@ def combination_is_in_manifest(combination, manifest): return combination in combination_names =20 =20 -def checkout(combination, verbose=3DFalse, override=3DFalse, log=3DNone): +def checkout(combination, verbose=3DFalse, override=3DFalse, log=3DNone, c= ache_obj=3DNone): workspace_path =3D get_workspace_path() manifest =3D get_workspace_manifest() =20 @@ -493,7 +507,10 @@ def checkout(combination, verbose=3DFalse, override=3D= False, log=3DNone): # Return to the initial combo, since there was an issue with cheki= ng out the selected combo checkout_repos(verbose, override, initial_repo_sources, workspace_= path, manifest) finally: - maintain_submodules(workspace_path, manifest, submodule_combo, ver= bose) + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_REPO_N= AME) + maintain_submodules(workspace_path, manifest, submodule_combo,=20 + verbose, cache_path) if sparse_enabled or sparse_diff: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, current_repos, manifest) diff = --git a/project_utils/submodule.py b/project_utils/submodule.py index 3d1b6= 20..f735125 100644 --- a/project_utils/submodule.py +++ b/project_utils/submodule.py @@ -61,7 +61,7 @@ def _deinit(repo, submodules=3DNone, verbose=3DFalse): return =20 =20 -def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse): +def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse, c= ache_path=3DNone): """ Performs the update of submodules. This includes the sync and update = operations. =20 @@ -82,6 +82,8 @@ def _update(repo, submodules=3DNone, verbose=3DFalse, rec= ursive=3DFalse): cmd =3D ['git', 'submodule', 'update', '--init'] if recursive: cmd.append('--recursive') + if cache_path is not None: + cmd.extend(['--reference', cache_path]) output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) display_git_output(output_data, verbose) else: @@ -99,6 +101,8 @@ def _update(repo, submodules=3DNone, verbose=3DFalse, re= cursive=3DFalse): cmd =3D ['git', 'submodule', 'update', '--init'] if sub.recursive: cmd.append('--recursive') + if cache_path is not None: + cmd.extend(['--reference', cache_path]) cmd.extend(['--', sub.path]) output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) display_git_output(output_data, verbose) @@ -269,7 +273,7 @@ d= ef deinit_submodules(workspace, start_manifest, start_combo, _deinit(repo, deinit_list, verbose) =20 =20 -def maintain_submodules(workspace, manifest, combo_name, verbose=3DFalse): +def maintain_submodules(workspace, manifest, combo_name, verbose=3DFalse, = cache_path=3DNone): """ Updates the submodules for a specific repo. =20 @@ -277,6 +281,7 @@ def maintain_submodules(workspace, manifest, combo_name= , verbose=3DFalse): manifest - The manifest parser object for the project. combo_name - The combination name to use for submodule maintenance= . verbose - Enable verbose messages. + cache_path - Path to the submodule cache repo. A value of None in= dicates that no cache repo exists. """ # Process each repo that may have submodules enabled print(strings.SUBMOD_INIT_UPDATE) @@ -303,9 +308,9 @@ def maintain_submodules(workspace, manifest, combo_name= , verbose=3DFalse): =20 # Perform sync/update if len(repo_subs) =3D=3D 0: - _update(repo, None, verbose) + _update(repo, None, verbose, cache_path=3Dcache_path) else: - _update(repo, repo_subs, verbose) + _update(repo, repo_subs, verbose, cache_path=3Dcache_path) =20 =20 if __name__ =3D=3D '__main__': -- 2.21.0.windows.1