From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga05.intel.com (mga05.intel.com []) by mx.groups.io with SMTP id smtpd.web08.1760.1607022743175244184 for ; Thu, 03 Dec 2020 11:12:24 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=r95wFIgc; spf=fail (domain: intel.com, ip: , mailfrom: ashley.e.desimone@intel.com) IronPort-SDR: 3CdIr/Neqqe+/wa7K+voqIzmA6+q7DSPyCaWHVm01uqOC3t9WaFweOxJBBlmQmru8vAwwWEuvb r+CZh3BT5X8g== X-IronPort-AV: E=McAfee;i="6000,8403,9824"; a="257967356" X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="257967356" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2020 11:12:23 -0800 IronPort-SDR: EByyFJ+u6PZ9ckTsZjBwIk/4PAaC5r+kri74w3m7cXxfJRI3NL27wMKSeTeQ9a5KGY9pGY1cIa ax3mMgHAOKhA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,390,1599548400"; d="scan'208";a="550629380" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by orsmga005.jf.intel.com with ESMTP; 03 Dec 2020 11:12:23 -0800 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 11:12:22 -0800 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 3 Dec 2020 11:12:21 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 3 Dec 2020 11:12:21 -0800 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.100) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Thu, 3 Dec 2020 11:11:46 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ly1NTl4j6kSEvP/FKnLkMRCh7MEmD3Hi3vVoiczxlEDl9TN/2Kx8XibMJDABr4Lrh9Tkq0Q1TWTzix0hR7OK0i97jNdk6Lj7xI4JjxN+t4irq9ghrG/8Xb9EAwiM/HA739HznjQLP+ajMss2mO4cr4GRB49yFP10mbXXStQL57TGKoWULGEMrHnDoeB+JoanvBP84QVPSzNeeyD64fSowtoW9LxGhBpGxXcKFRz3ZJZg9MBAgHnmeyw7MF9WZHhFc9Vb2zTCiOlr7qfm+mJqYEwl2KFtlyHk5k5pNcEWCzLUcnwyCxxdm8rPXN3fcXxeKMtH/pTudujg2UZWQVursQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Obg6exU78D3LpfS3EmPGoLtnioevkwS1KIyfcYIkABc=; b=RDY9VQGQgYiM4OgM3AwwL+SxvQaODySwwqQ7uz816KPN3ky6AxxVkDKRAjRCbFKLAmdpIKIxmhrW1r5geibdiPT2r9Tuubateq4jTq/QvrzM6wi2P61hi4XtBF2PMEyZrejFI0DUdnA4c3/Jm3IPi97B/ukH15RE/WdMEErYUxGtcaBAgmQ57QflhQbJQosJQcLN3uQQTbz4NPOX0K/g8uhgEWSYRFJV8P2E3fOZ9fGY+r9RIfHV2hU5OyLjxywHTLGKW8EWknhNCMekJ4PkfLj4GvinpeYl+ZRXM2IBeBLNpW8jwKKla7R78UtlzfUdQVX2//6i+HfDpkYl2B4NJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Obg6exU78D3LpfS3EmPGoLtnioevkwS1KIyfcYIkABc=; b=r95wFIgcO0cDLfKf9oq4UsrVs2gKAbx9wMY2AJFK4HRekSNT/4H10xlAlyS5vMtApttQIdBt8C+j35vnqn3fdeFTKwc5q5oWJtiel4m1c8vhdareYviW4+pg9Xefy3xfh+CThcUhu/RMCDK5eyllYaSssUehSjrhvHpnRo+M+Ks= Received: from BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) by SJ0PR11MB5134.namprd11.prod.outlook.com (2603:10b6:a03:2de::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.21; Thu, 3 Dec 2020 19:11:38 +0000 Received: from BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5]) by BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::506:7bd4:a0a0:b3c5%6]) with mapi id 15.20.3611.034; Thu, 3 Dec 2020 19:11:38 +0000 From: "Ashley E Desimone" To: "Bjorge, Erik C" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo cache support. Thread-Topic: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo cache support. Thread-Index: AQHWvJ0m7C8nCn72hkucu4vaWYeZFanl1utw Date: Thu, 3 Dec 2020 19:11:38 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [50.53.190.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7bdf59c5-375f-431a-d4c9-08d897bf42c8 x-ms-traffictypediagnostic: SJ0PR11MB5134: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8kS7m/dTdFnd084M04jhHSvHW8n47nPHaQcGIMC/5yTgGIu82EPy8k9iUgpV3UyYuTl8yp1LEkGqbc+huMSk7+UtgSknBkKEV+KMyXdhKKRzNr+e3LReVJtorWSM4frEDC+IVmo4C2pkrpS09OB+V6ldHBnHU9sxlQxI5zPKwDRPMPmY30ZzXBP7G5cVPBstS5VtwnylDYEhAWWZh283e03rUagI5mOvG44uKT9hUBw1SzLrdytoa+Y+t6W431bZK/6MwJqFm4mI4XqFWX3ELl0zWAizdJZSqbFmcXX+besAPyYRhqty0M/5LqWWdOm47pOx5zV7XDfrgzXW9e0U7w== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3973.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(136003)(366004)(346002)(396003)(376002)(39850400004)(8936002)(7696005)(478600001)(71200400001)(9686003)(4326008)(52536014)(53546011)(30864003)(86362001)(5660300002)(66446008)(6506007)(66556008)(2906002)(8676002)(186003)(66946007)(64756008)(66476007)(26005)(110136005)(55016002)(33656002)(83380400001)(76116006)(54906003)(316002)(107886003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?dOe2aQMwlxsJA2/AqNbHJCGwwj7mQi+FHt0gVdBf5jD+rzpqDHkbQnTO9lTE?= =?us-ascii?Q?gkiL0eb7DDFBS4bT75hGqd3qsnVaoJi1MSLK+qXlV47ZTng7AoEJhtW1X3oV?= =?us-ascii?Q?Uelm0MwmvB44R6Ceq8Jw2K5MEjGJwZVSXgWee0uf4JS/cZ/ULduavms0Ffq7?= =?us-ascii?Q?G+iXuTF6/sbrqmpoQJ7aBIaMHcZxyq82uEBiJrwtkL9zEZ1LSmLRDIPCuPJ6?= =?us-ascii?Q?JBSTdDymVduxe2XGmUz5oVeErfzl4uGB1/KXT2rbOnlCiGG2Xvy4iAvfHr5N?= =?us-ascii?Q?EIsKC1ueumWc4ZiX+enKweP3x8TTG37u67kAb46a+bCqpv06BvyUKXFMbEyc?= =?us-ascii?Q?a5QylinOPritu0uGp214zWcMEv5g4UFJ5WU75qjO7Px3spi1L631kvueokt4?= =?us-ascii?Q?Kl8BmWDeKEll98zoWZlEI4yxakTdR/7iUPipTL6yg3/aB6lxyNFtGpPjuwO4?= =?us-ascii?Q?BS/zn8FolObVbw4IVdNU+v+wEXCoOBCwT5ctt7SKx8X8W7H1+hAQOpvNoP6J?= =?us-ascii?Q?LK7nyF7yvHxEoQvvrGnutOJVBJHWxFzQv1GaRM6MjHSnIw2Gfys3t2jw2RPv?= =?us-ascii?Q?UCwt9k+4oa6s5BQHG3y5jv1b5E3IaHt9ozbEbxWkBHfH3dPUjs/CNEdWSlRF?= =?us-ascii?Q?KG3Exa/pUqsk9fpKgDnfxFACpSq9JNtCdy9/1Wgg/hDXIREZ2MUc7j+Ws9Ny?= =?us-ascii?Q?5EG1KbqEuMbYAt8CZaVI02cFlArg19lfSGn7y/7kHSL0OjXstqumAMHS0ft8?= =?us-ascii?Q?LQN8C9hUyC3R2Rs4Eeq4HS6YudGwtjrkKXZtAThxRkxrXSZUUYDBqj+oAR1X?= =?us-ascii?Q?/EUXmD4YwkL5l2qkX35tPe73GNhdiuXOeJ6o5oAd8PTSlaXtAPVckmDNfQcJ?= =?us-ascii?Q?zBEWWjOTa4x8N5tfApOQaF6iNqei355+wXX5MAYstJUjcOAvigST1t6LMDt6?= =?us-ascii?Q?Et8wrVqkY0KKecnqT+aF6LquRrn/zFM9QljSwYJLKS8=3D?= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR11MB3973.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7bdf59c5-375f-431a-d4c9-08d897bf42c8 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2020 19:11:38.6351 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Fm2YBOsCq94iB69GU4O9buRHaqzPB8oVs2n7ZkbDh0ug47vFdlhXp9LBAn14FuhLX9KZ21CxWhv1hO/k3s8eLj4QZpP5WfBYLblHsibToCw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5134 Return-Path: ashley.e.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Pushed as: bc55138c51548d80db4416d91e55b2c35e48ff4e -----Original Message----- From: Erik Bjorge =20 Sent: Monday, November 16, 2020 8:50 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v2 2/2] EdkRepo: Enable use of repo = cache support. This changes enables the local repo cache to be used when cloning and synci= ng changes. The repo cache applies to submodules as well. Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge Signed-off-by: Erik Bjorge --- edkrepo/commands/checkout_command.py | 3 ++- edkrepo/commands/checkout_pin_command.py | 8 +++++++- edkrepo/commands/clone_command.py | 15 ++++++++++++-- edkrepo/commands/sync_command.py | 12 +++++++++--- edkrepo/common/common_repo_functions.py | 25 ++++++++++++++++++++---- project_utils/submodule.py | 13 ++++++++---- 6 files changed, 61 insertions(+), 15 deletions(-) diff --git a/edkrepo/commands/checkout_command.py b/edkrepo/commands/checko= ut_command.py index 0169f30..2ce26c0 100644 --- a/edkrepo/commands/checkout_command.py +++ b/edkrepo/commands/checkout_command.py @@ -16,6 +16,7 @@ import os from edkrepo.commands.edkrepo_command import EdkrepoCommand, OverrideArgum= ent import edkrepo.commands.arguments.checkout_args as arguments import e= dkrepo.commands.humble.checkout_humble as humble +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import checkout, combination_is_= in_manifest from edkrepo.common.edkrepo_exception import EdkrepoInvalidPar= ametersException from edkrepo.config.config_factory import get_workspace_m= anifest @@ -42,6 +43,6 @@ class CheckoutCommand(EdkrepoCommand): =20 def run_command(self, args, config): if combination_is_in_manifest(args.Combination, get_workspace_mani= fest()): - checkout(args.Combination, args.verbose, args.override) + checkout(args.Combination, args.verbose, args.override,=20 + get_repo_cache_obj(config)) else: raise EdkrepoInvalidParametersException(humble.NO_COMBO.format= (args.Combination)) diff --git a/edkrepo/commands/checkout_pin_command.py b/edkrepo/commands/ch= eckout_pin_command.py index 1c58113..0ad1b48 100644 --- a/edkrepo/commands/checkout_pin_command.py +++ b/edkrepo/commands/checkout_pin_command.py @@ -14,6 +14,7 @@ from git import Repo from edkrepo.commands.edkrepo_command import EdkrepoCommand, OverrideArgum= ent, SourceManifestRepoArgument import edkrepo.commands.arguments.checkout= _pin_args as arguments import edkrepo.commands.humble.checkout_pin_humble = as humble +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import sparse_checkout_enabled, = reset_sparse_checkout, sparse_checkout from edkrepo.common.common_repo_fun= ctions import check_dirty_repos, checkout_repos, combinations_in_manifest = from edkrepo.common.humble import SPARSE_CHECKOUT, SPARSE_RESET, SUBMODULE_= DEINIT_FAILED @@ -21,6 +22,7 @@ from edkrepo.common.edkrepo_exception impor= t EdkrepoInvalidParametersException, from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impor= t list_available_manifest_repos from edkrepo.common.workspace_maintenance.= manifest_repos_maintenance import find_source_manifest_repo from edkrepo.c= onfig.config_factory import get_workspace_path, get_workspace_manifest +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo_manifest_parser.edk_manifest import ManifestXml from project= _utils.submodule import deinit_full, maintain_submodules =20 @@ -82,7 +84,11 @@ class CheckoutPinCommand(EdkrepoCommand): checkout_repos(args.verbose, args.override, pin_repo_sources, = workspace_path, manifest) manifest.write_current_combo(humble.PIN_COMBO.format(args.pinf= ile)) finally: - maintain_submodules(workspace_path, pin, submodule_combo, args= .verbose) + cache_path =3D None + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_path, pin, submodule_combo,=20 + args.verbose, cache_path) if sparse_enabled: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, pin_repo_sources, manifest= ) diff --git a/edkrepo/commands/clone_command.py b/edkrepo/commands/clone_c= ommand.py index 8769102..56c15c9 100644 --- a/edkrepo/commands/clone_command.py +++ b/edkrepo/commands/clone_command.py @@ -14,6 +14,8 @@ import sys from edkrepo.commands.edkrepo_command import EdkrepoCommand from edkrepo.= commands.edkrepo_command import SubmoduleSkipArgument, SourceManifestRepoAr= gument import edkrepo.commands.arguments.clone_args as arguments +from edkrepo.common.common_cache_functions import get_repo_cache_obj=20 +from edkrepo.common.common_cache_functions import=20 +add_missing_cache_repos from edkrepo.common.common_repo_functions import clone_repos, sparse_check= out, verify_single_manifest from edkrepo.common.common_repo_functions impo= rt update_editor_config, combinations_in_manifest from edkrepo.common.comm= on_repo_functions import write_included_config, write_conditional_include @= @ -28,6 +30,7 @@ from edkrepo.common.workspace_maintenance.manifest_repos_m= aintenance import list from edkrepo.common.workspace_maintenance.humble.ma= nifest_repos_maintenance_humble import PROJ_NOT_IN_REPO, SOURCE_MANIFEST_RE= PO_NOT_FOUND from edkrepo_manifest_parser.edk_manifest import CiIndexXml, = ManifestXml from project_utils.submodule import maintain_submodules +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME =20 =20 class CloneCommand(EdkrepoCommand): @@ -151,11 +154,19 @@ class CloneCommand(EdkrepoCommand): # Set up submodule alt url config settings prior to cloning any re= pos submodule_included_configs =3D write_included_config(manifest.remo= tes, manifest.submodule_alternate_remotes, local_manifest_dir) write_conditional_include(workspace_dir, repo_sources_to_clone, su= bmodule_included_configs) - clone_repos(args, workspace_dir, repo_sources_to_clone, project_cl= ient_side_hooks, config, manifest) + + # Determine if caching is going to be used and then clone + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + add_missing_cache_repos(cache_obj, manifest, args.verbose) + clone_repos(args, workspace_dir, repo_sources_to_clone,=20 + project_client_side_hooks, config, manifest, cache_obj) =20 # Init submodules if not args.skip_submodule: - maintain_submodules(workspace_dir, manifest, combo_name, args.= verbose) + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_dir, manifest, combo_name,=20 + args.verbose, cache_path) =20 # Perform a sparse checkout if requested. use_sparse =3D args.sparse diff --git a/edkrepo/commands/sync_command.py b/edkrepo/commands/sync_comma= nd.py index c4ee330..ff48f50 100644 --- a/edkrepo/commands/sync_command.py +++ b/edkrepo/commands/sync_command.py @@ -32,6 +32,7 @@ from edkrepo.common.humble import MIRROR_BEHIND_PRIMARY_R= EPO, SYNC_NEEDS_REBASE, from edkrepo.common.humble import SYNC_BRANCH_CHAN= GE_ON_LOCAL, SYNC_INCOMPATIBLE_COMBO from edkrepo.common.humble import SYN= C_REBASE_CALC_FAIL from edkrepo.common.pathfix import get_actual_path, exp= anduser +from edkrepo.common.common_cache_functions import get_repo_cache_obj from edkrepo.common.common_repo_functions import clone_repos, sparse_check= out_enabled from edkrepo.common.common_repo_functions import reset_sparse_= checkout, sparse_checkout, verify_single_manifest from edkrepo.common.comm= on_repo_functions import checkout_repos, check_dirty_repos @@ -47,6 +48,7 @= @ from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impo= rt list from edkrepo.common.ui_functions import init_color_console from e= dkrepo.config.config_factory import get_workspace_path, get_workspace_manif= est, get_edkrepo_global_data_directory from edkrepo.config.config_factory = import get_workspace_manifest_file +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml = from project_utils.submodule import deinit_submodules, maintain_submodules =20 @@ -102,7 +104,7 @@ class SyncCommand(EdkrepoCommand): if not args.update_local_manifest: self.__check_for_new_manifest(args, config, initial_manifest, = workspace_path, global_manifest_directory) check_dirty_repos(initial_manifest, workspace_path) - + # Determine if sparse checkout needs to be disabled for this opera= tion sparse_settings =3D initial_manifest.sparse_settings sparse_enabled =3D sparse_checkout_enabled(workspace_path, initial= _sources) @@ -116,7 +118,7 @@ class SyncCommand(EdkrepoCommand): reset_sparse_checkout(workspace_path, initial_sources) =20 # Get the latest manifest if requested - if args.update_local_manifest: # NOTE: hyphens in arg name replac= ed with underscores due to argparse + if args.update_local_manifest: # NOTE: hyphens in arg name=20 + replaced with underscores due to argparse self.__update_local_manifest(args, config, initial_manifest, w= orkspace_path, global_manifest_directory) manifest =3D get_workspace_manifest() if args.update_local_manifest: @@ -212,7 +214,11 @@ class SyncCommand(EdkrepoCommand): =20 # Initialize submodules if not args.skip_submodule: - maintain_submodules(workspace_path, manifest, current_combo, a= rgs.verbose) + cache_path =3D None + cache_obj =3D get_repo_cache_obj(config) + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_RE= PO_NAME) + maintain_submodules(workspace_path, manifest,=20 + current_combo, args.verbose, cache_path) =20 # Restore sparse checkout state if sparse_enabled: diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/commo= n_repo_functions.py index 2277c1e..336661a 100644 --- a/edkrepo/common/common_repo_functions.py +++ b/edkrepo/common/common_repo_functions.py @@ -56,6 +56,7 @@ from project_utils.sparse import BuildInfo, process_spars= e_checkout from edkrepo.config.config_factory import get_workspace_path f= rom edkrepo.config.config_factory import get_workspace_manifest from edkre= po.config.tool_config import CI_INDEX_FILE_NAME +from edkrepo.config.tool_config import SUBMODULE_CACHE_REPO_NAME from edkrepo.common.edkrepo_exception import EdkrepoInvalidParametersExcep= tion from edkrepo_manifest_parser.edk_manifest import CiIndexXml, Manifest= Xml from edkrepo.common.edkrepo_exception import EdkrepoNotFoundException,= EdkrepoGitException, EdkrepoWarningException @@ -75,12 +76,25 @@ CLEAR_LIN= E =3D '\x1b[K' DEFAULT_REMOTE_NAME =3D 'origin' PRIMARY_REMOTE_NAME =3D 'primary' =20 -def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, manifest): + +def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, manifest, cache_obj=3DNone): for repo_to_clone in repos_to_clone: local_repo_path =3D os.path.join(workspace_dir, repo_to_clone.root= ) local_repo_url =3D repo_to_clone.remote_url + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(local_repo_url) print("Cloning from: " + str(local_repo_url)) - repo =3D Repo.clone_from(local_repo_url, local_repo_path, progress= =3DGitProgressHandler(), no_checkout=3DTrue) + if cache_path is not None: + print('+ Using cache at {}'.format(cache_path)) + repo =3D Repo.clone_from(local_repo_url, local_repo_path, + progress=3DGitProgressHandler(), + reference_if_able=3Dcache_path, + no_checkout=3DTrue) + else: + repo =3D Repo.clone_from(local_repo_url, local_repo_path, + progress=3DGitProgressHandler(), + no_checkout=3DTrue) # Fetch notes repo.remotes.origin.fetch("refs/notes/*:refs/notes/*") =20 @@ -427,7 +441,7 @@ def combination_is_in_manifest(combination, manifest): return combination in combination_names =20 =20 -def checkout(combination, verbose=3DFalse, override=3DFalse, log=3DNone): +def checkout(combination, verbose=3DFalse, override=3DFalse, log=3DNone, c= ache_obj=3DNone): workspace_path =3D get_workspace_path() manifest =3D get_workspace_manifest() =20 @@ -493,7 +507,10 @@ def checkout(combination, verbose=3DFalse, override=3D= False, log=3DNone): # Return to the initial combo, since there was an issue with cheki= ng out the selected combo checkout_repos(verbose, override, initial_repo_sources, workspace_= path, manifest) finally: - maintain_submodules(workspace_path, manifest, submodule_combo, ver= bose) + cache_path =3D None + if cache_obj is not None: + cache_path =3D cache_obj.get_cache_path(SUBMODULE_CACHE_REPO_N= AME) + maintain_submodules(workspace_path, manifest, submodule_combo,=20 + verbose, cache_path) if sparse_enabled or sparse_diff: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, current_repos, manifest) diff = --git a/project_utils/submodule.py b/project_utils/submodule.py index 3d1b6= 20..f735125 100644 --- a/project_utils/submodule.py +++ b/project_utils/submodule.py @@ -61,7 +61,7 @@ def _deinit(repo, submodules=3DNone, verbose=3DFalse): return =20 =20 -def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse): +def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse, c= ache_path=3DNone): """ Performs the update of submodules. This includes the sync and update = operations. =20 @@ -82,6 +82,8 @@ def _update(repo, submodules=3DNone, verbose=3DFalse, rec= ursive=3DFalse): cmd =3D ['git', 'submodule', 'update', '--init'] if recursive: cmd.append('--recursive') + if cache_path is not None: + cmd.extend(['--reference', cache_path]) output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) display_git_output(output_data, verbose) else: @@ -99,6 +101,8 @@ def _update(repo, submodules=3DNone, verbose=3DFalse, re= cursive=3DFalse): cmd =3D ['git', 'submodule', 'update', '--init'] if sub.recursive: cmd.append('--recursive') + if cache_path is not None: + cmd.extend(['--reference', cache_path]) cmd.extend(['--', sub.path]) output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) display_git_output(output_data, verbose) @@ -269,7 +273,7 @@ d= ef deinit_submodules(workspace, start_manifest, start_combo, _deinit(repo, deinit_list, verbose) =20 =20 -def maintain_submodules(workspace, manifest, combo_name, verbose=3DFalse): +def maintain_submodules(workspace, manifest, combo_name, verbose=3DFalse, = cache_path=3DNone): """ Updates the submodules for a specific repo. =20 @@ -277,6 +281,7 @@ def maintain_submodules(workspace, manifest, combo_name= , verbose=3DFalse): manifest - The manifest parser object for the project. combo_name - The combination name to use for submodule maintenance= . verbose - Enable verbose messages. + cache_path - Path to the submodule cache repo. A value of None in= dicates that no cache repo exists. """ # Process each repo that may have submodules enabled print(strings.SUBMOD_INIT_UPDATE) @@ -303,9 +308,9 @@ def maintain_submodules(workspace, manifest, combo_name= , verbose=3DFalse): =20 # Perform sync/update if len(repo_subs) =3D=3D 0: - _update(repo, None, verbose) + _update(repo, None, verbose, cache_path=3Dcache_path) else: - _update(repo, repo_subs, verbose) + _update(repo, repo_subs, verbose, cache_path=3Dcache_path) =20 =20 if __name__ =3D=3D '__main__': -- 2.21.0.windows.1