From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx.groups.io with SMTP id smtpd.web12.2385.1589910379917348858 for ; Tue, 19 May 2020 10:46:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=wSnPAZ0x; spf=pass (domain: intel.com, ip: 192.55.52.88, mailfrom: erik.c.bjorge@intel.com) IronPort-SDR: +tQamgLQrbb8QW+dX43CToLGOgBstrM7MYW2CnqkJ8vqaw0xOuZLZtNwZf/KDj/b+n+Ha7HuVG Ug76MjHuId/g== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 May 2020 10:46:19 -0700 IronPort-SDR: +elhco8KB4iNZSIn2Y4devshb93OY32axSVbnMm3OxfHpmaHciMSczUdqeWu+i46rsnXxwN7sg lHTbP4+VeazA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,410,1583222400"; d="scan'208";a="289041287" Received: from orsmsx104.amr.corp.intel.com ([10.22.225.131]) by fmsmga004.fm.intel.com with ESMTP; 19 May 2020 10:46:19 -0700 Received: from ORSEDG001.ED.cps.intel.com (10.7.248.4) by ORSMSX104.amr.corp.intel.com (10.22.225.131) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 19 May 2020 10:46:18 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.44) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 19 May 2020 10:46:18 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WUFvWehzy7zRM4cBk8lBq1l4NhEhgKWJPwFd652rkI75Oq0PJhBKLBi15lXmjNxNHUXLkqsf6lFK+8clp+dOIwJZsNxAZQpGg3zoI8XJZYMoh1EJZDnebSwC3wyvfVQPeDAQLDcKsjE2iDyzxzjz9RHrY+da6sUUk4RuuYflWKoZdR5Cya2SD+8IeeFe2mkFmhif1J1aReoLAamvjimk1pJBb1tBy7H/IOhCn8/juyRgOQrncU3icQmY9a2W4Z7WJ7EHeAHQGUYri3qDO3GTSMUx/AUdh7eJk8kypFJjoYLxx78WK9IqVr6Xkl+eJCmv3XJuirNA0d7OnHyU5rRCEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2qV2dHGPhC5VmuKfc6cIOzJ0QLoOAKqlqaKnWXCe0w=; b=JgWnR8aiJTe35cqI45HI+35o5H970u7kd8fGILlEh4pZ4x/oF/DnSkDIgc4WcblFGVHy/eZyfXsm3VYT+LPwSkD+fAKu/6CkLC8N+dmSF0ulh9h+F8iVobqoSe5VP8vId43MHC3zzHk9y2XjQFwqhtgxD05Kot8/UD2NVRsMwSprgtEboQIJMijhP2n0bgvvZRMblEJdXKkOfNExXFMtJnO0t/SO9MUZh96DhlS3R3nzeFgrmAdp5cG7Q+KLpi4g0WSQYoQMv5YOKfsBYzS5nxSd2uXG+fa0wPwJ0j76S2UZ/7bqbPYcSi+NVYpmO5XidsO4K7NX4v1aP7E9spKjYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2qV2dHGPhC5VmuKfc6cIOzJ0QLoOAKqlqaKnWXCe0w=; b=wSnPAZ0xFK8z7MzD2uHOKm5qNz4U6+0Xb9+WqGYedE+bkJmgAZNywP1QiEGA4h4KAsZ0UhF+C84s4tlDpZu9oGK/jYnUI0qCs+USEqHeg195ufP9jIh5oICt+AnfKaNNnFhE3Q/FZgGSO4Ja4sj/J5B6riI6fhB9wTgOJetPzsI= Received: from MW3PR11MB4554.namprd11.prod.outlook.com (2603:10b6:303:5d::7) by MW3PR11MB4682.namprd11.prod.outlook.com (2603:10b6:303:2e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23; Tue, 19 May 2020 17:46:17 +0000 Received: from MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281]) by MW3PR11MB4554.namprd11.prod.outlook.com ([fe80::c569:9903:4c10:9281%6]) with mapi id 15.20.3000.033; Tue, 19 May 2020 17:46:17 +0000 From: "Bjorge, Erik C" To: "Desimone, Ashley E" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v1 1/2] EdkRepo: Adding selective submodule init script Thread-Topic: [edk2-staging/EdkRepo] [PATCH v1 1/2] EdkRepo: Adding selective submodule init script Thread-Index: AQHWK+qxnzhbe9wYEUyATtPXUVVrDKiutdHAgAD7auA= Date: Tue, 19 May 2020 17:46:17 +0000 Message-ID: References: <0b6d0db81fbb9309a1644ea34eeb4ccf68b19b3b.1589678697.git.erik.c.bjorge@intel.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [134.134.136.217] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ca118dca-4342-4795-7898-08d7fc1c88b4 x-ms-traffictypediagnostic: MW3PR11MB4682: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 040866B734 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: OfwIHNAU6c5NDcY2lkChrvs22iGvcIXLqCNtzl9lj3UoXzUo3GpRjIRXTpqYhlIGQnWd+G9xg3Y855C0TyItJtPoPwAwtSKe2jHA2XVATyAKCMAUm0qpZJw11qYfmjOBS2lz0wBJJolXVxeQQDy5Hnntwv0xiAGU9cBnAyOJvjLFjzk+H6XmesF+p2K2lGObZPa//aYqN+wN71n7NSmsSwIqkTRRaQj+MUkofdULxoz/BRQujV9jTnV3ILIEXMI5mIvDNxBxD6iFaeIKamcTMzLXwzRl1hBhHHenoa02Q5lUtDfxLEWvK0q/ioKd1KyToExaKe72OShHcLz3/wMDsg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW3PR11MB4554.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(366004)(346002)(39860400002)(136003)(376002)(396003)(66476007)(110136005)(64756008)(66446008)(66556008)(4326008)(7696005)(107886003)(316002)(54906003)(55016002)(9686003)(71200400001)(2906002)(478600001)(86362001)(5660300002)(33656002)(186003)(66946007)(8676002)(6506007)(30864003)(76116006)(53546011)(26005)(52536014)(8936002)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: vTZgK90QYkNM2+T1Rf6IqSkrzIOdsCPpHY21cxBkkvXxTVqlfbQCOeVtrMjf9RWd7bTI6AHVq4OGM2eAIatwm0/NQK2lMoUimPgSA5SsX1WToAKlLzXY8qcElZbdPFzubf47t6E8oKobxRsVDXgCdUw+8AJ536jrZ2128GWFdt87Pr7QnXXmqHrXUm3qdJmjEqRVKr54VSud6xAG+VJD5irVVR1+iufZyBiCE38mghaw5UY8Gfq0r+pN4HnOAIzfRJPpavqBQqDBpCRdRscDRFu6mk+OzUMl8/D8HGHmeInxzLrSqeXvBY6QZIs/GwGKDDGvhfmavuYERJEAU3uT4mQ49cLMiLQ11HJs7iMLG05j8e9oteWY2qhVCuWRI3okIppHphhHJwlS5K9M0iuqf+q3H2+rYcKEDp6Xnc+/G6MdKQEPGGQ17ak9njvBKsA1DxjHOdxx3T+22gr+ylqqfoHcT2GYNtZIYRErbsgmfCHOvXF1aXdUVjIcZMwLi8Op MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: ca118dca-4342-4795-7898-08d7fc1c88b4 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2020 17:46:17.7623 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: RN9BlJkddNnhyBrCY5xPLhJi+T9b35Osp1sfBS4Yd/kn/1h9KgSUlKyeBHb7c7j5Ext3R75rMzBsADWkCI6e/A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4682 Return-Path: erik.c.bjorge@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Good feedback. - I will move _display_git_output to edkrepo.common.common_repo_functions a= nd remove leading '_' from the name - I will clean up the comments in init_submodules - I will rename init_submodules to maintain_submodules to remove the confus= ion I will submit a new patch set today. Thanks, -Erik -----Original Message----- From: Desimone, Ashley E =20 Sent: Monday, May 18, 2020 7:51 PM To: Bjorge, Erik C ; devel@edk2.groups.io Cc: Desimone, Nathaniel L ; Pandya, Puja ; Bret Barkelew ; Agyeman= , Prince Subject: RE: [edk2-staging/EdkRepo] [PATCH v1 1/2] EdkRepo: Adding selectiv= e submodule init script Hi Erik, Please see feedback inline below. Thanks, Ashley -----Original Message----- From: Bjorge, Erik C Sent: Saturday, May 16, 2020 6:29 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v1 1/2] EdkRepo: Adding selective su= bmodule init script Adds selective submodule support functions and command line scripting suppo= rt. The existing support will be ported over to use this functionality. Signed-off-by: Erik Bjorge Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge --- project_utils/arguments/__init__.py | 8 + project_utils/arguments/submodule_args.py | 24 ++ project_utils/project_utils_strings.py | 24 ++ project_utils/submodule.py | 389 ++++++++++++++++++++++ setup.py | 50 +-- 5 files changed, 470 insertions(+), 25 deletions(-) create mode 100644 pr= oject_utils/arguments/__init__.py create mode 100644 project_utils/arguments/submodule_args.py create mode 100644 project_utils/project_utils_strings.py create mode 100644 project_utils/submodule.py diff --git a/project_utils/arguments/__init__.py b/project_utils/arguments/= __init__.py new file mode 100644 index 0000000..0486261 --- /dev/null +++ b/project_utils/arguments/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +# +## @file +# __init__.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
# +SPDX-License-Identifier: BSD-2-Clause-Patent # diff --git a/project_utils/arguments/submodule_args.py b/project_utils/argu= ments/submodule_args.py new file mode 100644 index 0000000..804ce98 --- /dev/null +++ b/project_utils/arguments/submodule_args.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
# +SPDX-License-Identifier: BSD-2-Clause-Patent # + +""" +Strings for command line arguments. +""" + +SUBMOD_MANIFEST_HELP =3D 'The current manifest file.' +SUBMOD_COMBO_HELP =3D 'The current branch combination in use. If a combo i= s not specified, the current ' \ + 'combo will be used.' +SUBMOD_NEW_MANIFEST_HELP =3D 'The new manifest file. Used to test manifes= t upgrade paths.' +SUBMOD_NEW_COMBO_HELP =3D 'The new branch combination to use. Used to tes= t switching combos.' +SUBMOD_DEINIT_HELP =3D 'Performs submodule deinitialization before initial= izing submodules.' +SUBMOD_INIT_FULL_HELP =3D 'Initialize all submodules (recursive).' +SUBMOD_DEINIT_FULL_HELP =3D 'Deinitialize all submodules.' +SUBMOD_WORKSPACE_HELP =3D 'The project workspace root. If not specified t= he current working directory will ' \ + 'be used.' +SUBMOD_VERBOSE_HELP =3D 'Enables verbose messaging.' diff --git a/project_utils/project_utils_strings.py b/project_utils/project= _utils_strings.py new file mode 100644 index 0000000..33c22d2 --- /dev/null +++ b/project_utils/project_utils_strings.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# humble.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
# +SPDX-License-Identifier: BSD-2-Clause-Patent # + +""" +Contains strings used by the project_utils modules. +""" +# Status update messages +SUBMOD_DEINIT =3D 'Deinitializing submodules' +SUBMOD_DEINIT_FULL =3D 'Deinitializing all submodules' +SUBMOD_INIT_UPDATE =3D 'Initializing/Updating submodules' +SUBMOD_INIT_FULL =3D 'Initializing/Updating all submodules (recursive)' + +# Verbose messages +SUBMOD_INIT_PATH =3D 'Submodule init: {}' +SUBMOD_DEINIT_PATH =3D 'Submodule deinit: {}' +SUBMOD_SYNC_PATH =3D 'Submodule sync: {}' +SUBMOD_UPDATE_PATH =3D 'Submodule update: {}' +SUBMOD_EXCEPTION =3D '- Exception: {}' diff --git a/project_utils/submodule.py b/project_utils/submodule.py new fi= le mode 100644 index 0000000..6534f2c --- /dev/null +++ b/project_utils/submodule.py @@ -0,0 +1,389 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
# +SPDX-License-Identifier: BSD-2-Clause-Patent # import argparse import=20 +git import os import sys import traceback + +from edkrepo_manifest_parser.edk_manifest import ManifestXml import=20 +project_utils.project_utils_strings as strings import=20 +project_utils.arguments.submodule_args as arguments + + +def _init(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs submodule initialization. Note that if no selective submodul= e initialization + is enabled the init will be handled in _update. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + # Only handle selective initialization with this method. For fully re= cursive init + # it is cleaner to do it as part of the update process. + if submodules is not None: + for sub in submodules: + if verbose: + print(strings.SUBMOD_INIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'init', = '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + + +def _deinit(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs deinitialization of submodules. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + if submodules is None: + output_data =3D repo.git.execute(['git', 'submodule', 'deinit', '-= -all'], + with_extended_output=3DTrue, with_s= tdout=3DTrue) + _display_git_outupt(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_DEINIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'deinit'= , '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + + +def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse): + """ + Performs the update of submodules. This includes the sync and update = operations. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + recursive - If submodules is None then use this parameter to determin= e if initialization + should be recursive + """ + # Now perform the update of the submodules. For a fully recursive sub= module init + # this code will update and initialize at the same time. + if submodules is None: + cmd =3D ['git', 'submodule', 'sync'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + cmd =3D ['git', 'submodule', 'update', '--init'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_SYNC_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'sync'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + if verbose: + print(strings.SUBMOD_UPDATE_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'update', '--init'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + This is a common pattern used for parsing git output throughout EdkRepo I w= ould like to see it placed somewhere where it can be consumed without impor= ting all of the submodule content as well.=20 +def _display_git_outupt(output_data, verbose=3DFalse): + """ + Displays output from GitPython git commands + + output_data - Output from the git.execute method + verbose - Enable verbose messages + """ + if verbose and output_data[0]: + print(output_data[0]) + if output_data[1]: + print(output_data[1]) + if verbose and output_data[2]: + print(output_data[2]) + + +def _compute_change(current_subs, new_subs): + """ + Determines the list of submodules that have been removed. Also needs = to + determine if any submodules need to have recursive init disabled + + current_subs - List of selective submodule init entries for the curren= t combo + new_subs - List of selective submodule init entries for the new co= mbo + """ + # Create data objects for determining what submodules need to be deini= tialized + tmp_current =3D {x.path: x for x in current_subs} + tmp_new =3D {x.path: x for x in new_subs} + + # Initial deinitialization list + deinit_paths =3D list(set(tmp_current).difference(set(tmp_new))) + + # Check for change in recursive initialization for the specific submod= ules + for path in tmp_new: + if path in tmp_new and path in tmp_current: + if tmp_current[path].recursive and not tmp_new[path].recursive= : + deinit_paths.append(path) + + # Create the final list of submodules that need to be deinitialized + return [x for x in current_subs if x.path in deinit_paths] + + +def _get_submodule_enable(manifest, remote_name, combo): + """ + Determines if submodules are enabled for the current repo and combo + + manifest - Manifest object + remote_name - The name of the current remote being processed + combo - The current combo name being processed + """ + repo_sources =3D manifest.get_repo_sources(combo) + for source in repo_sources: + if source.remote_name =3D=3D remote_name: + return source.enable_submodule + return False + + +def _get_submodule_state(remote_name, start_manifest, start_combo, end_man= ifest=3DNone, end_combo=3DNone): + """ + Determines the state of submodules across manifest and combo changes. + + remote_name - The name of the current remote being processed. + start_manifest - Initial manifest parser object. + start_combo - Initial combo. + end_manifest - The manifest parser object for the project if the= manifest is being updated. + end_combo - The combination name at the end of the operation = if being modified. + """ + start_subs =3D start_manifest.get_submodule_init_paths(remote_name, st= art_combo) + start_subs_enabled =3D _get_submodule_enable(start_manifest, remote_na= me, start_combo) + if end_combo is not None: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , end_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, end_combo) + else: + end_subs =3D start_manifest.get_submodule_init_paths(remote_na= me, end_combo) + end_subs_enabled =3D _get_submodule_enable(start_manifest, rem= ote_name, end_combo) + else: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , start_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, start_combo) + else: + end_subs =3D start_subs + end_subs_enabled =3D start_subs_enabled + return start_subs, start_subs_enabled, end_subs, end_subs_enabled + + +def deinit_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule deinit based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_DEINIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _deinit(repo, None, verbose) + + +def init_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule init based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_INIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _update(repo, None, verbose, True) + + +def deinit_submodules(workspace, start_manifest, start_combo, + end_manifest=3DNone, end_combo=3DNone, + verbose=3DFalse): + """ + Deinitializes the submodules for a project. + + workspace - Path to the current workspace. + start_manifest - The manifest parser object for the project. + start_combo - The combination name at the start of the operatio= n. + end_manifest - The manifest parser object for the project if the= manifest is being updated. If the + manifest file is not being updated us the value N= one. + end_combo - The combination name at the end of the operation.= If the combo will not change + use the value None. + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_DEINIT) + repo_sources =3D start_manifest.get_repo_sources(start_combo) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + start_subs, start_subs_enabled, end_subs, end_subs_enabled =3D _ge= t_submodule_state(source.remote_name, + = start_manifest, + = start_combo, + = end_manifest, + = end_combo) + if not start_subs_enabled and not end_subs_enabled: + # At this point submodules are not enabled on this repo + continue + + # Compute the list of submodules that need to be removed and added + deinit_list =3D _compute_change(start_subs, end_subs) + + # Deinitialize submodules + if (start_subs_enabled and not end_subs_enabled) or (len(start_sub= s) =3D=3D 0 and len(end_subs) !=3D 0): + # Submodules are being disabled for the entire repo so do a + # full deinit on the repo. + _deinit(repo, None, verbose) + else: + # Do the deinit based on the list + _deinit(repo, deinit_list, verbose) + + The comments mention needing to use the start combo/end combo parameters bu= t I only see one combo_name parameter in the definition of this function. A= dditionally init_submodules implies that this will initialize them not upd= ate them as described in the comment. +def init_submodules(workspace, manifest, combo_name, verbose=3DFalse): + """ + Updates the submodules for a specific repo. Note that operations like= checkout will need to use + the start_combo and end_combo parameter. Operations like clone will j= ust need to use the + start_combo parameter. + + workspace - Path to the current workspace. + manifest - The manifest parser object for the project. + combo_name - The combination name at the start of the operation. + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_INIT_UPDATE) + repo_sources =3D manifest.get_repo_sources(combo_name) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + repo_subs =3D manifest.get_submodule_init_paths(source.remote_name= , combo_name) + repo_subs_enabled =3D _get_submodule_enable(manifest, source.remot= e_name, combo_name) + if not repo_subs_enabled: + continue + + # Initialize submodules + if len(repo_subs) > 0: + _init(repo, repo_subs, verbose) + + # Perform sync/update + if len(repo_subs) =3D=3D 0: + _update(repo, None, verbose) + else: + _update(repo, repo_subs, verbose) + + +if __name__ =3D=3D '__main__': + def parse_args(): + parser =3D argparse.ArgumentParser() + parser.add_argument('manifest_file', metavar=3D'MANIFEST', help=3D= arguments.SUBMOD_MANIFEST_HELP) + parser.add_argument('--combo', default=3DNone, help=3Darguments.SU= BMOD_COMBO_HELP) + parser.add_argument('--new-manifest', default=3DNone, help=3Dargum= ents.SUBMOD_NEW_MANIFEST_HELP) + parser.add_argument('--new-combo', default=3DNone, help=3Dargument= s.SUBMOD_NEW_COMBO_HELP) + parser.add_argument('--deinit', action=3D'store_true', help=3Dargu= ments.SUBMOD_DEINIT_HELP) + parser.add_argument('--init-full', action=3D'store_true', help=3Da= rguments.SUBMOD_INIT_FULL_HELP) + parser.add_argument('--deinit-full', action=3D'store_true', help= =3Darguments.SUBMOD_DEINIT_FULL_HELP) + parser.add_argument('--workspace', default=3D'.', help=3Darguments= .SUBMOD_WORKSPACE_HELP) + parser.add_argument('--verbose', action=3D'store_true', help=3Darg= uments.SUBMOD_VERBOSE_HELP) + return parser.parse_args() + + def main(args): + # Optional data init + new_manifest =3D None + all_new_combos =3D None + + # Extract basic manifest data + manifest =3D ManifestXml(args.manifest_file) + init_manifest =3D manifest + all_combos =3D manifest.combinations + all_combos.extend(manifest.archived_combinations) + if args.new_manifest is not None: + new_manifest =3D ManifestXml(args.new_manifest) + all_new_combos =3D new_manifest.combinations + all_new_combos.extend(new_manifest.archived_combinations) + init_manifest =3D new_manifest + + # Determine current and new combo information + current_combo =3D manifest.general_config.current_combo + new_combo =3D args.new_combo + for combo in all_combos: + if args.combo is not None: + if args.combo.lower() =3D=3D combo.name.lower(): + current_combo =3D combo.name + if args.new_combo is not None: + if args.new_combo.lower() =3D=3D combo.name.lower(): + new_combo =3D combo.name + init_combo =3D current_combo + if new_combo is not None: + init_combo =3D new_combo + + if args.deinit_full: + deinit_full(args.workspace, manifest, args.verbose) + elif args.init_full: + init_full(args.workspace, manifest, args.verbose) + else: + if args.deinit: + deinit_submodules(args.workspace, manifest, current_combo, + new_manifest, new_combo, args.verbose) + init_submodules(args.workspace, init_manifest, init_combo, + args.verbose) + + return 0 + + try: + args =3D parse_args() + sys.exit(main(args)) + except Exception: + if args.verbose: + traceback.print_exc() + sys.exit(1) diff --git a/setup.py b/setup.py index 5d2f55a..a855000 100755 --- a/setup.py +++ b/setup.py @@ -1,27 +1,27 @@ -#!/usr/bin/env python3 -# -## @file setup.py -# -# Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.
-= # SPDX-License-Identifier: BSD-2-Clause-Patent -# - -from setuptools import setup - -setup(name=3D'edkrepo', - version=3D'2.0.0', - description=3D'The edkrepo tools', - packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.commands.humble', +#!/usr/bin/env python3 +# +## @file setup.py +# +# Copyright (c) 2017 - 2020, Intel Corporation. All rights=20 +reserved.
# SPDX-License-Identifier: BSD-2-Clause-Patent # + +from setuptools import setup + +setup(name=3D'edkrepo', + version=3D'2.0.0', + description=3D'The edkrepo tools', + packages=3D['edkrepo', 'edkrepo.commands',=20 +'edkrepo.commands.arguments', 'edkrepo.commands.humble', 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.commo= n.workspace_maintenance', 'edkrepo.common.workspace_maintenance.humble', 'edkrepo.co= nfig', 'edkrepo.config.humble', - 'edkrepo_manifest_parser', 'project_utils'], - package_data=3D{ - }, - include_package_data=3DTrue, - entry_points=3D{ - 'console_scripts': [ - 'edkrepo =3D edkrepo.edkrepo_entry_point:main', - 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' - ] - } - ) + 'edkrepo_manifest_parser', 'project_utils', 'project_utils= .arguments'], + package_data=3D{ + }, + include_package_data=3DTrue, + entry_points=3D{ + 'console_scripts': [ + 'edkrepo =3D edkrepo.edkrepo_entry_point:main', + 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' + ] + } + ) -- 2.21.0.windows.1