From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mx.groups.io with SMTP id smtpd.web12.4413.1590632186693134315 for ; Wed, 27 May 2020 19:16:26 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=qd2kz5PO; spf=pass (domain: intel.com, ip: 192.55.52.120, mailfrom: ashley.e.desimone@intel.com) IronPort-SDR: Gt8x5BjOxOtDIMcO8HRN2zEfXYAZVpHSLlG3kSW6+hxScclFThWdh89QC2I8vFUGQqQxgsdszQ xPaOOp/8Nu4g== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 May 2020 19:16:25 -0700 IronPort-SDR: aKQ/ek7QJI4zs/1iG7HTvtfP0HtDuMLf/bOotquZsrRdOyrvLhNTNxRBBlJVkrJOKo7sHo2woO y9hpNjr4ND9A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,443,1583222400"; d="scan'208";a="468946986" Received: from orsmsx106.amr.corp.intel.com ([10.22.225.133]) by fmsmga005.fm.intel.com with ESMTP; 27 May 2020 19:16:25 -0700 Received: from orsmsx163.amr.corp.intel.com (10.22.240.88) by ORSMSX106.amr.corp.intel.com (10.22.225.133) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 27 May 2020 19:16:24 -0700 Received: from ORSEDG001.ED.cps.intel.com (10.7.248.4) by ORSMSX163.amr.corp.intel.com (10.22.240.88) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 27 May 2020 19:16:24 -0700 Received: from NAM04-SN1-obe.outbound.protection.outlook.com (104.47.44.53) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 27 May 2020 19:16:24 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jAUjqCL5tUpxNG7TxcrrKFYaKn6vdUpaUGACo9LT6AfKywtbbwOB8xQKQDzS4lJKaQZDyveEXDPjn/B07k0xWlAbe55+PekgajvSA2Vg/ioiNOGlirfudyvnby2tylAoUyf3T/uLmA9HDZcTujNdNt6QIM7LF6tQIYbFbfssgQu3yY2ahvQintkRsy0NEAR6UahfWquRXV9ly4leKZBFSkY0DYEAvJOKO6hI+1yt6EFx6xR3PXdInfz4tEQogBWUFyObo7hpsu2fN9w+wbQ66Yz2bCyV79swcGqMvPh4vn1hQ++JBrnhlVrCyZfoXHHF43MviuqEoMQLeas9WCbljw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H2dwMLWCo4vX2qgAtRbzd4M9O+dmW0xN6+M/Pc20+rc=; b=Dlzeg4X92JIS6vB8HvGBdnQKlev5XzOt8p3lY4QUZ5LrqSW+8CBaBkMJnRvvtpUVbnOsPNq0ESsAhzGDdiM78CleRUvw5+NjjQLVWw4oDC+TRBIat2ttt6WolMP806SRNfPrBMKqxQvQ+/ld8tsGluusZ1/rrFmHKHuxeUYN+l4AAlrUKqIhlm3t+Roy5AvtfCGQImd8ky82k1nbNcaPM8iEcEaYgeAj2sBhfiNvhdrMt+wR/FX/RAU9hID6nLdN0ZFSqDGmTU8msXi9GwkuuLDGZ3JwXczB7pI1q0UleJr+Cz/sNYJ2v/bnJkDY94NfzK7IM9dTT46W3IfxhwZQoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H2dwMLWCo4vX2qgAtRbzd4M9O+dmW0xN6+M/Pc20+rc=; b=qd2kz5POzPpuLnD0iJBeDvQyEtSFn8daBIctKVxyW8+eZlaU8pgiSMXHDCTUbd8jdKG38aUNzAPRva+RRZWbTsN32DXsy5gQas1l/2q6B6NBka1lD66XheN6DKfrx9OnUMwPcduEjH/V58tQAtzIs3NR2SFWYYVSVSGvHu+Ltc8= Received: from BY5PR11MB3973.namprd11.prod.outlook.com (2603:10b6:a03:185::29) by BY5PR11MB4436.namprd11.prod.outlook.com (2603:10b6:a03:1c3::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.27; Thu, 28 May 2020 02:16:22 +0000 Received: from BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::296a:eb1c:2b52:2904]) by BY5PR11MB3973.namprd11.prod.outlook.com ([fe80::296a:eb1c:2b52:2904%7]) with mapi id 15.20.3021.030; Thu, 28 May 2020 02:16:22 +0000 From: "Ashley E Desimone" To: "Bjorge, Erik C" , "devel@edk2.groups.io" CC: "Desimone, Nathaniel L" , "Pandya, Puja" , Bret Barkelew , "Agyeman, Prince" Subject: Re: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Adding selective submodule init script Thread-Topic: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Adding selective submodule init script Thread-Index: AQHWLievMB37Y24DiUSfTG/0ryzzm6i8z09g Date: Thu, 28 May 2020 02:16:22 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.2.0.6 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [134.134.136.201] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: aa974933-7ea4-489a-ad79-08d802ad1de6 x-ms-traffictypediagnostic: BY5PR11MB4436: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0417A3FFD2 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 2bxjGsawoLI8iM93uspjOfkJifB9GqKuer8VKVuhy2meUolczpIwkUjcS4/052LDvSAv3Fqpzs12gfvR8XszUfmpiks3iRO3ze+nZbkFtJamy67KazdnE3fzgn5CAtQc6z+LKFRqmakaBGCr/dwg52HSVu2KjAUwZKaSwSY4Tywq81GP4k7dYpwc4nwQEcTn49dclAka382R6WgXMu3KayhVnXO6YXqwb565QzkVxBubHxjxnfFap8J0z739883vDSF0IIW+DbVvN9qmsNkE6d6MUAlAArF9Obwk2LboAEMbaUBtRjcWnaCzJTqh0VOfm0FtT8JInRlENGfzn5V0Bg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR11MB3973.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:(39860400002)(136003)(396003)(366004)(376002)(346002)(110136005)(478600001)(316002)(54906003)(52536014)(33656002)(8676002)(9686003)(8936002)(86362001)(7696005)(107886003)(5660300002)(4326008)(71200400001)(2906002)(186003)(30864003)(55016002)(26005)(64756008)(6506007)(83380400001)(53546011)(66946007)(66446008)(76116006)(66556008)(66476007)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata: b1+XPRJK6rt0sbcNoNLkMZAS34x9ks7FtySXnkSsWPRfmb3JbCj/JZVFSpp+xuzYraAjPA/PjMbECKampSoPXugy+ZibxBTD4C+L7XVRScS30axnrd09jFaMvPQRR50y3NyBoU9LtqhMMBDMrktubUgNjua9AJJ334G1fgqo3z8aeNNBD2gYJivxnoASX7Zy/OBtll6mHFRRmWMjgOx+txyylKRzIR7WAah9G+treTbTIFcy3joJaNMXOb7bNO7oX2+eqx7+xw3bUVB4qf6WqoFu77poJPiAkvTXZDL00fFV697Gr2v37X+ZIAr6cQKjZaESJ7jlZwTzNIzQsktLOdExVOIasWsoV5D79zNIloBwfzrx2xUF0uUciQKVU1DLGQDCmrnDT0PFApg5+koea2c/ekyGiQs3SbtHpPkLT+FNyDNn8y1h38zcMFsTGwaE5KpNm/4WJCLVUGmVLBAGXdPfH6cDlqIbX3WZqSo5fJb6wAmBR7SfMP3wz3FpY0Qm MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: aa974933-7ea4-489a-ad79-08d802ad1de6 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 May 2020 02:16:22.4631 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 5W9mLFhA+/AWXZDP5EU7UAJI+seDQWGhpwNtlL7DzEPqEFs7kt+8wGZcuesioltlEfRVpQNO9EpfKp+OCCJNWuN5D+DltqVlz+XayYFKlRg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4436 Return-Path: ashley.e.desimone@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Pushed: e405eb585095a0c1c238c56821423362a59a0265 -----Original Message----- From: Bjorge, Erik C =20 Sent: Tuesday, May 19, 2020 2:51 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E ; Desimone, Nathaniel L= ; Pandya, Puja ; Br= et Barkelew ; Agyeman, Prince Subject: [edk2-staging/EdkRepo] [PATCH v2 1/2] EdkRepo: Adding selective su= bmodule init script Adds selective submodule support functions and command line scripting suppo= rt. The existing support will be ported over to use this functionality. Signed-off-by: Erik Bjorge Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge --- edkrepo/common/ui_functions.py | 15 + project_utils/arguments/__init__.py | 8 + project_utils/arguments/submodule_args.py | 24 ++ project_utils/project_utils_strings.py | 24 ++ project_utils/submodule.py | 373 ++++++++++++++++++++++ setup.py | 50 +-- 6 files changed, 469 insertions(+), 25 deletions(-) create mode 100644 pr= oject_utils/arguments/__init__.py create mode 100644 project_utils/arguments/submodule_args.py create mode 100644 project_utils/project_utils_strings.py create mode 100644 project_utils/submodule.py diff --git a/edkrepo/common/ui_functions.py b/edkrepo/common/ui_functions.p= y index 844dd06..d42c2de 100644 --- a/edkrepo/common/ui_functions.py +++ b/edkrepo/common/ui_functions.py @@ -30,3 +30,18 @@ def init_color_console(force_color_output): convert=3DNone colorama.init(strip=3Dstrip, convert=3Dconvert) return strip, convert + + +def display_git_output(output_data, verbose=3DFalse): + """ + Displays output from GitPython git commands + + output_data - Output from the git.execute method + verbose - Enable verbose messages + """ + if verbose and output_data[0]: + print(output_data[0]) + if output_data[1]: + print(output_data[1]) + if verbose and output_data[2]: + print(output_data[2]) diff --git a/project_utils/arguments/__init__.py b/project_utils/arguments/= __init__.py new file mode 100644 index 0000000..0486261 --- /dev/null +++ b/project_utils/arguments/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +# +## @file +# __init__.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # diff --git a/project_utils/arguments/submodule_args.py b/project_utils/argu= ments/submodule_args.py new file mode 100644 index 0000000..804ce98 --- /dev/null +++ b/project_utils/arguments/submodule_args.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +""" +Strings for command line arguments. +""" + +SUBMOD_MANIFEST_HELP =3D 'The current manifest file.' +SUBMOD_COMBO_HELP =3D 'The current branch combination in use. If a combo i= s not specified, the current ' \ + 'combo will be used.' +SUBMOD_NEW_MANIFEST_HELP =3D 'The new manifest file. Used to test manifes= t upgrade paths.' +SUBMOD_NEW_COMBO_HELP =3D 'The new branch combination to use. Used to tes= t switching combos.' +SUBMOD_DEINIT_HELP =3D 'Performs submodule deinitialization before initial= izing submodules.' +SUBMOD_INIT_FULL_HELP =3D 'Initialize all submodules (recursive).' +SUBMOD_DEINIT_FULL_HELP =3D 'Deinitialize all submodules.' +SUBMOD_WORKSPACE_HELP =3D 'The project workspace root. If not specified t= he current working directory will ' \ + 'be used.' +SUBMOD_VERBOSE_HELP =3D 'Enables verbose messaging.' diff --git a/project_utils/project_utils_strings.py b/project_utils/project= _utils_strings.py new file mode 100644 index 0000000..33c22d2 --- /dev/null +++ b/project_utils/project_utils_strings.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# humble.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # + +""" +Contains strings used by the project_utils modules. +""" +# Status update messages +SUBMOD_DEINIT =3D 'Deinitializing submodules' +SUBMOD_DEINIT_FULL =3D 'Deinitializing all submodules' +SUBMOD_INIT_UPDATE =3D 'Initializing/Updating submodules' +SUBMOD_INIT_FULL =3D 'Initializing/Updating all submodules (recursive)' + +# Verbose messages +SUBMOD_INIT_PATH =3D 'Submodule init: {}' +SUBMOD_DEINIT_PATH =3D 'Submodule deinit: {}' +SUBMOD_SYNC_PATH =3D 'Submodule sync: {}' +SUBMOD_UPDATE_PATH =3D 'Submodule update: {}' +SUBMOD_EXCEPTION =3D '- Exception: {}' diff --git a/project_utils/submodule.py b/project_utils/submodule.py new fi= le mode 100644 index 0000000..170629b --- /dev/null +++ b/project_utils/submodule.py @@ -0,0 +1,373 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
#=20 +SPDX-License-Identifier: BSD-2-Clause-Patent # import argparse import=20 +git import os import sys import traceback + +from edkrepo_manifest_parser.edk_manifest import ManifestXml from=20 +edkrepo.common.ui_functions import display_git_output import=20 +project_utils.project_utils_strings as strings import=20 +project_utils.arguments.submodule_args as arguments + + +def _init(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs submodule initialization. Note that if no selective submodul= e initialization + is enabled the init will be handled in _update. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + # Only handle selective initialization with this method. For fully re= cursive init + # it is cleaner to do it as part of the update process. + if submodules is not None: + for sub in submodules: + if verbose: + print(strings.SUBMOD_INIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'init', = '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + display_git_output(output_data, verbose) + return + + +def _deinit(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs deinitialization of submodules. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + if submodules is None: + output_data =3D repo.git.execute(['git', 'submodule', 'deinit', '-= -all'], + with_extended_output=3DTrue, with_s= tdout=3DTrue) + display_git_output(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_DEINIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'deinit'= , '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + display_git_output(output_data, verbose) + return + + +def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse): + """ + Performs the update of submodules. This includes the sync and update = operations. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + recursive - If submodules is None then use this parameter to determin= e if initialization + should be recursive + """ + # Now perform the update of the submodules. For a fully recursive sub= module init + # this code will update and initialize at the same time. + if submodules is None: + cmd =3D ['git', 'submodule', 'sync'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + display_git_output(output_data, verbose) + cmd =3D ['git', 'submodule', 'update', '--init'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + display_git_output(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_SYNC_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'sync'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + display_git_output(output_data, verbose) + if verbose: + print(strings.SUBMOD_UPDATE_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'update', '--init'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + display_git_output(output_data, verbose) + return + + +def _compute_change(current_subs, new_subs): + """ + Determines the list of submodules that have been removed. Also needs = to + determine if any submodules need to have recursive init disabled + + current_subs - List of selective submodule init entries for the curren= t combo + new_subs - List of selective submodule init entries for the new co= mbo + """ + # Create data objects for determining what submodules need to be deini= tialized + tmp_current =3D {x.path: x for x in current_subs} + tmp_new =3D {x.path: x for x in new_subs} + + # Initial deinitialization list + deinit_paths =3D list(set(tmp_current).difference(set(tmp_new))) + + # Check for change in recursive initialization for the specific submod= ules + for path in tmp_new: + if path in tmp_new and path in tmp_current: + if tmp_current[path].recursive and not tmp_new[path].recursive= : + deinit_paths.append(path) + + # Create the final list of submodules that need to be deinitialized + return [x for x in current_subs if x.path in deinit_paths] + + +def _get_submodule_enable(manifest, remote_name, combo): + """ + Determines if submodules are enabled for the current repo and combo + + manifest - Manifest object + remote_name - The name of the current remote being processed + combo - The current combo name being processed + """ + repo_sources =3D manifest.get_repo_sources(combo) + for source in repo_sources: + if source.remote_name =3D=3D remote_name: + return source.enable_submodule + return False + + +def _get_submodule_state(remote_name, start_manifest, start_combo, end_man= ifest=3DNone, end_combo=3DNone): + """ + Determines the state of submodules across manifest and combo changes. + + remote_name - The name of the current remote being processed. + start_manifest - Initial manifest parser object. + start_combo - Initial combo. + end_manifest - The manifest parser object for the project if the= manifest is being updated. + end_combo - The combination name at the end of the operation = if being modified. + """ + start_subs =3D start_manifest.get_submodule_init_paths(remote_name, st= art_combo) + start_subs_enabled =3D _get_submodule_enable(start_manifest, remote_na= me, start_combo) + if end_combo is not None: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , end_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, end_combo) + else: + end_subs =3D start_manifest.get_submodule_init_paths(remote_na= me, end_combo) + end_subs_enabled =3D _get_submodule_enable(start_manifest, rem= ote_name, end_combo) + else: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , start_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, start_combo) + else: + end_subs =3D start_subs + end_subs_enabled =3D start_subs_enabled + return start_subs, start_subs_enabled, end_subs, end_subs_enabled + + +def deinit_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule deinit based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_DEINIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _deinit(repo, None, verbose) + + +def init_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule init based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_INIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _update(repo, None, verbose, True) + + +def deinit_submodules(workspace, start_manifest, start_combo, + end_manifest=3DNone, end_combo=3DNone, + verbose=3DFalse): + """ + Deinitializes the submodules for a project. + + workspace - Path to the current workspace. + start_manifest - The manifest parser object for the project. + start_combo - The combination name at the start of the operatio= n. + end_manifest - The manifest parser object for the project if the= manifest is being updated. If the + manifest file is not being updated us the value N= one. + end_combo - The combination name at the end of the operation.= If the combo will not change + use the value None. + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_DEINIT) + repo_sources =3D start_manifest.get_repo_sources(start_combo) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + start_subs, start_subs_enabled, end_subs, end_subs_enabled =3D _ge= t_submodule_state(source.remote_name, + = start_manifest, + = start_combo, + = end_manifest, + = end_combo) + if not start_subs_enabled and not end_subs_enabled: + # At this point submodules are not enabled on this repo + continue + + # Compute the list of submodules that need to be removed and added + deinit_list =3D _compute_change(start_subs, end_subs) + + # Deinitialize submodules + if (start_subs_enabled and not end_subs_enabled) or (len(start_sub= s) =3D=3D 0 and len(end_subs) !=3D 0): + # Submodules are being disabled for the entire repo so do a + # full deinit on the repo. + _deinit(repo, None, verbose) + else: + # Do the deinit based on the list + _deinit(repo, deinit_list, verbose) + + +def maintain_submodules(workspace, manifest, combo_name, verbose=3DFalse): + """ + Updates the submodules for a specific repo. + + workspace - Path to the current workspace. + manifest - The manifest parser object for the project. + combo_name - The combination name to use for submodule maintenance= . + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_INIT_UPDATE) + repo_sources =3D manifest.get_repo_sources(combo_name) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + repo_subs =3D manifest.get_submodule_init_paths(source.remote_name= , combo_name) + repo_subs_enabled =3D _get_submodule_enable(manifest, source.remot= e_name, combo_name) + if not repo_subs_enabled: + continue + + # Initialize submodules + if len(repo_subs) > 0: + _init(repo, repo_subs, verbose) + + # Perform sync/update + if len(repo_subs) =3D=3D 0: + _update(repo, None, verbose) + else: + _update(repo, repo_subs, verbose) + + +if __name__ =3D=3D '__main__': + def parse_args(): + parser =3D argparse.ArgumentParser() + parser.add_argument('manifest_file', metavar=3D'MANIFEST', help=3D= arguments.SUBMOD_MANIFEST_HELP) + parser.add_argument('--combo', default=3DNone, help=3Darguments.SU= BMOD_COMBO_HELP) + parser.add_argument('--new-manifest', default=3DNone, help=3Dargum= ents.SUBMOD_NEW_MANIFEST_HELP) + parser.add_argument('--new-combo', default=3DNone, help=3Dargument= s.SUBMOD_NEW_COMBO_HELP) + parser.add_argument('--deinit', action=3D'store_true', help=3Dargu= ments.SUBMOD_DEINIT_HELP) + parser.add_argument('--init-full', action=3D'store_true', help=3Da= rguments.SUBMOD_INIT_FULL_HELP) + parser.add_argument('--deinit-full', action=3D'store_true', help= =3Darguments.SUBMOD_DEINIT_FULL_HELP) + parser.add_argument('--workspace', default=3D'.', help=3Darguments= .SUBMOD_WORKSPACE_HELP) + parser.add_argument('--verbose', action=3D'store_true', help=3Darg= uments.SUBMOD_VERBOSE_HELP) + return parser.parse_args() + + def main(args): + # Optional data init + new_manifest =3D None + all_new_combos =3D None + + # Extract basic manifest data + manifest =3D ManifestXml(args.manifest_file) + init_manifest =3D manifest + all_combos =3D manifest.combinations + all_combos.extend(manifest.archived_combinations) + if args.new_manifest is not None: + new_manifest =3D ManifestXml(args.new_manifest) + all_new_combos =3D new_manifest.combinations + all_new_combos.extend(new_manifest.archived_combinations) + init_manifest =3D new_manifest + + # Determine current and new combo information + current_combo =3D manifest.general_config.current_combo + new_combo =3D args.new_combo + for combo in all_combos: + if args.combo is not None: + if args.combo.lower() =3D=3D combo.name.lower(): + current_combo =3D combo.name + if args.new_combo is not None: + if args.new_combo.lower() =3D=3D combo.name.lower(): + new_combo =3D combo.name + init_combo =3D current_combo + if new_combo is not None: + init_combo =3D new_combo + + if args.deinit_full: + deinit_full(args.workspace, manifest, args.verbose) + elif args.init_full: + init_full(args.workspace, manifest, args.verbose) + else: + if args.deinit: + deinit_submodules(args.workspace, manifest, current_combo, + new_manifest, new_combo, args.verbose) + maintain_submodules(args.workspace, init_manifest,=20 + init_combo, args.verbose) + + return 0 + + try: + args =3D parse_args() + sys.exit(main(args)) + except Exception: + if args.verbose: + traceback.print_exc() + sys.exit(1) diff --git a/setup.py b/setup.py index 5d2f55a..a855000 100755 --- a/setup.py +++ b/setup.py @@ -1,27 +1,27 @@ -#!/usr/bin/env python3 -# -## @file setup.py -# -# Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.
-= # SPDX-License-Identifier: BSD-2-Clause-Patent -# - -from setuptools import setup - -setup(name=3D'edkrepo', - version=3D'2.0.0', - description=3D'The edkrepo tools', - packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.commands.humble', +#!/usr/bin/env python3 +# +## @file setup.py +# +# Copyright (c) 2017 - 2020, Intel Corporation. All rights=20 +reserved.
# SPDX-License-Identifier: BSD-2-Clause-Patent # + +from setuptools import setup + +setup(name=3D'edkrepo', + version=3D'2.0.0', + description=3D'The edkrepo tools', + packages=3D['edkrepo', 'edkrepo.commands',=20 +'edkrepo.commands.arguments', 'edkrepo.commands.humble', 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.commo= n.workspace_maintenance', 'edkrepo.common.workspace_maintenance.humble', 'edkrepo.co= nfig', 'edkrepo.config.humble', - 'edkrepo_manifest_parser', 'project_utils'], - package_data=3D{ - }, - include_package_data=3DTrue, - entry_points=3D{ - 'console_scripts': [ - 'edkrepo =3D edkrepo.edkrepo_entry_point:main', - 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' - ] - } - ) + 'edkrepo_manifest_parser', 'project_utils', 'project_utils= .arguments'], + package_data=3D{ + }, + include_package_data=3DTrue, + entry_points=3D{ + 'console_scripts': [ + 'edkrepo =3D edkrepo.edkrepo_entry_point:main', + 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' + ] + } + ) -- 2.21.0.windows.1