[edk2-staging/EdkRepo] [PATCH v2 0/3] Small fixes to submodule support

[edk2-staging/EdkRepo] [PATCH v2 0/3] Small fixes to submodule support

[Bjorge, Erik C]

Fixed issues when submodule has untracked files and can't be deinitialized.
Fixed backwards compatibility issue with old pin files.
Fixed issue when switching combos to handle submodule removals.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>


Erik Bjorge (3):
  EdkRepo: Add force flag when removing submodules
  EdkRepo: Adding backwards compatibility for old pin files
  EdkRepo: Fix submodule failures when switching combos.

 edkrepo/commands/checkout_pin_command.py |  5 ++---
 edkrepo/common/common_repo_functions.py  | 11 ++++++-----
 edkrepo_manifest_parser/edk_manifest.py  |  6 +++++-
 project_utils/submodule.py               |  4 ++--
 4 files changed, 15 insertions(+), 11 deletions(-)

--
2.27.0.windows.1

[edk2-staging/EdkRepo] [PATCH v2 1/3] EdkRepo: Add force flag when removing submodules

[Bjorge, Erik C]

Submodules may generate un-tracked or modify files in the repo.  when
removing the submodule do so even with modifications.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 project_utils/submodule.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/project_utils/submodule.py b/project_utils/submodule.py
index 170629b..3d1b620 100644
--- a/project_utils/submodule.py
+++ b/project_utils/submodule.py
@@ -48,14 +48,14 @@ def _deinit(repo, submodules=None, verbose=False):
     verbose    - Enable verbose messages
     """
     if submodules is None:
-        output_data = repo.git.execute(['git', 'submodule', 'deinit', '--all'],
+        output_data = repo.git.execute(['git', 'submodule', 'deinit', '-f', '--all'],
                                        with_extended_output=True, with_stdout=True)
         display_git_output(output_data, verbose)
     else:
         for sub in submodules:
             if verbose:
                 print(strings.SUBMOD_DEINIT_PATH.format(sub.path))
-            output_data = repo.git.execute(['git', 'submodule', 'deinit', '--', sub.path],
+            output_data = repo.git.execute(['git', 'submodule', 'deinit', '-f', '--', sub.path],
                                            with_extended_output=True, with_stdout=True)
             display_git_output(output_data, verbose)
     return
-- 
2.27.0.windows.1

[edk2-staging/EdkRepo] [PATCH v2 2/3] EdkRepo: Adding backwards compatibility for old pin files

[Bjorge, Erik C]

Older pin files used the invalid enable_submodule attribute.  This has
been fixed for new pin files but we need to be able to support older pin
files as well.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 edkrepo_manifest_parser/edk_manifest.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/edkrepo_manifest_parser/edk_manifest.py b/edkrepo_manifest_parser/edk_manifest.py
index adf6d52..00ad392 100644
--- a/edkrepo_manifest_parser/edk_manifest.py
+++ b/edkrepo_manifest_parser/edk_manifest.py
@@ -795,7 +795,11 @@ class _RepoSource():
             # If enableSubmodule is not set to True then default to False
             self.enableSub = (element.attrib['enableSubmodule'].lower() == 'true')
         except Exception:
-            self.enableSub = False
+            try:
+                # Adding backwards compatibility with pin files that used incorrect attribute
+                self.enableSub = (element.attrib['enable_submodule'].lower() == 'true')
+            except Exception:
+                self.enableSub = False
 
         if self.branch is None and self.commit is None and self.tag is None:
             raise KeyError(ATTRIBUTE_MISSING_ERROR)
-- 
2.27.0.windows.1

[edk2-staging/EdkRepo] [PATCH v2 3/3] EdkRepo: Fix submodule failures when switching combos.

[Bjorge, Erik C]

When switching combinations or checking out pins the submodule
information may change or have modifications that will cause errors.
Before changing to a new combination all submodules should be removed.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 edkrepo/commands/checkout_pin_command.py |  5 ++---
 edkrepo/common/common_repo_functions.py  | 11 ++++++-----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/edkrepo/commands/checkout_pin_command.py b/edkrepo/commands/checkout_pin_command.py
index 9d7346a..39c5aeb 100644
--- a/edkrepo/commands/checkout_pin_command.py
+++ b/edkrepo/commands/checkout_pin_command.py
@@ -22,7 +22,7 @@ from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import list
 from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import find_source_manifest_repo
 from edkrepo.config.config_factory import get_workspace_path, get_workspace_manifest
 from edkrepo_manifest_parser.edk_manifest import ManifestXml
-from project_utils.submodule import deinit_submodules, maintain_submodules
+from project_utils.submodule import deinit_full, maintain_submodules
 
 
 class CheckoutPinCommand(EdkrepoCommand):
@@ -71,8 +71,7 @@ class CheckoutPinCommand(EdkrepoCommand):
             print(SPARSE_RESET)
             reset_sparse_checkout(workspace_path, manifest_sources)
         submodule_combo = pin.general_config.current_combo
-        deinit_submodules(workspace_path, manifest, manifest.general_config.current_combo,
-                          pin, submodule_combo, args.verbose)
+        deinit_full(workspace_path, manifest, args.verbose)
         pin_repo_sources = pin.get_repo_sources(pin.general_config.current_combo)
         try:
             checkout_repos(args.verbose, args.override, pin_repo_sources, workspace_path, manifest)
diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
index 8c79f3d..59c198f 100644
--- a/edkrepo/common/common_repo_functions.py
+++ b/edkrepo/common/common_repo_functions.py
@@ -68,7 +68,7 @@ from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrep
 from edkrepo_manifest_parser.edk_manifest_validation import get_manifest_validation_status
 from edkrepo_manifest_parser.edk_manifest_validation import print_manifest_errors
 from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestfiles
-from project_utils.submodule import deinit_submodules, maintain_submodules
+from project_utils.submodule import deinit_full, maintain_submodules
 
 CLEAR_LINE = '\x1b[K'
 DEFAULT_REMOTE_NAME = 'origin'
@@ -498,10 +498,6 @@ def checkout(combination_or_sha, verbose=False, override=False, log=None):
         log=log)
     initial_repo_sources = manifest.get_repo_sources(manifest.general_config.current_combo)
 
-    # Deinit any submodules that have been removed.
-    deinit_submodules(workspace_path, manifest, manifest.general_config.current_combo,
-                      manifest, submodule_combo, verbose)
-
     # Disable sparse checkout
     current_repos = initial_repo_sources
     sparse_enabled = sparse_checkout_enabled(workspace_path, initial_repo_sources)
@@ -524,6 +520,11 @@ def checkout(combination_or_sha, verbose=False, override=False, log=None):
         print(SPARSE_RESET)
         reset_sparse_checkout(workspace_path, current_repos)
 
+    # Deinit all submodules due to the potential for issues when switching
+    # branches.
+    if combo_or_sha != manifest.general_config.current_combo:
+        deinit_full(workspace_path, manifest, verbose)
+
     print(CHECKING_OUT_COMBO.format(combo_or_sha))
 
     try:
-- 
2.27.0.windows.1

Re: [edk2-staging/EdkRepo] [PATCH v2 0/3] Small fixes to submodule support

[Ashley E Desimone]

For the series...

Reviewed-by: Ashley DeSimone <ashley.e.desimone@intel.com>

-----Original Message-----
From: Bjorge, Erik C <erik.c.bjorge@intel.com> 
Sent: Wednesday, June 10, 2020 3:14 PM
To: devel@edk2.groups.io
Cc: Desimone, Ashley E <ashley.e.desimone@intel.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH v2 0/3] Small fixes to submodule support

Fixed issues when submodule has untracked files and can't be deinitialized.
Fixed backwards compatibility issue with old pin files.
Fixed issue when switching combos to handle submodule removals.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>


Erik Bjorge (3):
  EdkRepo: Add force flag when removing submodules
  EdkRepo: Adding backwards compatibility for old pin files
  EdkRepo: Fix submodule failures when switching combos.

 edkrepo/commands/checkout_pin_command.py |  5 ++---  edkrepo/common/common_repo_functions.py  | 11 ++++++-----  edkrepo_manifest_parser/edk_manifest.py  |  6 +++++-
 project_utils/submodule.py               |  4 ++--
 4 files changed, 15 insertions(+), 11 deletions(-)

--
2.27.0.windows.1

Re: [edk2-staging/EdkRepo] [PATCH v2 1/3] EdkRepo: Add force flag when removing submodules

[Ashley E Desimone]

Pushed as: be6fadb6b0398cbcdf8ddbd40f5c16b67d17e551

-----Original Message-----
From: Bjorge, Erik C <erik.c.bjorge@intel.com> 
Sent: Wednesday, June 10, 2020 3:14 PM
To: devel@edk2.groups.io
Cc: Desimone, Ashley E <ashley.e.desimone@intel.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH v2 1/3] EdkRepo: Add force flag when removing submodules

Submodules may generate un-tracked or modify files in the repo.  when removing the submodule do so even with modifications.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 project_utils/submodule.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/project_utils/submodule.py b/project_utils/submodule.py index 170629b..3d1b620 100644
--- a/project_utils/submodule.py
+++ b/project_utils/submodule.py
@@ -48,14 +48,14 @@ def _deinit(repo, submodules=None, verbose=False):
     verbose    - Enable verbose messages     """     if submodules is None:-        output_data = repo.git.execute(['git', 'submodule', 'deinit', '--all'],+        output_data = repo.git.execute(['git', 'submodule', 'deinit', '-f', '--all'],                                        with_extended_output=True, with_stdout=True)         display_git_output(output_data, verbose)     else:         for sub in submodules:             if verbose:                 print(strings.SUBMOD_DEINIT_PATH.format(sub.path))-            output_data = repo.git.execute(['git', 'submodule', 'deinit', '--', sub.path],+            output_data = repo.git.execute(['git', 'submodule', 'deinit', '-f', '--', sub.path],                                            with_extended_output=True, with_stdout=True)             display_git_output(output_data, verbose)     return-- 
2.27.0.windows.1

Re: [edk2-staging/EdkRepo] [PATCH v2 2/3] EdkRepo: Adding backwards compatibility for old pin files

[Ashley E Desimone]

Pushed as: a1ae934c28b5781b358a270fed77e35d23b205b0

-----Original Message-----
From: Bjorge, Erik C <erik.c.bjorge@intel.com> 
Sent: Wednesday, June 10, 2020 3:14 PM
To: devel@edk2.groups.io
Cc: Desimone, Ashley E <ashley.e.desimone@intel.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-staging/EdkRepo] [PATCH v2 2/3] EdkRepo: Adding backwards compatibility for old pin files

Older pin files used the invalid enable_submodule attribute.  This has been fixed for new pin files but we need to be able to support older pin files as well.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 edkrepo_manifest_parser/edk_manifest.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/edkrepo_manifest_parser/edk_manifest.py b/edkrepo_manifest_parser/edk_manifest.py
index adf6d52..00ad392 100644
--- a/edkrepo_manifest_parser/edk_manifest.py
+++ b/edkrepo_manifest_parser/edk_manifest.py
@@ -795,7 +795,11 @@ class _RepoSource():
             # If enableSubmodule is not set to True then default to False             self.enableSub = (element.attrib['enableSubmodule'].lower() == 'true')         except Exception:-            self.enableSub = False+            try:+                # Adding backwards compatibility with pin files that used incorrect attribute+                self.enableSub = (element.attrib['enable_submodule'].lower() == 'true')+            except Exception:+                self.enableSub = False          if self.branch is None and self.commit is None and self.tag is None:             raise KeyError(ATTRIBUTE_MISSING_ERROR)-- 
2.27.0.windows.1

Re: [edk2-devel] [edk2-staging/EdkRepo] [PATCH v2 3/3] EdkRepo: Fix submodule failures when switching combos.

[Ashley E Desimone]

Pushed as: 2392e3fa84883958822e056ae35f5b4afade5a80

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bjorge, Erik C
Sent: Wednesday, June 10, 2020 3:14 PM
To: devel@edk2.groups.io
Cc: Desimone, Ashley E <ashley.e.desimone@intel.com>; Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Agyeman, Prince <prince.agyeman@intel.com>
Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH v2 3/3] EdkRepo: Fix submodule failures when switching combos.

When switching combinations or checking out pins the submodule information may change or have modifications that will cause errors.
Before changing to a new combination all submodules should be removed.

Cc: Ashley E Desimone <ashley.e.desimone@intel.com>
Cc: Nate DeSimone <nathaniel.l.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Prince Agyeman <prince.agyeman@intel.com>
Cc: Erik Bjorge <erik.c.bjorge@intel.com>
Signed-off-by: Erik Bjorge <erik.c.bjorge@intel.com>
---
 edkrepo/commands/checkout_pin_command.py |  5 ++---  edkrepo/common/common_repo_functions.py  | 11 ++++++-----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/edkrepo/commands/checkout_pin_command.py b/edkrepo/commands/checkout_pin_command.py
index 9d7346a..39c5aeb 100644
--- a/edkrepo/commands/checkout_pin_command.py
+++ b/edkrepo/commands/checkout_pin_command.py
@@ -22,7 +22,7 @@ from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import list
 from edkrepo.common.workspace_maintenance.manifest_repos_maintenance import find_source_manifest_repo from edkrepo.config.config_factory import get_workspace_path, get_workspace_manifest from edkrepo_manifest_parser.edk_manifest import ManifestXml-from project_utils.submodule import deinit_submodules, maintain_submodules+from project_utils.submodule import deinit_full, maintain_submodules   class CheckoutPinCommand(EdkrepoCommand):@@ -71,8 +71,7 @@ class CheckoutPinCommand(EdkrepoCommand):
             print(SPARSE_RESET)             reset_sparse_checkout(workspace_path, manifest_sources)         submodule_combo = pin.general_config.current_combo-        deinit_submodules(workspace_path, manifest, manifest.general_config.current_combo,-                          pin, submodule_combo, args.verbose)+        deinit_full(workspace_path, manifest, args.verbose)         pin_repo_sources = pin.get_repo_sources(pin.general_config.current_combo)         try:             checkout_repos(args.verbose, args.override, pin_repo_sources, workspace_path, manifest)diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/common_repo_functions.py
index 8c79f3d..59c198f 100644
--- a/edkrepo/common/common_repo_functions.py
+++ b/edkrepo/common/common_repo_functions.py
@@ -68,7 +68,7 @@ from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestrep  from edkrepo_manifest_parser.edk_manifest_validation import get_manifest_validation_status from edkrepo_manifest_parser.edk_manifest_validation import print_manifest_errors from edkrepo_manifest_parser.edk_manifest_validation import validate_manifestfiles-from project_utils.submodule import deinit_submodules, maintain_submodules+from project_utils.submodule import deinit_full, maintain_submodules  CLEAR_LINE = '\x1b[K' DEFAULT_REMOTE_NAME = 'origin'@@ -498,10 +498,6 @@ def checkout(combination_or_sha, verbose=False, override=False, log=None):
         log=log)     initial_repo_sources = manifest.get_repo_sources(manifest.general_config.current_combo) -    # Deinit any submodules that have been removed.-    deinit_submodules(workspace_path, manifest, manifest.general_config.current_combo,-                      manifest, submodule_combo, verbose)-     # Disable sparse checkout     current_repos = initial_repo_sources     sparse_enabled = sparse_checkout_enabled(workspace_path, initial_repo_sources)@@ -524,6 +520,11 @@ def checkout(combination_or_sha, verbose=False, override=False, log=None):
         print(SPARSE_RESET)         reset_sparse_checkout(workspace_path, current_repos) +    # Deinit all submodules due to the potential for issues when switching+    # branches.+    if combo_or_sha != manifest.general_config.current_combo:+        deinit_full(workspace_path, manifest, verbose)+     print(CHECKING_OUT_COMBO.format(combo_or_sha))      try:-- 
2.27.0.windows.1


-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#61101): https://edk2.groups.io/g/devel/message/61101
Mute This Topic: https://groups.io/mt/74807390/1860321
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [ashley.e.desimone@intel.com] -=-=-=-=-=-=