[edk2-devel] [PATCH v2 0/1] MdeModulePkg: Warn if out of space when writing variables

[edk2-devel] [PATCH v2 0/1] MdeModulePkg: Warn if out of space when writing variables

[Oliver Steffen]

Emit a DEBUG_ERROR message if there is not enough space left to
write/update a variable. This condition is currently not logged
appropriately in all cases, given that full variable store can easily
render the system unbootable.
This new message helps identifying this condition quickly.

Background:
We had the case where an OVMF based VM ran out of flash space after a
large number of reboots and hardware configuration changes. The MAC
address of the NIC was changing a lot over the life time of the VM. The
variable store was consumed entirely by networking related non-volatile
variables, written for each network card/MAC address present during
boot. Almost all of those were obsolete, of course, but the error that
presented itself at first was a VM that failed to boot the firmware and
ended up in an rather cryptic ASSERT message.

The hope is that this new message makes it easier for non-experts to
diagnose the situation.

The root cause is of course not addressed by this, but it might also be
hard to solve. One would have to identify obsolete network/hardware
settings somehow, but there is no way to tell if a certain NIC/MAC
might come back at a later point or not.


Launching OVMF with a full variable store then looks like this (Debug
Output):
[...]
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space
Select Item: 0xE
Select Item: 0x19
RecordVarErrorFlag (0xEF) Timeout:8BE4DF61-93CA-11D2-AA0D-00E098032B8C - 0x00000007 - 0x4E
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space
PlatformBootManagerBeforeConsole: SetVariable(Timeout, 3): Out of Resources
RecordVarErrorFlag (0xEF) BootOrder:8BE4DF61-93CA-11D2-AA0D-00E098032B8C - 0x00000007 - 0x52
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space

ASSERT_EFI_ERROR (Status = Out of Resources)
ASSERT OvmfPkg/Library/PlatformBootManagerLib/BdsPlatform.c(361): !(((INTN)(RETURN_STATUS)(Status)) < 0)


PR: https://github.com/tianocore/edk2/pull/5435

Changes since v1:
- Mention "variable space" instead of "flash space"
- Emit DEBUG_ERROR instead of DEBUG_WARN

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>

Oliver Steffen (1):
  MdeModulePkg: Warn if out of space when writing variables

 MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c | 2 ++
 1 file changed, 2 insertions(+)

--
2.44.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#117803): https://edk2.groups.io/g/devel/message/117803
Mute This Topic: https://groups.io/mt/105543778/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v2 1/1] MdeModulePkg: Warn if out of space when writing variables

[Oliver Steffen]

Emit a DEBUG_ERROR message if there is not enough flash variable left to
write/update a variable. This condition is currently not logged
appropriately in all cases, given that full variable store can easily
render the system unbootable.
This new message helps identifying this condition.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
---
 MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c b/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
index d394d237a53f..97317270c4a7 100644
--- a/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
+++ b/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
@@ -2364,6 +2364,8 @@ UpdateVariable (
                   );
       ASSERT_EFI_ERROR (Status);
     }
+  } else if (Status == EFI_OUT_OF_RESOURCES) {
+    DEBUG ((DEBUG_ERROR, "UpdateVariable failed: Out of variable space\n"));
   }
 
   return Status;
-- 
2.44.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#117804): https://edk2.groups.io/g/devel/message/117804
Mute This Topic: https://groups.io/mt/105543779/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v2 1/1] MdeModulePkg: Warn if out of space when writing variables

[Gerd Hoffmann]

On Mon, Apr 15, 2024 at 09:46:37PM +0200, Oliver Steffen wrote:
> Emit a DEBUG_ERROR message if there is not enough flash variable left to
> write/update a variable. This condition is currently not logged
> appropriately in all cases, given that full variable store can easily
> render the system unbootable.
> This new message helps identifying this condition.
> 
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Rahul Kumar <rahul1.kumar@intel.com>
> Cc: Rebecca Cran <rebecca@bsdio.com>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> 
> Signed-off-by: Oliver Steffen <osteffen@redhat.com>
> Reviewed-by: Laszlo Ersek <lersek@redhat.com>

Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>

take care,
  Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#117854): https://edk2.groups.io/g/devel/message/117854
Mute This Topic: https://groups.io/mt/105543779/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v2 0/1] MdeModulePkg: Warn if out of space when writing variables

[Oliver Steffen]

Emit a DEBUG_ERROR message if there is not enough space left to
write/update a variable. This condition is currently not logged
appropriately in all cases, given that full variable store can easily
render the system unbootable.
This new message helps identifying this condition quickly.

Background:
We had the case where an OVMF based VM ran out of flash space after a
large number of reboots and hardware configuration changes. The MAC
address of the NIC was changing a lot over the life time of the VM. The
variable store was consumed entirely by networking related non-volatile
variables, written for each network card/MAC address present during
boot. Almost all of those were obsolete, of course, but the error that
presented itself at first was a VM that failed to boot the firmware and
ended up in an rather cryptic ASSERT message.

The hope is that this new message makes it easier for non-experts to
diagnose the situation.

The root cause is of course not addressed by this, but it might also be
hard to solve. One would have to identify obsolete network/hardware
settings somehow, but there is no way to tell if a certain NIC/MAC
might come back at a later point or not.


Launching OVMF with a full variable store then looks like this (Debug
Output):
[...]
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space
Select Item: 0xE
Select Item: 0x19
RecordVarErrorFlag (0xEF) Timeout:8BE4DF61-93CA-11D2-AA0D-00E098032B8C - 0x00000007 - 0x4E
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space
PlatformBootManagerBeforeConsole: SetVariable(Timeout, 3): Out of Resources
RecordVarErrorFlag (0xEF) BootOrder:8BE4DF61-93CA-11D2-AA0D-00E098032B8C - 0x00000007 - 0x52
CommonVariableSpace = 0x3FF9C - CommonVariableTotalSize = 0x3FF60
UpdateVariable failed: Out of variable space

ASSERT_EFI_ERROR (Status = Out of Resources)
ASSERT OvmfPkg/Library/PlatformBootManagerLib/BdsPlatform.c(361): !(((INTN)(RETURN_STATUS)(Status)) < 0)


PR: https://github.com/tianocore/edk2/pull/5435

Changes since v1:
- Mention "variable space" instead of "flash space"
- Emit DEBUG_ERROR instead of DEBUG_WARN

Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>

Oliver Steffen (1):
  MdeModulePkg: Warn if out of space when writing variables

 MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c | 2 ++
 1 file changed, 2 insertions(+)

--
2.44.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#118819): https://edk2.groups.io/g/devel/message/118819
Mute This Topic: https://groups.io/mt/106017604/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

[edk2-devel] [PATCH v2 1/1] MdeModulePkg: Warn if out of space when writing variables

[Oliver Steffen]

Emit a DEBUG_ERROR message if there is not enough flash variable left to
write/update a variable. This condition is currently not logged
appropriately in all cases, given that full variable store can easily
render the system unbootable.
This new message helps identifying this condition.

Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Zhiguang Liu <zhiguang.liu@intel.com>

Signed-off-by: Oliver Steffen <osteffen@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
---
 MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c b/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
index d394d237a53f..97317270c4a7 100644
--- a/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
+++ b/MdeModulePkg/Universal/Variable/RuntimeDxe/Variable.c
@@ -2364,6 +2364,8 @@ UpdateVariable (
                   );
       ASSERT_EFI_ERROR (Status);
     }
+  } else if (Status == EFI_OUT_OF_RESOURCES) {
+    DEBUG ((DEBUG_ERROR, "UpdateVariable failed: Out of variable space\n"));
   }
 
   return Status;
-- 
2.44.0



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#118820): https://edk2.groups.io/g/devel/message/118820
Mute This Topic: https://groups.io/mt/106022074/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-