How to handle pflash backed OVMF FW upgrade and live migration best?

How to handle pflash backed OVMF FW upgrade and live migration best?

[Thomas Lamprecht]

Hi,

I'm currently evaluating how to update the firmware (OVMF) code image
without impacting a KVM/QEMU VM on live migration. I.e., the FW code lives
under /usr/share/OVMF/OVMF_CODE.fd and gets passed to the QEMU command with:

qemu-binary [...] -drive "if=pflash,unit=0,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd"

Now if the target node has an updated version of OVMF the VM does not really
likes that, as from its POV it gets effectively another code image loaded
from one moment at the other without any notice.

So my questions is if it would make sense to see this read-only pflash
content as "VM state" and send it over during live migration? This would
make migration way easier. Else we need to save all FW files and track which
one the VM is using, so that when starting the migration target VM we pass
along the correct pflash drive file. Sending over a pflash drive could maybe
only get done when a special flag is set for the pflash drive?

As said I can work around in our management stack, but saving the FW image
and tracking which VM uses what version, and that cluster wide, may get
quite a headache and we would need to keep all older OVMF binaries around...

If I'm missing something and there's already an easy way for this I'd be
very happy to hear from it.

Besides qemu-discuss I posted it to edk2-devel as there maybe more people
are in the QEMU and OVMF user intersection. :)

cheers,
Thomas

Re: How to handle pflash backed OVMF FW upgrade and live migration best?

[Laszlo Ersek]

CC Dave

On 03/01/18 12:21, Thomas Lamprecht wrote:
> Hi,
> 
> I'm currently evaluating how to update the firmware (OVMF) code image
> without impacting a KVM/QEMU VM on live migration. I.e., the FW code lives
> under /usr/share/OVMF/OVMF_CODE.fd and gets passed to the QEMU command with:
> 
> qemu-binary [...] -drive "if=pflash,unit=0,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd"
> 
> Now if the target node has an updated version of OVMF the VM does not really
> likes that, as from its POV it gets effectively another code image loaded
> from one moment at the other without any notice.

This should not cause any issues. On the destination host, the
destination QEMU instance should load the (different) OVMF_CODE.fd image
into the pflash chip, at startup. However, the incoming migration stream
contains, as a RAMBlock, the original OVMF_CODE.fd image. In other
words, the original firmware image is migrated (in memory, as part of
the migration stream) too.

(

BTW, there is very little firmware code in OVMF that actually *executes*
from pflash -- that's just the SEC module. SEC decompresses the PEI and
DXE firmware volumes from pflash to RAM, and the rest of the firmware
runs from normal RAM. This applies to runtime firmware services as well.
So about the only times when OVMF_CODE.fd (in the pflash chip) and
migration intersect are:
- if you migrate while SEC is running from pflash (i.e. the earliest
part of the boot),
- if you warm-reboot on the destination host after migration -- in this
case, the OVMF_CODE.fd binary (that got migrated in the pflash RAMBlock
from the source host) will again boot from pflash.

)

> So my questions is if it would make sense to see this read-only pflash
> content as "VM state" and send it over during live migration?

That's what already happens.

Now, if you have a differently *sized* OVMF_CODE.fd image on the
destination host, that could be a problem. Avoiding such problems is an
IT / distro job.

There are some "build aspects" of OVMF that can make two OVMF binaries
"incompatible" in this sense. Using *some* different build flags it's
also possible to make (a) an OVMF binary and (b) a varstore file
originally created for another OVMF binary, incompatible.

> This would
> make migration way easier. Else we need to save all FW files and track which
> one the VM is using, so that when starting the migration target VM we pass
> along the correct pflash drive file. Sending over a pflash drive could maybe
> only get done when a special flag is set for the pflash drive?
> 
> As said I can work around in our management stack, but saving the FW image
> and tracking which VM uses what version, and that cluster wide, may get
> quite a headache and we would need to keep all older OVMF binaries around...

When you deploy new OVMF binaries (packages) to a subset of your
virtualization hosts, you are responsible for keeping those compatible.
(They *can* contain code updates, but those updates have to be
compatible.) If a new OVMF binary is built that is known to be
incompatible, then it has to be installed under a different pathname
(either via a separate package; or in the same package, but still under
a different pathname).

To give you the simplest example, binaries (and varstores) corresponding
to FD_SIZE_2MB and FD_SIZE_4MB are incompatible. If a domain is
originally defined on top of an FD_SIZE_2MB OVMF, then it likely cannot
be migrated to a host where the same OVMF pathname refers to an
FD_SIZE_4MB binary. If you have a mixed environment, then you need to
carry both binaries to all hosts (and if you backport fixes from
upstream edk2, you need to backport those to both binaries).

In addition, assuming the domain is powered down for good (the QEMU
process terminates), and you update the domain XML from the FD_SIZE_2MB
OVMF binary to the FD_SIZE_4MB binary, you *also* have to
delete/recreate the domain's variable store file (losing all UEFI
variables the domain has accumulated until then). This is because the
FD_SIZE_4MB binary is incompatible with the varstore that was originally
created for the FD_SIZE_2MB binary (and vice versa).

Thanks
Laszlo

> If I'm missing something and there's already an easy way for this I'd be
> very happy to hear from it.
> 
> Besides qemu-discuss I posted it to edk2-devel as there maybe more people
> are in the QEMU and OVMF user intersection. :)

Re: How to handle pflash backed OVMF FW upgrade and live migration best?

[Dr. David Alan Gilbert]

* Laszlo Ersek (lersek@redhat.com) wrote:
> CC Dave
> 
> On 03/01/18 12:21, Thomas Lamprecht wrote:
> > Hi,
> > 
> > I'm currently evaluating how to update the firmware (OVMF) code image
> > without impacting a KVM/QEMU VM on live migration. I.e., the FW code lives
> > under /usr/share/OVMF/OVMF_CODE.fd and gets passed to the QEMU command with:
> > 
> > qemu-binary [...] -drive "if=pflash,unit=0,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd"
> > 
> > Now if the target node has an updated version of OVMF the VM does not really
> > likes that, as from its POV it gets effectively another code image loaded
> > from one moment at the other without any notice.
> 
> This should not cause any issues. On the destination host, the
> destination QEMU instance should load the (different) OVMF_CODE.fd image
> into the pflash chip, at startup. However, the incoming migration stream
> contains, as a RAMBlock, the original OVMF_CODE.fd image. In other
> words, the original firmware image is migrated (in memory, as part of
> the migration stream) too.

Yep.

> (
> 
> BTW, there is very little firmware code in OVMF that actually *executes*
> from pflash -- that's just the SEC module. SEC decompresses the PEI and
> DXE firmware volumes from pflash to RAM, and the rest of the firmware
> runs from normal RAM. This applies to runtime firmware services as well.
> So about the only times when OVMF_CODE.fd (in the pflash chip) and
> migration intersect are:
> - if you migrate while SEC is running from pflash (i.e. the earliest
> part of the boot),
> - if you warm-reboot on the destination host after migration -- in this
> case, the OVMF_CODE.fd binary (that got migrated in the pflash RAMBlock
> from the source host) will again boot from pflash.
> 
> )
> 
> > So my questions is if it would make sense to see this read-only pflash
> > content as "VM state" and send it over during live migration?
> 
> That's what already happens.
> 
> Now, if you have a differently *sized* OVMF_CODE.fd image on the
> destination host, that could be a problem. Avoiding such problems is an
> IT / distro job.

Yeh; padding the binaries to a power-of-2 with a bit of space left is
a good rule of thumb.

> There are some "build aspects" of OVMF that can make two OVMF binaries
> "incompatible" in this sense. Using *some* different build flags it's
> also possible to make (a) an OVMF binary and (b) a varstore file
> originally created for another OVMF binary, incompatible.

Fun.

> > This would
> > make migration way easier. Else we need to save all FW files and track which
> > one the VM is using, so that when starting the migration target VM we pass
> > along the correct pflash drive file. Sending over a pflash drive could maybe
> > only get done when a special flag is set for the pflash drive?
> > 
> > As said I can work around in our management stack, but saving the FW image
> > and tracking which VM uses what version, and that cluster wide, may get
> > quite a headache and we would need to keep all older OVMF binaries around...
> 
> When you deploy new OVMF binaries (packages) to a subset of your
> virtualization hosts, you are responsible for keeping those compatible.
> (They *can* contain code updates, but those updates have to be
> compatible.) If a new OVMF binary is built that is known to be
> incompatible, then it has to be installed under a different pathname
> (either via a separate package; or in the same package, but still under
> a different pathname).
> 
> To give you the simplest example, binaries (and varstores) corresponding
> to FD_SIZE_2MB and FD_SIZE_4MB are incompatible. If a domain is
> originally defined on top of an FD_SIZE_2MB OVMF, then it likely cannot
> be migrated to a host where the same OVMF pathname refers to an
> FD_SIZE_4MB binary. If you have a mixed environment, then you need to
> carry both binaries to all hosts (and if you backport fixes from
> upstream edk2, you need to backport those to both binaries).
> 
> In addition, assuming the domain is powered down for good (the QEMU
> process terminates), and you update the domain XML from the FD_SIZE_2MB
> OVMF binary to the FD_SIZE_4MB binary, you *also* have to
> delete/recreate the domain's variable store file (losing all UEFI
> variables the domain has accumulated until then). This is because the
> FD_SIZE_4MB binary is incompatible with the varstore that was originally
> created for the FD_SIZE_2MB binary (and vice versa).

I assume that gives a clear and obvious error message - right?

Dave

> Thanks
> Laszlo
> 
> > If I'm missing something and there's already an easy way for this I'd be
> > very happy to hear from it.
> > 
> > Besides qemu-discuss I posted it to edk2-devel as there maybe more people
> > are in the QEMU and OVMF user intersection. :)
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: How to handle pflash backed OVMF FW upgrade and live migration best?

[Laszlo Ersek]

On 03/02/18 13:53, Dr. David Alan Gilbert wrote:
> * Laszlo Ersek (lersek@redhat.com) wrote:
>> CC Dave

>> To give you the simplest example, binaries (and varstores) corresponding
>> to FD_SIZE_2MB and FD_SIZE_4MB are incompatible. If a domain is
>> originally defined on top of an FD_SIZE_2MB OVMF, then it likely cannot
>> be migrated to a host where the same OVMF pathname refers to an
>> FD_SIZE_4MB binary. If you have a mixed environment, then you need to
>> carry both binaries to all hosts (and if you backport fixes from
>> upstream edk2, you need to backport those to both binaries).
>>
>> In addition, assuming the domain is powered down for good (the QEMU
>> process terminates), and you update the domain XML from the FD_SIZE_2MB
>> OVMF binary to the FD_SIZE_4MB binary, you *also* have to
>> delete/recreate the domain's variable store file (losing all UEFI
>> variables the domain has accumulated until then). This is because the
>> FD_SIZE_4MB binary is incompatible with the varstore that was originally
>> created for the FD_SIZE_2MB binary (and vice versa).
> 
> I assume that gives a clear and obvious error message - right?

Unfortunately, no, it doesn't.

The constants that describe the varstore flash location are generated at
build time, and they are cooked into the OVMF binary. Varstore flash
presence is checked / verified, but the flash is not "searched for". In
addition, flash is a pretty early / low-level requirement, so no
display, no serial console etc. So the end result is, you may get a
semi-obscure error message on the QEMU debug port. :( The error message
might vary with what QEMU actually mapped under the addresses that the
OVMF binary expects to belong to the flash chip(s).

Edk2 doesn't really consider "mismatched flash" a condition that can be
handled gracefully; I guess it would be similar to soldering a bad flash
chip to the board and expecting a friendly error message on the PCI
display -- the boot will likely not get that far. ... I'm not defending
the OVMF situation, it is really raw for virt users, and it sucks. :(

Laszlo