OVMF Secure Boot variable storage issue

OVMF Secure Boot variable storage issue

[Jason Dickens]

All,

I'm trying to understand why the secure boot variables (PK, KEK, db, 
etc) when using the OVMF build are not retained across reboot? It seems 
that this code uses roughly the same SetVariable, GetVariable2 approach 
as say the PlatformConfig uses to store screen resolution (which is 
retained). Additionally, the NvVars file is being at least touched by 
the secure boot configuration. So why are none of the keys retained on 
the next reboot?

I know this was an issue in the past, but I haven't found the resolution?

Jason

Re: OVMF Secure Boot variable storage issue

[Bill Paul]

Of all the gin joints in all the towns in all the world, Jason Dickens had to 
walk into mine at 10:31:18 on Thursday 06 July 2017 and say:

> All,
> 
> I'm trying to understand why the secure boot variables (PK, KEK, db,
> etc) when using the OVMF build are not retained across reboot? It seems
> that this code uses roughly the same SetVariable, GetVariable2 approach
> as say the PlatformConfig uses to store screen resolution (which is
> retained). Additionally, the NvVars file is being at least touched by
> the secure boot configuration. So why are none of the keys retained on
> the next reboot?

If you're running OVMF in the QEMU simulator, and you're using the -bios 
option, try using the -pflash option instead.

I know that when using -bios, QEMU only pretends to allow writes to the 
firmware region, and if you stop QEMU all changes are discarded. The same 
might be true if you just trigger a hard reboot in the simulator too.

If you use -pflash instead, your changes will be saved. Note that this means 
your OVMF image will be modified, so keep a copy of the original elsewhere so 
that you can start over fresh again if you need to.

(Unfortunately I don't think OVMF has a "load factor defaults" option in its 
internal menus.)

-Bill
 
> I know this was an issue in the past, but I haven't found the resolution?
> 
> Jason
> 
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

-- 
=============================================================================
-Bill Paul            (510) 749-2329 | Senior Member of Technical Staff,
                 wpaul@windriver.com | Master of Unix-Fu - Wind River Systems
=============================================================================
   "I put a dollar in a change machine. Nothing changed." - George Carlin
=============================================================================

Re: OVMF Secure Boot variable storage issue

[Jason Dickens]

Thanks for the response Bill. If I should recognize your name, I'm 
sorry, I'm bad with names, but I have been doing a lot of work with Wind 
River recently (and in the past) so its possible I should.

Actually, I should have mentioned I'm using Xen with full 
virtualization. This means that OVMF firmware can't change without 
rebuilding Xen. For reasons I don't know, it seems the Xen build uses 
the firmware image by first converting it to a C array and then 
compiling it in.

However, I'm not sure that's the real issue. As far as I'm aware, OVMF 
implements NvVars on the VM image to provide non-volatile storage 
instead of actually modifying the image. As I mentioned, "some" 
configuration changes do persist such as changing the screen resolution 
in the OVMF settings. Also I can see that NvVars is updating its 
modification time after setting secure boot variables. What I'm trying 
to determine is if there a particular reason or implementation problem 
that causes secure boot settings not persist.



On 7/6/2017 2:30 PM, Bill Paul wrote:
> Of all the gin joints in all the towns in all the world, Jason Dickens had to
> walk into mine at 10:31:18 on Thursday 06 July 2017 and say:
>
>> All,
>>
>> I'm trying to understand why the secure boot variables (PK, KEK, db,
>> etc) when using the OVMF build are not retained across reboot? It seems
>> that this code uses roughly the same SetVariable, GetVariable2 approach
>> as say the PlatformConfig uses to store screen resolution (which is
>> retained). Additionally, the NvVars file is being at least touched by
>> the secure boot configuration. So why are none of the keys retained on
>> the next reboot?
> If you're running OVMF in the QEMU simulator, and you're using the -bios
> option, try using the -pflash option instead.
>
> I know that when using -bios, QEMU only pretends to allow writes to the
> firmware region, and if you stop QEMU all changes are discarded. The same
> might be true if you just trigger a hard reboot in the simulator too.
>
> If you use -pflash instead, your changes will be saved. Note that this means
> your OVMF image will be modified, so keep a copy of the original elsewhere so
> that you can start over fresh again if you need to.
>
> (Unfortunately I don't think OVMF has a "load factor defaults" option in its
> internal menus.)
>
> -Bill
>   
>> I know this was an issue in the past, but I haven't found the resolution?
>>
>> Jason
>>
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel

Re: OVMF Secure Boot variable storage issue

[Konrad Rzeszutek Wilk]

On Thu, Jul 06, 2017 at 02:55:27PM -0400, Jason Dickens wrote:
> Thanks for the response Bill. If I should recognize your name, I'm sorry,
> I'm bad with names, but I have been doing a lot of work with Wind River
> recently (and in the past) so its possible I should.
> 
> Actually, I should have mentioned I'm using Xen with full virtualization.
> This means that OVMF firmware can't change without rebuilding Xen. For
> reasons I don't know, it seems the Xen build uses the firmware image by
> first converting it to a C array and then compiling it in.
> 

That can be changed - it already has the mechanism to "slurp" binary
blobs from the host. The only reason it does that right now is mostly
historic - all BIOS images where compiled in 'hvmloader'.

> However, I'm not sure that's the real issue. As far as I'm aware, OVMF
> implements NvVars on the VM image to provide non-volatile storage instead of
> actually modifying the image. As I mentioned, "some" configuration changes
> do persist such as changing the screen resolution in the OVMF settings. Also
> I can see that NvVars is updating its modification time after setting secure
> boot variables. What I'm trying to determine is if there a particular reason
> or implementation problem that causes secure boot settings not persist.
> 
> 
> 
> On 7/6/2017 2:30 PM, Bill Paul wrote:
> > Of all the gin joints in all the towns in all the world, Jason Dickens had to
> > walk into mine at 10:31:18 on Thursday 06 July 2017 and say:
> > 
> > > All,
> > > 
> > > I'm trying to understand why the secure boot variables (PK, KEK, db,
> > > etc) when using the OVMF build are not retained across reboot? It seems
> > > that this code uses roughly the same SetVariable, GetVariable2 approach
> > > as say the PlatformConfig uses to store screen resolution (which is
> > > retained). Additionally, the NvVars file is being at least touched by
> > > the secure boot configuration. So why are none of the keys retained on
> > > the next reboot?
> > If you're running OVMF in the QEMU simulator, and you're using the -bios
> > option, try using the -pflash option instead.
> > 
> > I know that when using -bios, QEMU only pretends to allow writes to the
> > firmware region, and if you stop QEMU all changes are discarded. The same
> > might be true if you just trigger a hard reboot in the simulator too.
> > 
> > If you use -pflash instead, your changes will be saved. Note that this means
> > your OVMF image will be modified, so keep a copy of the original elsewhere so
> > that you can start over fresh again if you need to.
> > 
> > (Unfortunately I don't think OVMF has a "load factor defaults" option in its
> > internal menus.)
> > 
> > -Bill
> > > I know this was an issue in the past, but I haven't found the resolution?
> > > 
> > > Jason
> > > 
> > > 
> > > _______________________________________________
> > > edk2-devel mailing list
> > > edk2-devel@lists.01.org
> > > https://lists.01.org/mailman/listinfo/edk2-devel
> 
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: OVMF Secure Boot variable storage issue

[Laszlo Ersek]

On 07/06/17 20:30, Bill Paul wrote:
> Of all the gin joints in all the towns in all the world, Jason Dickens
> had to walk into mine at 10:31:18 on Thursday 06 July 2017 and say:
>
>> All,
>>
>> I'm trying to understand why the secure boot variables (PK, KEK, db,
>> etc) when using the OVMF build are not retained across reboot? It
>> seems that this code uses roughly the same SetVariable, GetVariable2
>> approach as say the PlatformConfig uses to store screen resolution
>> (which is retained). Additionally, the NvVars file is being at least
>> touched by the secure boot configuration. So why are none of the keys
>> retained on the next reboot?
>
> If you're running OVMF in the QEMU simulator, and you're using the
> -bios option, try using the -pflash option instead.
>
> I know that when using -bios, QEMU only pretends to allow writes to
> the firmware region, and if you stop QEMU all changes are discarded.
> The same might be true if you just trigger a hard reboot in the
> simulator too.
>
> If you use -pflash instead, your changes will be saved. Note that this
> means your OVMF image will be modified, so keep a copy of the original
> elsewhere so that you can start over fresh again if you need to.
>
> (Unfortunately I don't think OVMF has a "load factor defaults" option
> in its internal menus.)

Libvirt has a good pile of functionality built on top of the split image
that I described earlier. For example:

In /etc/libvirt/qemu.conf, you can provide an "nvram map", listing
several different firmware binaries, with a matching variable store
template for each.

(Different fw binaries may require different varstore formats: for
example, you can run aarch64 guests in emulation (TCG) mode, using the
ArmVirtQemu edk2 platform as your virtual firmware, and x86_64 guests in
hw virtualization (KVM) mode, using OVMF as your virtual firmware, on
the same x86_64 host. These firmwares expect a different flash layout,
so they need different varstore templates.

Another example is that you can build OVMF with 1M, 2M or 4M (unified)
image sizes, and the 4M layout uses a different (much larger) varstore
-- so the template is different for that fw binary.)

When defining a new VM, a subset of the libvirt client tools allows you
to pick a specific firmware binary. And when starting a VM that has no
varstore file under /var/lib/libvirt/qemu/nvram, libvirt will map its fw
binary to the matching varstore template, and instantiate the VM's own
varstore from the template.

As a side benefit, if you are fed up with the current contents of your
varstore, simply shut down the VM, and delete its varstore file from
under /var/lib/libvirt/qemu/nvram. When you next start the VM, libvirtd
will re-create the varstore from the pristine template.

Libvirt is an insanely powerful and great project -- I can praise it
without holding back, as I personally almost never contribute to it --,
and I do all my virt development (incl. edk2 development) exclusively
with guests that are managed by libvirt.

While the QEMU project is similarly powerful and great :) , the QEMU
command line is *not* a human interface.

(This is not to say that libvirt only composes command lines for the
VMMs that it supports -- it does a whole lot of other host management
too, such as SELinux labeling of images, setting up migration across the
network, configuring virtual networks, handling assigned physical PCI
devices, and much more.)

Thanks,
Laszlo

Re: OVMF Secure Boot variable storage issue

[Laszlo Ersek]

Hi Jason,

On 07/06/17 19:31, Jason Dickens wrote:
> All,
> 
> I'm trying to understand why the secure boot variables (PK, KEK, db,
> etc) when using the OVMF build are not retained across reboot?

They *are* retained across reboot, you just have to use a virtualization
platform that provides a working pflash (persistent flash) device model.

With regards to the variable store, you can use OVMF in three different
QEMU configurations. In chronological order (which also matches
increasing usefulness order):

(1) -bios OVMF.fd

With this option, the unified firmware image called "OVMF.fd" (which
contains both the firmware executable and the variable store) is mapped
into the guest address space as ROM.

In this case, the lowest level of the variable driver stack -- the flash
block driver -- is provided by "OvmfPkg/EmuVariableFvbRuntimeDxe", which
emulates the "flash" in a normal RAM block, pre-reserved by
"OvmfPkg/PlatformPei", at an address that does not vary across reboots.

Whenever there is a write to the flash, EmuVariableFvbRuntimeDxe does
the following things:

- updates the in-memory (fake) flash,
- calls out to Library/EmuVariableFvbLib (PlatformFvbDataWritten()),
- that function signals an event identified by PcdEmuVariableEvent
  before ExitBootServices(), and does nothing after ExitBootServices().

>From the other side, "OvmfPkg/Library/PlatformBootManagerLib", which is
built into "MdeModulePkg/Universal/BdsDxe", does the following things:

- Using OvmfPkg/Library/NvVarsFileLib -- which in turn uses
  OvmfPkg/Library/SerializeVariablesLib --, PlatformBootManagerLib
  de-serializes variable content from the \NvVars binary file from
  any EFI system partition that it finds, and restores the content to
  the UEFI variables with the SetVariable() runtime service.

- PlatformBootManagerLib also configures the event mentioned above,
  setting up PcdEmuVariableEvent. Whenever EmuVariableFvbRuntimeDxe
  signals the event (upon a flash write), BdsDxe will call out to
  NvVarsFileLib, and serialize the full set of variables (using the
  GetVariable() runtime service)) to said \NvVars file.

Problems with this approach:

- Variable updates after ExitBootServices() are only stored in memory.
  This is because at that point, the ESP is owned (mounted) by the
  operating system, so the firmware must not write to it.

- If the VM is powered down with variable updates pending in memory,
  those updates are lost.

- If the VM is rebooted instead, then the next startup of the firmware
  will sync out the memory contents to disk, with the same
  SaveNvVarsToFs() function that is used by the serialization logic
  described above.

- If a DXE driver specifies an explicit dependency on the read-only
  variable protocol (or installs a protocol notify for it), it may
  validly attempt to read variables before BDS restores variables from
  \NvVars. The result is a GetVariable() runtime service that lies to
  callers. (This issue is exacerbated when considering read-only
  variable access in the PEI phase.)

- Authenticated variables cannot be restored by way of SetVariable()
  calls issued in an unspecified order, hence the deserialization
  doesn't work for them.

- I vaguely recall that the auth variables are also lost during an in-VM
  reboot (when de-serialization from \NvVars shouldn't be necessary). I
  think by now I've forgotten the details about this. It's possible that
  writing out & reloading always happens to and from \NvVars, and that
  loses the auth variables somehow.

  The following commits are related:

  - e678f9db899a ("OvmfPkg/SerializeVariablesLib: ignore secure variable
    restore errors", 2013-05-28)
  - db827286e283 ("OvmfPkg/SerializeVariablesLib: Relax check for the
    read-only variable", 2016-05-26)

On Xen, only "-bios" is usable.

"-bios" is incompatible with -D SMM_REQUIRE (the firmware will refuse
continuing after a specific point). The primary goal of SMM is to
protect the variable store from direct hardware access by the guest OS,
and that's impossible to do if the varstore lives in a normal RAM area
and in a file on the ESP.

(2) -pflash OVMF.fd

In this case, the same unified image is mapped into guest-phys address
space as a flash device. All of the emulation described above is
inactive. A real pflash driver is provided by
OvmfPkg/QemuFlashFvbServicesRuntimeDxe (which can be built both as a
runtime DXE driver and as an SMM driver, see the INF files in that
directory).

Variables behave as required by the UEFI spec and seen on physical
hardware. Updates are written back to the unified OVMF.fd file [*],
on-line, and variable material (auth and non-auth alike) simply survives
in the low-level flash representation.

[*] This also explains one drawback of "-pflash OVMF.fd": each virtual
machine has its own copy of the firmware executable, along with its
(justifiedly private) variable store. Updating the firmware executable
without losing variables is infeasible in practice.

This method is compatible with -D SMM_REQUIRE. For full protection, some
additional command line switches are necessary, but I'll mention those
under the next point.

(3) -drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on \
    -drive if=pflash,format=raw,unit=1,file=copy_of_OVMF_VARS.fd \

To the guest, this looks mostly the same (the only guest-visible
difference is that under (2), if the guest tried to reprogram the flash
area that holds the firmware binary, it would succeed, whereas in this
case the same attempt would fail -- see "readonly=on").

The firmware image is split in two parts. OVMF_CODE.fd is mapped as
read-only flash, and can be shared by multiple VMs. OVMF_VARS.fd is only
used as a *template* for creating private variable store files that
belong to individual VMs (see the filename "copy_of_OVMF_VARS.fd").

This method has all the benefits of (2), but we can also update
OVMF_CODE.fd centrally on a virtualization host.

When using a -D SMM_REQUIRE build, the following additions are needed:

* -machine q35

  SMM emulation is only available on pc-q35-2.5 and later

* -machine smm=on

  it's best to turn on SMM emulation explicitly

* -global driver=cfi.pflash01,property=secure,value=on

  this is the important setting that causes QEMU to actually restrict
  pflash writes to guest code that runs in System Management Mode.
  (I.e., firmware. The guest OS can itself raise an SMI if it wants to,
  but then control is transferred to the firmware just the same.)

> It seems
> that this code uses roughly the same SetVariable, GetVariable2 approach
> as say the PlatformConfig uses to store screen resolution (which is
> retained). Additionally, the NvVars file is being at least touched by
> the secure boot configuration. So why are none of the keys retained on
> the next reboot?
> 
> I know this was an issue in the past, but I haven't found the resolution?

Secure Boot with "-bios OVMF.fd" has never been resolved, due to the
reasons described above:
- inherent insecurity of the emulation,
- basic UEFI-non-compliance if the VM is powered off with runtime
  variable updates pending in the memory buffer, or if a DXE driver
  tries to consume variables before BDS de-serializes them,
- de-serialization from disk, with SetVariable(), would have to conform
  to the auth variable particulars (attributes, order)

Implementing pflash in QEMU and KVM, and adding the matching driver
(QemuFlashFvbServicesRuntimeDxe) were the future-proof solution. Jordan
did those things in the QEMU-1.6 timeframe (second half of 2013, it seems).

Thanks
Laszlo