[OvmfPkg] Secure Boot issues

[OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

Hey people,

We are experiencing issues with UEFI secure boot enabled
on UDK 2018 for the OvmfPkg.

Reproducible issue:

1) Add following code + files as dxe driver.
https://gist.github.com/zaolin/976d0d2ad68bcd05c10ffdb2530341fc
2) Build OvmfPkg with -DSECURE_BOOT_ENABLE=TRUE
3) Windows 10 boots and crashes in Qemu with a
/KMODE_EXCEPTION_NOT_HANDLED./

If we don't populate the keys or use Linux in with secure boot turned on
everything is totally fine.

Any suggestions ?


Best Regards, Philipp

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/12/18 15:12, Philipp Deppenwiese wrote:
> Hey people,
> 
> We are experiencing issues with UEFI secure boot enabled
> on UDK 2018 for the OvmfPkg.

UDK2018 does not include OvmfPkg; no UDK does, to my knowledge.

> Reproducible issue:
> 
> 1) Add following code + files as dxe driver.
> https://gist.github.com/zaolin/976d0d2ad68bcd05c10ffdb2530341fc

This looks like a modified copy of (a possibly older version of) my
EnrollDefaultKeys module. The latest source for that is available from
the "edk2-20180529gitee3198e672e2-1.fc29" SRPM at
<https://koji.fedoraproject.org/koji/buildinfo?buildID=1087595>.

> 2) Build OvmfPkg with -DSECURE_BOOT_ENABLE=TRUE
> 3) Windows 10 boots and crashes in Qemu with a
> /KMODE_EXCEPTION_NOT_HANDLED./
> 
> If we don't populate the keys or use Linux in with secure boot turned on
> everything is totally fine.

Relative to the EnrollDefaultKeys.c source that I know, your variant
does not include the certificates as UINT8 arrays in the source code;
instead it seems to include them in firmware filesystem (FFS) files, and
to look them up with GetSectionFromAnyFv(). I assume you have some INF
file changes as well, where you build the certificates as binary blobs
into DXEFV.

Did you verify that the exact same blobs (and same other arguments) are
passed to the gRT->SetVariable() calls in your variant?

I've now retested my variant with Windows 10 Enterprise N 2015 LTSB; it
works as expected.

Thanks,
Laszlo

Re: [OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

Hey Laszlo,


On 12.06.2018 15:59, Laszlo Ersek wrote:
> On 06/12/18 15:12, Philipp Deppenwiese wrote:
>> Hey people,
>>
>> We are experiencing issues with UEFI secure boot enabled
>> on UDK 2018 for the OvmfPkg.
> UDK2018 does not include OvmfPkg; no UDK does, to my knowledge.
I mean the UDK2018 branch of https://github.com/tianocore/edk2/tree/UDK2018
>
>> Reproducible issue:
>>
>> 1) Add following code + files as dxe driver.
>> https://gist.github.com/zaolin/976d0d2ad68bcd05c10ffdb2530341fc
> This looks like a modified copy of (a possibly older version of) my
> EnrollDefaultKeys module. The latest source for that is available from
> the "edk2-20180529gitee3198e672e2-1.fc29" SRPM at
> <https://koji.fedoraproject.org/koji/buildinfo?buildID=1087595>.
Correct, I just moved it into a DXE driver and load certificates from
the FFS.
Do you know if there is a more common/normal/better way for populating
vendor certificates?
>
>> 2) Build OvmfPkg with -DSECURE_BOOT_ENABLE=TRUE
>> 3) Windows 10 boots and crashes in Qemu with a
>> /KMODE_EXCEPTION_NOT_HANDLED./
>>
>> If we don't populate the keys or use Linux in with secure boot turned on
>> everything is totally fine.
> Relative to the EnrollDefaultKeys.c source that I know, your variant
> does not include the certificates as UINT8 arrays in the source code;
> instead it seems to include them in firmware filesystem (FFS) files, and
> to look them up with GetSectionFromAnyFv(). I assume you have some INF
> file changes as well, where you build the certificates as binary blobs
> into DXEFV.
>
> Did you verify that the exact same blobs (and same other arguments) are
> passed to the gRT->SetVariable() calls in your variant?
>
> I've now retested my variant with Windows 10 Enterprise N 2015 LTSB; it
> works as expected.
Thanks for the help. I am going to check if I do something wrong. But I
have ran
the FWTS testsuite and checked the certificates with the mokutil under
Linux, everything
looks fine so far. Also Windows 10 in safe mode with secure boot works
but not the normal mode.

We use the 14393.0.160715-1616.RS1_RELEASE_CLIENTENTERPRISE_S_EVAL_X64
LTSB release for testing.
>
> Thanks,
> Laszlo

Best Regards, Philipp

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/12/18 16:51, Philipp Deppenwiese wrote:

> Also Windows 10 in safe mode with secure boot works but not the
> normal mode.
> 
> We use the
> 14393.0.160715-1616.RS1_RELEASE_CLIENTENTERPRISE_S_EVAL_X64 LTSB
> release for testing.

Interesting, this reminds me of the "new" driver signing requirements
when Secure Boot is enabled. Something something about cross-signed
drivers not being accepted by recent Windows 10 when SB is enabled.

I could imagine that some of your native guest drivers (paravirt /
virtio drivers) aren't "appropriately signed" (whatever that may mean),
and then something crashes when windows *rejects* loading those drivers.

In safe mode, I could imagine Windows doesn't even attempt to load those
drivers.

Really I'm just speculating here. In support of the speculation:

https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html

"""
Fedora VirtIO Drivers vs. RHEL VirtIO Drivers

The RPMs in the virtio-win-stable repository are the same driver builds
as what is shipped with Red Hat Enterprise Linux. All the Windows
binaries are from builds done on Red Hat’s internal build system, which
are generated using publicly available code. For more details about how
the RPM and repo are built, see the README for this repo.

The drivers are cryptographically signed with Red Hat’s vendor
signature. However they are not signed with Microsoft’s WHQL signature.
"""

In addition, please see
<https://bugzilla.redhat.com/show_bug.cgi?id=1376048#c0>.

Thanks
Laszlo

Re: [OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

Hey Laszlo,

We are using VirtualBox as virtualization solution and
don't load guest drivers. But we had the same issue with
the current Qemu version as well.

Can you try to test your setup with the latest Windows 10 LTSB ?
That would help us to understand if that's a general EDK2
issue or just our problem.


Best Regards, Philipp

On 12.06.2018 20:14, Laszlo Ersek wrote:
> On 06/12/18 16:51, Philipp Deppenwiese wrote:
>
>> Also Windows 10 in safe mode with secure boot works but not the
>> normal mode.
>>
>> We use the
>> 14393.0.160715-1616.RS1_RELEASE_CLIENTENTERPRISE_S_EVAL_X64 LTSB
>> release for testing.
> Interesting, this reminds me of the "new" driver signing requirements
> when Secure Boot is enabled. Something something about cross-signed
> drivers not being accepted by recent Windows 10 when SB is enabled.
>
> I could imagine that some of your native guest drivers (paravirt /
> virtio drivers) aren't "appropriately signed" (whatever that may mean),
> and then something crashes when windows *rejects* loading those drivers.
>
> In safe mode, I could imagine Windows doesn't even attempt to load those
> drivers.
>
> Really I'm just speculating here. In support of the speculation:
>
> https://docs.fedoraproject.org/quick-docs/en-US/creating-windows-virtual-machines-using-virtio-drivers.html
>
> """
> Fedora VirtIO Drivers vs. RHEL VirtIO Drivers
>
> The RPMs in the virtio-win-stable repository are the same driver builds
> as what is shipped with Red Hat Enterprise Linux. All the Windows
> binaries are from builds done on Red Hat’s internal build system, which
> are generated using publicly available code. For more details about how
> the RPM and repo are built, see the README for this repo.
>
> The drivers are cryptographically signed with Red Hat’s vendor
> signature. However they are not signed with Microsoft’s WHQL signature.
> """
>
> In addition, please see
> <https://bugzilla.redhat.com/show_bug.cgi?id=1376048#c0>.
>
> Thanks
> Laszlo

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/13/18 15:45, Philipp Deppenwiese wrote:
> Hey Laszlo,
> 
> We are using VirtualBox as virtualization solution and
> don't load guest drivers. But we had the same issue with
> the current Qemu version as well.
> 
> Can you try to test your setup with the latest Windows 10 LTSB ?
> That would help us to understand if that's a general EDK2
> issue or just our problem.

My testing yesterday was supposed to cover the "latest in Windows 10
LTSB" -- I had indeed installed the OS earlier from a Windows 10
Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
pulled down all pending updates.

Is that not what you mean? Can you give me an ISO filename and a SHA1
checksum then? I could try looking it up in MSDN. (It's not guaranteed
that my subscription will allow me access to it though.)

Thanks
Laszlo

Re: [OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

Hey Laszlo,

It's free for trial and available under:

https://www.microsoft.com/de-de/evalcenter/evaluate-windows-10-enterprise


Best Regards, Philipp

On 13.06.2018 21:21, Laszlo Ersek wrote:
> On 06/13/18 15:45, Philipp Deppenwiese wrote:
>> Hey Laszlo,
>>
>> We are using VirtualBox as virtualization solution and
>> don't load guest drivers. But we had the same issue with
>> the current Qemu version as well.
>>
>> Can you try to test your setup with the latest Windows 10 LTSB ?
>> That would help us to understand if that's a general EDK2
>> issue or just our problem.
> My testing yesterday was supposed to cover the "latest in Windows 10
> LTSB" -- I had indeed installed the OS earlier from a Windows 10
> Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
> pulled down all pending updates.
>
> Is that not what you mean? Can you give me an ISO filename and a SHA1
> checksum then? I could try looking it up in MSDN. (It's not guaranteed
> that my subscription will allow me access to it though.)
>
> Thanks
> Laszlo

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/13/18 21:41, Philipp Deppenwiese wrote:
> Hey Laszlo,
> 
> It's free for trial and available under:
> 
> https://www.microsoft.com/de-de/evalcenter/evaluate-windows-10-enterprise

I tried the following ISO:

af9a46ddd2a88ea01e9d3a52f56cf48e6e9d989e5a35f6e88d68be48dccfcb8d
14393.0.160715-1616.rs1_release_cliententerprise_s_eval_x64fre_en-us.iso

I simply took my Windows 10 libvirt domain that I used yesterday for
testing, and booted it off of the above ISO, rather than the virtual
hard disk. I didn't go through with the entire installation, just until
it asked for the virtio-win driver CD (so that it could continue reading
the virtio-scsi Windows installer ISO, post-EFI), at which point I
forced off the VM. Nonetheless, until that point, everything seemed to
work fine. Did you encounter the crash after that point, or before it?

Thanks
Laszlo

> On 13.06.2018 21:21, Laszlo Ersek wrote:
>> On 06/13/18 15:45, Philipp Deppenwiese wrote:
>>> Hey Laszlo,
>>>
>>> We are using VirtualBox as virtualization solution and
>>> don't load guest drivers. But we had the same issue with
>>> the current Qemu version as well.
>>>
>>> Can you try to test your setup with the latest Windows 10 LTSB ?
>>> That would help us to understand if that's a general EDK2
>>> issue or just our problem.
>> My testing yesterday was supposed to cover the "latest in Windows 10
>> LTSB" -- I had indeed installed the OS earlier from a Windows 10
>> Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
>> pulled down all pending updates.
>>
>> Is that not what you mean? Can you give me an ISO filename and a SHA1
>> checksum then? I could try looking it up in MSDN. (It's not guaranteed
>> that my subscription will allow me access to it though.)
>>
>> Thanks
>> Laszlo
> 

Re: [OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

We are getting the error after the Windows 10 installation.

Windows boots up for few seconds and then crashes.


On 13.06.2018 23:18, Laszlo Ersek wrote:
> On 06/13/18 21:41, Philipp Deppenwiese wrote:
>> Hey Laszlo,
>>
>> It's free for trial and available under:
>>
>> https://www.microsoft.com/de-de/evalcenter/evaluate-windows-10-enterprise
> I tried the following ISO:
>
> af9a46ddd2a88ea01e9d3a52f56cf48e6e9d989e5a35f6e88d68be48dccfcb8d
> 14393.0.160715-1616.rs1_release_cliententerprise_s_eval_x64fre_en-us.iso
>
> I simply took my Windows 10 libvirt domain that I used yesterday for
> testing, and booted it off of the above ISO, rather than the virtual
> hard disk. I didn't go through with the entire installation, just until
> it asked for the virtio-win driver CD (so that it could continue reading
> the virtio-scsi Windows installer ISO, post-EFI), at which point I
> forced off the VM. Nonetheless, until that point, everything seemed to
> work fine. Did you encounter the crash after that point, or before it?
>
> Thanks
> Laszlo
>
>> On 13.06.2018 21:21, Laszlo Ersek wrote:
>>> On 06/13/18 15:45, Philipp Deppenwiese wrote:
>>>> Hey Laszlo,
>>>>
>>>> We are using VirtualBox as virtualization solution and
>>>> don't load guest drivers. But we had the same issue with
>>>> the current Qemu version as well.
>>>>
>>>> Can you try to test your setup with the latest Windows 10 LTSB ?
>>>> That would help us to understand if that's a general EDK2
>>>> issue or just our problem.
>>> My testing yesterday was supposed to cover the "latest in Windows 10
>>> LTSB" -- I had indeed installed the OS earlier from a Windows 10
>>> Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
>>> pulled down all pending updates.
>>>
>>> Is that not what you mean? Can you give me an ISO filename and a SHA1
>>> checksum then? I could try looking it up in MSDN. (It's not guaranteed
>>> that my subscription will allow me access to it though.)
>>>
>>> Thanks
>>> Laszlo

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/13/18 23:25, Philipp Deppenwiese wrote:
> We are getting the error after the Windows 10 installation.
> 
> Windows boots up for few seconds and then crashes.

I've installed a brand new libvirt domain from the installer ISO
discussed below, reusing the virtual hardware config from my previous
Windows 10 domain -- after enabling SB with EnrollDefaultKeys.efi as
very first step.

I've installed a bunch of virtio drivers in the guest. As I mentioned
earlier, Windows would permit the installation of those drivers from the
Fedora virtio-win ISO, but it would also block those drivers from being
launched:

"Windows cannot verify the digital signature for the drivers required
for this device. A recent hardware or software change might have
installed a file that is signed incorrectly or damaged, or that might be
malicious software from an unknown source. (Code 52)"

Using the WHQL'd RHEL variants, the virtio drivers work fine. I also
installed Firefox in the guest, Cygwin (for having an openssh server),
and the QEMU guest agent.

I haven't experienced any issues with the guest, beyond the usual
"Windows Update hogs resources beyond recognition" (once I installed the
virtio-net driver).

Thanks,
Laszlo

> On 13.06.2018 23:18, Laszlo Ersek wrote:
>> On 06/13/18 21:41, Philipp Deppenwiese wrote:
>>> Hey Laszlo,
>>>
>>> It's free for trial and available under:
>>>
>>> https://www.microsoft.com/de-de/evalcenter/evaluate-windows-10-enterprise
>> I tried the following ISO:
>>
>> af9a46ddd2a88ea01e9d3a52f56cf48e6e9d989e5a35f6e88d68be48dccfcb8d
>> 14393.0.160715-1616.rs1_release_cliententerprise_s_eval_x64fre_en-us.iso
>>
>> I simply took my Windows 10 libvirt domain that I used yesterday for
>> testing, and booted it off of the above ISO, rather than the virtual
>> hard disk. I didn't go through with the entire installation, just until
>> it asked for the virtio-win driver CD (so that it could continue reading
>> the virtio-scsi Windows installer ISO, post-EFI), at which point I
>> forced off the VM. Nonetheless, until that point, everything seemed to
>> work fine. Did you encounter the crash after that point, or before it?
>>
>> Thanks
>> Laszlo
>>
>>> On 13.06.2018 21:21, Laszlo Ersek wrote:
>>>> On 06/13/18 15:45, Philipp Deppenwiese wrote:
>>>>> Hey Laszlo,
>>>>>
>>>>> We are using VirtualBox as virtualization solution and
>>>>> don't load guest drivers. But we had the same issue with
>>>>> the current Qemu version as well.
>>>>>
>>>>> Can you try to test your setup with the latest Windows 10 LTSB ?
>>>>> That would help us to understand if that's a general EDK2
>>>>> issue or just our problem.
>>>> My testing yesterday was supposed to cover the "latest in Windows 10
>>>> LTSB" -- I had indeed installed the OS earlier from a Windows 10
>>>> Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
>>>> pulled down all pending updates.
>>>>
>>>> Is that not what you mean? Can you give me an ISO filename and a SHA1
>>>> checksum then? I could try looking it up in MSDN. (It's not guaranteed
>>>> that my subscription will allow me access to it though.)
>>>>
>>>> Thanks
>>>> Laszlo
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
> 

Re: [OvmfPkg] Secure Boot issues

[Philipp Deppenwiese]

Hey Laszlo,

We guess that's an issue on our side because of some
NVRAM issues with VirtualBox. We located it through Windows 10
debugging.

Thanks a lot for your support and testing!

Maybe you are interested to join the www.osfc.io
and talk about some OVMF details. :)


Best Regards, Philipp

On 14.06.2018 19:29, Laszlo Ersek wrote:
> On 06/13/18 23:25, Philipp Deppenwiese wrote:
>> We are getting the error after the Windows 10 installation.
>>
>> Windows boots up for few seconds and then crashes.
> I've installed a brand new libvirt domain from the installer ISO
> discussed below, reusing the virtual hardware config from my previous
> Windows 10 domain -- after enabling SB with EnrollDefaultKeys.efi as
> very first step.
>
> I've installed a bunch of virtio drivers in the guest. As I mentioned
> earlier, Windows would permit the installation of those drivers from the
> Fedora virtio-win ISO, but it would also block those drivers from being
> launched:
>
> "Windows cannot verify the digital signature for the drivers required
> for this device. A recent hardware or software change might have
> installed a file that is signed incorrectly or damaged, or that might be
> malicious software from an unknown source. (Code 52)"
>
> Using the WHQL'd RHEL variants, the virtio drivers work fine. I also
> installed Firefox in the guest, Cygwin (for having an openssh server),
> and the QEMU guest agent.
>
> I haven't experienced any issues with the guest, beyond the usual
> "Windows Update hogs resources beyond recognition" (once I installed the
> virtio-net driver).
>
> Thanks,
> Laszlo
>
>> On 13.06.2018 23:18, Laszlo Ersek wrote:
>>> On 06/13/18 21:41, Philipp Deppenwiese wrote:
>>>> Hey Laszlo,
>>>>
>>>> It's free for trial and available under:
>>>>
>>>> https://www.microsoft.com/de-de/evalcenter/evaluate-windows-10-enterprise
>>> I tried the following ISO:
>>>
>>> af9a46ddd2a88ea01e9d3a52f56cf48e6e9d989e5a35f6e88d68be48dccfcb8d
>>> 14393.0.160715-1616.rs1_release_cliententerprise_s_eval_x64fre_en-us.iso
>>>
>>> I simply took my Windows 10 libvirt domain that I used yesterday for
>>> testing, and booted it off of the above ISO, rather than the virtual
>>> hard disk. I didn't go through with the entire installation, just until
>>> it asked for the virtio-win driver CD (so that it could continue reading
>>> the virtio-scsi Windows installer ISO, post-EFI), at which point I
>>> forced off the VM. Nonetheless, until that point, everything seemed to
>>> work fine. Did you encounter the crash after that point, or before it?
>>>
>>> Thanks
>>> Laszlo
>>>
>>>> On 13.06.2018 21:21, Laszlo Ersek wrote:
>>>>> On 06/13/18 15:45, Philipp Deppenwiese wrote:
>>>>>> Hey Laszlo,
>>>>>>
>>>>>> We are using VirtualBox as virtualization solution and
>>>>>> don't load guest drivers. But we had the same issue with
>>>>>> the current Qemu version as well.
>>>>>>
>>>>>> Can you try to test your setup with the latest Windows 10 LTSB ?
>>>>>> That would help us to understand if that's a general EDK2
>>>>>> issue or just our problem.
>>>>> My testing yesterday was supposed to cover the "latest in Windows 10
>>>>> LTSB" -- I had indeed installed the OS earlier from a Windows 10
>>>>> Enterprise N 2015 LTSB ISO, but yesterday that long-term guest of mine
>>>>> pulled down all pending updates.
>>>>>
>>>>> Is that not what you mean? Can you give me an ISO filename and a SHA1
>>>>> checksum then? I could try looking it up in MSDN. (It's not guaranteed
>>>>> that my subscription will allow me access to it though.)
>>>>>
>>>>> Thanks
>>>>> Laszlo
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>>

Re: [OvmfPkg] Secure Boot issues

[Laszlo Ersek]

On 06/18/18 14:14, Philipp Deppenwiese wrote:
> Hey Laszlo,
> 
> We guess that's an issue on our side because of some
> NVRAM issues with VirtualBox. We located it through Windows 10
> debugging.

Good job! :)

> Thanks a lot for your support and testing!
> 
> Maybe you are interested to join the www.osfc.io
> and talk about some OVMF details. :)

Due to family circumstances, I travel once per year at most, and that's
usually to the KVM Forum (whenever I can manage).

If a conference is brought to Budapest, Hungary, there's a much better
chance I can drop by on a daily basis. (Linaro Connect 2017 was like
this, and it was fantastic.)


I don't do public speaking, as a rule. At the KVM Fora, I attempt to
participate in BoF / round-table discussions about OVMF. These have been
very small historically, and on the scale that's still approachable to me.

Thanks!
Laszlo