OVMF.fd and placement of EfiBootServicesData

OVMF.fd and placement of EfiBootServicesData

[spam collector]

Hello, 
Forgive me for not searching for this first, but gmane.org has 
been down for a little while. Therefore, forgive me if this 
question has been asked and answered. 
I am using OVMF.fd and QEMU for Windows and tried to load 
a large file to physical address 0x00800000. When an error 
was returned, I found that OVMF has reserved an amount of 
EfiBootServicesData within/around that location. 
Here is a memory dump using the BootServices MemoryMap Service: 
Start: 0x00000000->0x0009FFFF, Pgs: 160, EfiConventionalMem 
Start: 0x00100000->0x003FFFFF, Pgs: 768, EfiConventionalMem 
Start: 0x00400000->0x0040BFFF, Pgs: 12, EfiBootServicesCode 
Start: 0x0040C000->0x0081FFFF, Pgs: 1044, EfiConventionalMem 
Start: 0x00820000->0x00FFFFFF, Pgs: 2016, EfiBootServicesData 
Start: 0x01000000->0x0BFFFFFF, Pgs: 1044, EfiConventionalMem 
... and so on 
Without the idea of "you should make your code relocatable, 
i.e.: not care where it is in memory", or if I say that I *must* have 
the memory from 0x00800000 to 0x00FFFFFFavailable, 
without re-building OVMF.fd, is there a way to tell the system 
to use a different address for that portion of EfiBootServiceCode? 
I tried manipulating NvVars with a few entries to no avail. 
I added the "LoadFixedAddressConfigurationTable" entry hoping 
that I could set a minimum location, too without success. 
So, without rebuilding OVMF.fd, which would require everyone 
whom wished to use my code to do so, or at least download it 
from me, and I don't like modified packages of other's work 
floating around, is there a why to tell OVMF.fd/the EFI system 
to not use memory below a certain address? 
Thank you in advance for your suggestions, 
Ben 
-- 

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/03/16 05:16, spam collector wrote:
> Hello, 
> Forgive me for not searching for this first, but gmane.org has 
> been down for a little while. Therefore, forgive me if this 
> question has been asked and answered. 
> I am using OVMF.fd and QEMU for Windows and tried to load 
> a large file to physical address 0x00800000. When an error 
> was returned, I found that OVMF has reserved an amount of 
> EfiBootServicesData within/around that location. 
> Here is a memory dump using the BootServices MemoryMap Service: 
> Start: 0x00000000->0x0009FFFF, Pgs: 160, EfiConventionalMem 
> Start: 0x00100000->0x003FFFFF, Pgs: 768, EfiConventionalMem 
> Start: 0x00400000->0x0040BFFF, Pgs: 12, EfiBootServicesCode 
> Start: 0x0040C000->0x0081FFFF, Pgs: 1044, EfiConventionalMem 
> Start: 0x00820000->0x00FFFFFF, Pgs: 2016, EfiBootServicesData 
> Start: 0x01000000->0x0BFFFFFF, Pgs: 1044, EfiConventionalMem 
> ... and so on 
> Without the idea of "you should make your code relocatable, 
> i.e.: not care where it is in memory", or if I say that I *must* have 
> the memory from 0x00800000 to 0x00FFFFFFavailable, 
> without re-building OVMF.fd, is there a way to tell the system 
> to use a different address for that portion of EfiBootServiceCode? 
> I tried manipulating NvVars with a few entries to no avail. 
> I added the "LoadFixedAddressConfigurationTable" entry hoping 
> that I could set a minimum location, too without success. 
> So, without rebuilding OVMF.fd, which would require everyone 
> whom wished to use my code to do so, or at least download it 
> from me, and I don't like modified packages of other's work 
> floating around, is there a why to tell OVMF.fd/the EFI system 
> to not use memory below a certain address? 
> Thank you in advance for your suggestions, 

Sorry, I don't think I can help.

You can read about the OVMF memory map in the (now somewhat outdated)
OVMF whitepaper at

  http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt

section

  A comprehensive memory map of OVMF

You'd like to place your stuff at 8192 KB .. 16384 KB, but as you can
see from the map, that exact range is heavily used by OVMF itself.

(As I said, some of those exact values are no longer current, due to the
following commits since:

  08df58ec3043 OvmfPkg: raise DXEFV size to 9 MB
  2f7b34b20842 OvmfPkg: raise DXEFV size to 10 MB
  45d870815156 OvmfPkg/PlatformPei: rebase and resize the permanent PEI
               memory for S3
  6b04cca4d697 OvmfPkg: remove PcdS3AcpiReservedMemoryBase,
               PcdS3AcpiReservedMemorySize

but the main layout remains the same.)

Please consider making your code relocatable.

Thanks
Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Monday, October 3, 2016 2:42:08 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> On 10/03/16 05:16, spam collector wrote:
> > Hello,
> > Forgive me for not searching for this first, but gmane.org has
> > been down for a little while. Therefore, forgive me if this
> > question has been asked and answered.
> > I am using OVMF.fd and QEMU for Windows and tried to load
> > a large file to physical address 0x00800000. When an error
> > was returned, I found that OVMF has reserved an amount of
> > EfiBootServicesData within/around that location.
> > Here is a memory dump using the BootServices MemoryMap Service:
> > Start: 0x00000000->0x0009FFFF, Pgs: 160, EfiConventionalMem
> > Start: 0x00100000->0x003FFFFF, Pgs: 768, EfiConventionalMem
> > Start: 0x00400000->0x0040BFFF, Pgs: 12, EfiBootServicesCode
> > Start: 0x0040C000->0x0081FFFF, Pgs: 1044, EfiConventionalMem
> > Start: 0x00820000->0x00FFFFFF, Pgs: 2016, EfiBootServicesData
> > Start: 0x01000000->0x0BFFFFFF, Pgs: 1044, EfiConventionalMem
> > ... and so on
> > Without the idea of "you should make your code relocatable,
> > i.e.: not care where it is in memory", or if I say that I *must* have
> > the memory from 0x00800000 to 0x00FFFFFFavailable,
> > without re-building OVMF.fd, is there a way to tell the system
> > to use a different address for that portion of EfiBootServiceCode?
> > I tried manipulating NvVars with a few entries to no avail.
> > I added the "LoadFixedAddressConfigurationTable" entry hoping
> > that I could set a minimum location, too without success.
> > So, without rebuilding OVMF.fd, which would require everyone
> > whom wished to use my code to do so, or at least download it
> > from me, and I don't like modified packages of other's work
> > floating around, is there a why to tell OVMF.fd/the EFI system
> > to not use memory below a certain address?
> > Thank you in advance for your suggestions,
> 
> Sorry, I don't think I can help.
> 
> You can read about the OVMF memory map in the (now somewhat outdated)
> OVMF whitepaper at
> 
>   http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt
> 
> section
> 
>   A comprehensive memory map of OVMF
> 
> You'd like to place your stuff at 8192 KB .. 16384 KB, but as you can
> see from the map, that exact range is heavily used by OVMF itself.
> 
> (As I said, some of those exact values are no longer current, due to the
> following commits since:
> 
>   08df58ec3043 OvmfPkg: raise DXEFV size to 9 MB
>   2f7b34b20842 OvmfPkg: raise DXEFV size to 10 MB
>   45d870815156 OvmfPkg/PlatformPei: rebase and resize the permanent PEI
>                memory for S3
>   6b04cca4d697 OvmfPkg: remove PcdS3AcpiReservedMemoryBase,
>                PcdS3AcpiReservedMemorySize
> 
> but the main layout remains the same.)
> 
> Please consider making your code relocatable.

Without sounding rude, and please do not take it as so, making my
code relocatable due to the fact that the other code is not relocatable,
sounds a little hypocritical. :-)

Anyway, please don't take offense to that, it is just what first
came to mind when I read it.  Eventually I will make my code relocatable
but at the moment that would be quite a re-write of many other parts
of this project, and I am not ready or willing to do that yet. Yet...

May I please suggest to the maintainers of this project to add a function
that will allow a NvVars entry to indicate where the base of this 
"comprehensive memory map of OVMF" starts.  Allow an entry within the 
NvVars file to "suggest" a base address and let the code accommodate 
as close as it can.  Or even just a flag that states to use higher
area memory when set, or lower area memory when clear.  Some OS writers
like to use high memory and leave low memory available while others
like to use low memory and leave the remaining higher memory available.

Just a humble suggestion. Thank you,
Ben

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/04/16 04:45, spam collector wrote:
> 
> 
> ----- Original Message -----
>> From: "Laszlo Ersek" <lersek@redhat.com>
>> To: "spam collector" <spamcollector@cableone.net>
>> Cc: edk2-devel@ml01.01.org
>> Sent: Monday, October 3, 2016 2:42:08 AM
>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>>
>> On 10/03/16 05:16, spam collector wrote:
>>> Hello,
>>> Forgive me for not searching for this first, but gmane.org has
>>> been down for a little while. Therefore, forgive me if this
>>> question has been asked and answered.
>>> I am using OVMF.fd and QEMU for Windows and tried to load
>>> a large file to physical address 0x00800000. When an error
>>> was returned, I found that OVMF has reserved an amount of
>>> EfiBootServicesData within/around that location.
>>> Here is a memory dump using the BootServices MemoryMap Service:
>>> Start: 0x00000000->0x0009FFFF, Pgs: 160, EfiConventionalMem
>>> Start: 0x00100000->0x003FFFFF, Pgs: 768, EfiConventionalMem
>>> Start: 0x00400000->0x0040BFFF, Pgs: 12, EfiBootServicesCode
>>> Start: 0x0040C000->0x0081FFFF, Pgs: 1044, EfiConventionalMem
>>> Start: 0x00820000->0x00FFFFFF, Pgs: 2016, EfiBootServicesData
>>> Start: 0x01000000->0x0BFFFFFF, Pgs: 1044, EfiConventionalMem
>>> ... and so on
>>> Without the idea of "you should make your code relocatable,
>>> i.e.: not care where it is in memory", or if I say that I *must* have
>>> the memory from 0x00800000 to 0x00FFFFFFavailable,
>>> without re-building OVMF.fd, is there a way to tell the system
>>> to use a different address for that portion of EfiBootServiceCode?
>>> I tried manipulating NvVars with a few entries to no avail.
>>> I added the "LoadFixedAddressConfigurationTable" entry hoping
>>> that I could set a minimum location, too without success.
>>> So, without rebuilding OVMF.fd, which would require everyone
>>> whom wished to use my code to do so, or at least download it
>>> from me, and I don't like modified packages of other's work
>>> floating around, is there a why to tell OVMF.fd/the EFI system
>>> to not use memory below a certain address?
>>> Thank you in advance for your suggestions,
>>
>> Sorry, I don't think I can help.
>>
>> You can read about the OVMF memory map in the (now somewhat outdated)
>> OVMF whitepaper at
>>
>>   http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt
>>
>> section
>>
>>   A comprehensive memory map of OVMF
>>
>> You'd like to place your stuff at 8192 KB .. 16384 KB, but as you can
>> see from the map, that exact range is heavily used by OVMF itself.
>>
>> (As I said, some of those exact values are no longer current, due to the
>> following commits since:
>>
>>   08df58ec3043 OvmfPkg: raise DXEFV size to 9 MB
>>   2f7b34b20842 OvmfPkg: raise DXEFV size to 10 MB
>>   45d870815156 OvmfPkg/PlatformPei: rebase and resize the permanent PEI
>>                memory for S3
>>   6b04cca4d697 OvmfPkg: remove PcdS3AcpiReservedMemoryBase,
>>                PcdS3AcpiReservedMemorySize
>>
>> but the main layout remains the same.)
>>
>> Please consider making your code relocatable.
> 
> Without sounding rude, and please do not take it as so, making my
> code relocatable due to the fact that the other code is not relocatable,
> sounds a little hypocritical. :-)

Not taking it as rude, but you are wrong about this. OVMF is a platform
firmware, including SEC and PEI phase modules. Such modules are fully
expected to operate with a fixed address map.

In comparison your module is (I assume) a UEFI_DRIVER /
UEFI_APPLICATION, or maybe a DXE_DRIVER.
- If it's a UEFI_DRIVER or UEFI_APPLICATION, then it is expected to work
without hardcoded addresses.
- If it is a DXE_DRIVER, then it might be okay for it to operate with
fixed addresses. However, a DXE_DRIVER module is meant to be shipped by
the platform vendor. For that, your module would have to be integrated
with OVMF -- it should either be part of the public edk2 tree (or your
edk2 tree), or you should be providing a prebuilt OVMF binary that
contained it.

> Anyway, please don't take offense to that, it is just what first
> came to mind when I read it.  Eventually I will make my code relocatable
> but at the moment that would be quite a re-write of many other parts
> of this project, and I am not ready or willing to do that yet. Yet...
> 
> May I please suggest to the maintainers of this project to add a function
> that will allow a NvVars entry to indicate where the base of this 
> "comprehensive memory map of OVMF" starts.  Allow an entry within the 
> NvVars file to "suggest" a base address and let the code accommodate 
> as close as it can.

(1) You mention the NvVars file as storage for non-volatile variables.
If you are using OVMF with QEMU, I strongly suggest to abandon this
setup, and adopt the pflash based variable store instead. The behavior
of the NvVars file is not fully compatible with the UEFI spec, and it
had been invented as a stop-gap solution until real variable storage
would be implemented.

(2) The addresses that you seem interested in are build-time constants,
and even reset vector and SEC phase code depend on them. In those
phases, variable services don't exist.

If you want to move OVMF's fixed memory map to a different spot, for
your own builds, please grep OvmfPkg for the PCDs that are listed to the
right of the table that is in section "A comprehensive memory map of
OVMF" of the whitepaper, and customize them as you see fit. (And then
rebuild OVMF, of course.)

We have sought to make this as painless as possible -- for our own
sanity's sake too --; that is, most of those PCDs are already being
computed from other (more basic) PCDs in the same set, either at build
time or at runtime. It's just that the "root" PCDs have to be build-time
constants. For example, please refer to the MEMFD_BASE_ADDRESS macro in
"OvmfPkg.fdf.inc" and elsewhere.

> Or even just a flag that states to use higher
> area memory when set, or lower area memory when clear.  Some OS writers
> like to use high memory and leave low memory available while others
> like to use low memory and leave the remaining higher memory available.

If you want to move OVMF's fixed areas to different places, specific
addresses are necessary. Please refer to the PCDs mentioned above.

The current map works with at least Linux guests, Windows guests, OSX
guests (I've heard from users), and FreeBSD guests (based on their
Wiki). I'm quite sure the address constraints currently encoded in your
module (a boot loader perhaps?) could break on suitable physical
firmware as well; OVMF does nothing invalid in this regard.

Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Tuesday, October 4, 2016 12:52:00 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> On 10/04/16 04:45, spam collector wrote:
> > 
> > 
> > ----- Original Message -----
> >> From: "Laszlo Ersek" <lersek@redhat.com>
> >> To: "spam collector" <spamcollector@cableone.net>
> >> Cc: edk2-devel@ml01.01.org
> >> Sent: Monday, October 3, 2016 2:42:08 AM
> >> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> >>
> >> On 10/03/16 05:16, spam collector wrote:
> >>> Hello,
> >>> Forgive me for not searching for this first, but gmane.org has
> >>> been down for a little while. Therefore, forgive me if this
> >>> question has been asked and answered.
> >>> I am using OVMF.fd and QEMU for Windows and tried to load
> >>> a large file to physical address 0x00800000. When an error
> >>> was returned, I found that OVMF has reserved an amount of
> >>> EfiBootServicesData within/around that location.
> >>> Here is a memory dump using the BootServices MemoryMap Service:
> >>> Start: 0x00000000->0x0009FFFF, Pgs: 160, EfiConventionalMem
> >>> Start: 0x00100000->0x003FFFFF, Pgs: 768, EfiConventionalMem
> >>> Start: 0x00400000->0x0040BFFF, Pgs: 12, EfiBootServicesCode
> >>> Start: 0x0040C000->0x0081FFFF, Pgs: 1044, EfiConventionalMem
> >>> Start: 0x00820000->0x00FFFFFF, Pgs: 2016, EfiBootServicesData
> >>> Start: 0x01000000->0x0BFFFFFF, Pgs: 1044, EfiConventionalMem
> >>> ... and so on
> >>> Without the idea of "you should make your code relocatable,
> >>> i.e.: not care where it is in memory", or if I say that I *must* have
> >>> the memory from 0x00800000 to 0x00FFFFFFavailable,
> >>> without re-building OVMF.fd, is there a way to tell the system
> >>> to use a different address for that portion of EfiBootServiceCode?
> >>> I tried manipulating NvVars with a few entries to no avail.
> >>> I added the "LoadFixedAddressConfigurationTable" entry hoping
> >>> that I could set a minimum location, too without success.
> >>> So, without rebuilding OVMF.fd, which would require everyone
> >>> whom wished to use my code to do so, or at least download it
> >>> from me, and I don't like modified packages of other's work
> >>> floating around, is there a why to tell OVMF.fd/the EFI system
> >>> to not use memory below a certain address?
> >>> Thank you in advance for your suggestions,
> >>
> >> Sorry, I don't think I can help.
> >>
> >> You can read about the OVMF memory map in the (now somewhat outdated)
> >> OVMF whitepaper at
> >>
> >>   http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt
> >>
> >> section
> >>
> >>   A comprehensive memory map of OVMF
> >>
> >> You'd like to place your stuff at 8192 KB .. 16384 KB, but as you can
> >> see from the map, that exact range is heavily used by OVMF itself.
> >>
> >> (As I said, some of those exact values are no longer current, due to the
> >> following commits since:
> >>
> >>   08df58ec3043 OvmfPkg: raise DXEFV size to 9 MB
> >>   2f7b34b20842 OvmfPkg: raise DXEFV size to 10 MB
> >>   45d870815156 OvmfPkg/PlatformPei: rebase and resize the permanent PEI
> >>                memory for S3
> >>   6b04cca4d697 OvmfPkg: remove PcdS3AcpiReservedMemoryBase,
> >>                PcdS3AcpiReservedMemorySize
> >>
> >> but the main layout remains the same.)
> >>
> >> Please consider making your code relocatable.
> > 
> > Without sounding rude, and please do not take it as so, making my
> > code relocatable due to the fact that the other code is not relocatable,
> > sounds a little hypocritical. :-)
> 
> Not taking it as rude, but you are wrong about this. OVMF is a platform
> firmware, including SEC and PEI phase modules. Such modules are fully
> expected to operate with a fixed address map.
> 
> In comparison your module is (I assume) a UEFI_DRIVER /
> UEFI_APPLICATION, or maybe a DXE_DRIVER.
> - If it's a UEFI_DRIVER or UEFI_APPLICATION, then it is expected to work
> without hardcoded addresses.
> - If it is a DXE_DRIVER, then it might be okay for it to operate with
> fixed addresses. However, a DXE_DRIVER module is meant to be shipped by
> the platform vendor. For that, your module would have to be integrated
> with OVMF -- it should either be part of the public edk2 tree (or your
> edk2 tree), or you should be providing a prebuilt OVMF binary that
> contained it.
> 
> > Anyway, please don't take offense to that, it is just what first
> > came to mind when I read it.  Eventually I will make my code relocatable
> > but at the moment that would be quite a re-write of many other parts
> > of this project, and I am not ready or willing to do that yet. Yet...
> > 
> > May I please suggest to the maintainers of this project to add a function
> > that will allow a NvVars entry to indicate where the base of this
> > "comprehensive memory map of OVMF" starts.  Allow an entry within the
> > NvVars file to "suggest" a base address and let the code accommodate
> > as close as it can.
> 
> (1) You mention the NvVars file as storage for non-volatile variables.
> If you are using OVMF with QEMU, I strongly suggest to abandon this
> setup, and adopt the pflash based variable store instead. The behavior
> of the NvVars file is not fully compatible with the UEFI spec, and it
> had been invented as a stop-gap solution until real variable storage
> would be implemented.
> 
> (2) The addresses that you seem interested in are build-time constants,
> and even reset vector and SEC phase code depend on them. In those
> phases, variable services don't exist.
> 
> If you want to move OVMF's fixed memory map to a different spot, for
> your own builds, please grep OvmfPkg for the PCDs that are listed to the
> right of the table that is in section "A comprehensive memory map of
> OVMF" of the whitepaper, and customize them as you see fit. (And then
> rebuild OVMF, of course.)
> 
> We have sought to make this as painless as possible -- for our own
> sanity's sake too --; that is, most of those PCDs are already being
> computed from other (more basic) PCDs in the same set, either at build
> time or at runtime. It's just that the "root" PCDs have to be build-time
> constants. For example, please refer to the MEMFD_BASE_ADDRESS macro in
> "OvmfPkg.fdf.inc" and elsewhere.
> 
> > Or even just a flag that states to use higher
> > area memory when set, or lower area memory when clear.  Some OS writers
> > like to use high memory and leave low memory available while others
> > like to use low memory and leave the remaining higher memory available.
> 
> If you want to move OVMF's fixed areas to different places, specific
> addresses are necessary. Please refer to the PCDs mentioned above.
> 
> The current map works with at least Linux guests, Windows guests, OSX
> guests (I've heard from users), and FreeBSD guests (based on their
> Wiki). I'm quite sure the address constraints currently encoded in your
> module (a boot loader perhaps?) could break on suitable physical
> firmware as well; OVMF does nothing invalid in this regard.
> 
> Laszlo

Thank you for your comments and suggestions.  I have glanced over
the pflash storage a little, but not fully investigated it at all.

I appreciate the work and effort of those whom have created OVMF and
admire their commitment.

Again, thank you for your comments.  Maybe it is time to spend the
effort and time to make my code relocatable.

On a slightly different note, I was surprised to see that QEMU and/or
OVMF requires the NvVars file to exist on the partition, else OVMF
does not even show the shell, even on a perfectly valid partitioned
image.  I don't know if this is QEMU's fault or OVMF's fault.  If 
the NvVars file does not exist, is it the desired option to fail 
the loading of the shell? If the file does not exist, can OVMF use 
default values and still load the shell?

I ask because I spent a little while thinking that my GPT partitioned
image was in error since I could not get a shell to load.  Once I
added the NvVars file to the root directory, the shell loaded as
expected.  If NvVars is not compatible with UEFI, which I don't
believe it is, should OVMF still require it to boot to the shell?
Maybe if I study more on the pflash option and use it, OVMF will
not require the NvVars file.

I ask with the intent of better understanding the reason it
requires the NvVars file, as opposed to being sarcastic as it may
sound. :-)

Thank you once again,
Ben 

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/04/16 18:39, spam collector wrote:

> [...] I was surprised to see that QEMU and/or
> OVMF requires the NvVars file to exist on the partition, else OVMF
> does not even show the shell, even on a perfectly valid partitioned
> image.  I don't know if this is QEMU's fault or OVMF's fault.  If 
> the NvVars file does not exist, is it the desired option to fail 
> the loading of the shell? If the file does not exist, can OVMF use 
> default values and still load the shell?

OVMF does not require the NvVars file to exist. The NvVars file is only
used / relied on when the QEMU command line is incorrect, that is, when
it does not configure the pflash chip(s) for storing the firmware binary
and/or the variable store.

* The command line that you are likely used to goes like this:

  qemu-system-x86_64 -bios OVMF.fd ...

This is broken. Don't use it. The OVMF.fd file in the above usage is a
unified file (1MB or 2MB in size), containing both the firmware binary
and an empty (pre-formatted) variable store. The entire file is mapped
into guest memory as ROM. The variable store area is not writeable,
hence OVMF falls back to the NvVars based emulation. That emulation is
not entirely compatible with the UEFI spec. Don't use the above command
line.

* The first stage improvement is the following command line:

  qemu-system-x86_64 -pflash OVMF.fd

The OVMF.fd file has the same characteristics as above, but the same
stuff is mapped as a programmable pflash chip into guest memory. It
means that the guest memory area in question *normally* reads and
executes like normal RAM, but if you write to it (that is, if OVMF's
flash driver writes to it), it is flipped into "programming mode", where
you can read and write the device using various "commands". One of those
commands flips the device back to "normal" mode.

In this mode, the NvVars-based emulation is not used, and the variable
store / variable services will work as defined by the UEFI spec.

The problem with this approach is that the OVMF.fd file contains both a
firmware binary and a live variable store. You cannot upgrade the
firmware binary without losing the VM's long-term, non-volatile variables.

* The second stage improvement is the following command line:

  # create the virtual machine's private variable store from the
  # pristine variable store template, if the former doesn't exist yet,
  # or has been lost for some reason
  if ! [ -e GUEST_1_VARS.fd ]; then
    cp OVMF_VARS.fd GUEST_1_VARS.fd
  fi

  qemu-system-x86_64 \
    -drive if=pflash,format=raw,unit=0,readonly,file=OVMF_CODE.fd \
    -drive if=pflash,format=raw,unit=1,file=GUEST_1_VARS.fd \
    ...

In this case, the OVMF_CODE.fd and OVMF_VARS.fd are used separately.
(You can find both files under the Build directory; plus Gerd Hoffmann's
RPM files <https://www.kraxel.org/repos/> also package them.)

The OVMF_CODE.fd file is the read-only firmware binary (1920KB in size),
whereas the OVMF_VARS.fd file is an empty, preformatted variable store
*template* (128KB in size). The VM never uses OVMF_VARS.fd directly, you
(or the management layer, see below) uses that file only to instantiate
VM-specific varstores.

This pattern allows you to upgrade your firmware binary without messing
up each VM's own variable store.

The issue with this method is that a malicious guest OS can still poke
at the pflash chip (with unit=1) directly. If you have Secure Boot
enabled, you don't want the guest OS to tamper with the authenticated
UEFI variables in the varstore that contain certificates and such.

* The third stage improvement is SMM. For that, build OVMF like this
(I'm simplifying a lot, but this email should come to an end sometime):

  build -a IA32 -a X64 -p OvmfPkg/OvmfPkgIa32X64.dsc \
    -D SMM_REQUIRE -D SECURE_BOOT_ENABLE \
    ...

and run OVMF like this:

  # create the virtual machine's private variable store from the
  # pristine variable store template, if the former doesn't exist yet,
  # or has been lost for some reason
  if ! [ -e GUEST_1_VARS.fd ]; then
    cp OVMF_VARS.fd GUEST_1_VARS.fd
  fi

  qemu-system-x86_64 \
    -machine q35,smm=on \
    -global driver=cfi.pflash01,property=secure,value=on \
    -drive if=pflash,format=raw,unit=0,readonly,file=OVMF_CODE.fd \
    -drive if=pflash,format=raw,unit=1,file=GUEST_1_VARS.fd \

In this case, the guest OS won't be able to write to the variable store
directly. It will have to invoke the runtime UEFI variable services,
which will raise an SMI / enter SMM. The SMI handler (part of the
firmware) will validate / authenticate the request parameters, and
perform the modifications if they are valid.

* The fourth stage improvement is to use libvirt, which will handle all
of the above for you, assuming you give it the right domain XML.
(Virt-manager / virt-install can create domain XMLs that need no or
minimal further customizations.)

* In some builds / distributions of OVMF, the UEFI shell is excluded
from the firmware image, and it is provided on a small ISO image. This
is justified by two things:

(a) first, this way you can use QEMU's

  -device virtio-scsi-pci,id=scsi0 \
  -drive if=none,format=raw,readonly,file=UefiShell.iso \
  -device scsi-hd,bus=scsi0.0,bootindex=N \

options to boot the UEFI shell directly -- bootindex is not usable for
modules built into the firmware binary; QEMU's "bootorder" fw_cfg
doesn't have enough expressive power for that --,

(b) second, we've been advised that the UEFI shell can be used to
circumvent Secure Boot. By moving the shell binary out of the firmware
image, to a CD-ROM, as an external *unsigned* EFI application, it cannot
be launched when Secure Boot is enabled.

* Should your installed guest (such as a UEFI GNU/Linux distro, or a
UEFI Windows edition) lose the variable store (for example because you
unwittingly delete GUEST_1_VARS.fd), the UEFI spec defines a method for
the (guest) OS to recreate a sane set of boot options / boot order,
after you (or libvirt) recreate GUEST_1_VARS.fd from OVMF_VARS.fd again.
For Linux at least, the keyword here is "fallback.efi"; you can read
about it at
<http://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/>.

> 
> I ask because I spent a little while thinking that my GPT partitioned
> image was in error since I could not get a shell to load.  Once I
> added the NvVars file to the root directory, the shell loaded as
> expected.  If NvVars is not compatible with UEFI, which I don't
> believe it is,

(which you are correct about,)

> should OVMF still require it to boot to the shell?

No, and it doesn't. Use one of the above pflash-based methods please.

Since we're into boot option land anyway, I should mention the following
section of the OVMF whitepaper:

  Select features |
    Platform-specific boot policy

If you never use the "bootindex=N" QEMU device property (or libvirt's
equivalent <boot order='N'/> element), then OVMF will not touch your
UEFI boot options (Boot####, BootOrder), beyond regenerating options for
all possible bootable devices at the end of your current BootOrder.

However, if you want to influence the UEFI boot order from the QEMU
command line (or the libvirt domain XML), you can. There are a few
caveats; please read the above whitepaper section for understanding them.

> Maybe if I study more on the pflash option and use it, OVMF will
> not require the NvVars file.

Exactly.

> 
> I ask with the intent of better understanding the reason it
> requires the NvVars file, as opposed to being sarcastic as it may
> sound. :-)

It doesn't sound sarcastic, don't worry. ;)

NvVars dates back to a time when QEMU didn't have pflash yet, and
(consequently) OVMF didn't have a driver for QEMU's pflash. Now that
pflash can be used, it (and only it) should be used too.

OVMF also runs on Xen. Xen doesn't provide pflash emulation, so the
NvVars fallback is still in active use on Xen.

Thanks
Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Tuesday, October 4, 2016 10:22:34 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> On 10/04/16 18:39, spam collector wrote:
> 
> > [...] I was surprised to see that QEMU and/or
> > OVMF requires the NvVars file to exist on the partition, else OVMF
> > does not even show the shell, even on a perfectly valid partitioned
> > image.  I don't know if this is QEMU's fault or OVMF's fault.  If
> > the NvVars file does not exist, is it the desired option to fail
> > the loading of the shell? If the file does not exist, can OVMF use
> > default values and still load the shell?
> 
> OVMF does not require the NvVars file to exist. The NvVars file is only
> used / relied on when the QEMU command line is incorrect, that is, when
> it does not configure the pflash chip(s) for storing the firmware binary
> and/or the variable store.
> 
> * The command line that you are likely used to goes like this:
> 
>   qemu-system-x86_64 -bios OVMF.fd ...

Yes.  I am currently using that line with the only modification of 
the 32-bit emulation on both parts.
 
> This is broken. Don't use it. The OVMF.fd file in the above usage is a
> unified file (1MB or 2MB in size), containing both the firmware binary
> and an empty (pre-formatted) variable store. The entire file is mapped
> into guest memory as ROM. The variable store area is not writeable,
> hence OVMF falls back to the NvVars based emulation. That emulation is
> not entirely compatible with the UEFI spec. Don't use the above command
> line.
> 
> * The first stage improvement is the following command line:
> 
>   qemu-system-x86_64 -pflash OVMF.fd

This did not work either with or without the NvVars file present.
 
> The OVMF.fd file has the same characteristics as above, but the same
> stuff is mapped as a programmable pflash chip into guest memory. It
> means that the guest memory area in question *normally* reads and
> executes like normal RAM, but if you write to it (that is, if OVMF's
> flash driver writes to it), it is flipped into "programming mode", where
> you can read and write the device using various "commands". One of those
> commands flips the device back to "normal" mode.
> 
> In this mode, the NvVars-based emulation is not used, and the variable
> store / variable services will work as defined by the UEFI spec.
> 
> The problem with this approach is that the OVMF.fd file contains both a
> firmware binary and a live variable store. You cannot upgrade the
> firmware binary without losing the VM's long-term, non-volatile variables.
> 
> * The second stage improvement is the following command line:
> 
>   # create the virtual machine's private variable store from the
>   # pristine variable store template, if the former doesn't exist yet,
>   # or has been lost for some reason
>   if ! [ -e GUEST_1_VARS.fd ]; then
>     cp OVMF_VARS.fd GUEST_1_VARS.fd
>   fi
> 
>   qemu-system-x86_64 \
>     -drive if=pflash,format=raw,unit=0,readonly,file=OVMF_CODE.fd \
>     -drive if=pflash,format=raw,unit=1,file=GUEST_1_VARS.fd \
>     ...
> 
> In this case, the OVMF_CODE.fd and OVMF_VARS.fd are used separately.
> (You can find both files under the Build directory; plus Gerd Hoffmann's
> RPM files <https://www.kraxel.org/repos/> also package them.)

For the life of me I could not find any directory called "Build".  I
did find a few RPM files and one of them contained:
  OVMF_CODE-pure-efi.fd
 and
  OVMF_VARS-pure-efi.fd
However, no matter the emulation (32- or 64-bit), neither worked and
would get as far as printing the '.' just before the 
  .FLOPPY failed
  .CD-ROM failed
  etc.
lines and go no further.
 
Thank you for the excellent description, however sorry for my
ignorance. If you don't mind, would you please specify a specific
URL that contains the two files you speak of, whether it be a listing
of those files, a .gz file, or even a .rpm file.

Thank you,
Ben

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/05/16 04:47, spam collector wrote:
> 
> ----- Original Message -----
>> From: "Laszlo Ersek" <lersek@redhat.com>
>> To: "spam collector" <spamcollector@cableone.net>
>> Cc: edk2-devel@ml01.01.org
>> Sent: Tuesday, October 4, 2016 10:22:34 AM
>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData

>> * The first stage improvement is the following command line:
>>
>>   qemu-system-x86_64 -pflash OVMF.fd
> 
> This did not work either with or without the NvVars file present.

I think I'll need a more complete issue riport then -- what is your
exact QEMU command line, and what OVMF binary are you using?

>> * The second stage improvement is the following command line:
>>
>>   # create the virtual machine's private variable store from the
>>   # pristine variable store template, if the former doesn't exist yet,
>>   # or has been lost for some reason
>>   if ! [ -e GUEST_1_VARS.fd ]; then
>>     cp OVMF_VARS.fd GUEST_1_VARS.fd
>>   fi
>>
>>   qemu-system-x86_64 \
>>     -drive if=pflash,format=raw,unit=0,readonly,file=OVMF_CODE.fd \
>>     -drive if=pflash,format=raw,unit=1,file=GUEST_1_VARS.fd \
>>     ...
>>
>> In this case, the OVMF_CODE.fd and OVMF_VARS.fd are used separately.
>> (You can find both files under the Build directory; plus Gerd Hoffmann's
>> RPM files <https://www.kraxel.org/repos/> also package them.)
> 
> For the life of me I could not find any directory called "Build".

The Build directory exists in the source edk2 tree, if you build the
firmware yourself.

>  I did find a few RPM files and one of them contained:
>   OVMF_CODE-pure-efi.fd
>  and
>   OVMF_VARS-pure-efi.fd

Yes, they are from Gerd's "edk2.git-ovmf-ia32" package (since you
mention IA32), from <http://www.kraxel.org/repos/>, and they are
recommended to use.

> However, no matter the emulation (32- or 64-bit), neither worked and
> would get as far as printing the '.' just before the 
>   .FLOPPY failed
>   .CD-ROM failed
>   etc.
> lines and go no further.

These messages mean the following: you are running QEMU in a default
configuration, with a default NIC, a default floppy, a default CD-ROM,
and so on. OVMF automatically generates boot options for these (in some
unspecified order), and then tries to boot those options. Specifically
the dots with the long delay mean that OVMF is trying to PXE boot from
your virtual NIC.

There are two ways around this:

- First, if you have a dedicated device you want to boot off of, use
  the "bootindex" property I mentioned earlier.
  - Specifically, if you want to boot the UEFI shell like this, then
    use "/usr/share/edk2.git/ovmf-ia32/UefiShell.iso" with an ide-cd or
    scsi-cd device.

- Second, you can pass "-nodefaults" to QEMU. This option will prevent
  the automatic creation of the NIC, the floppy, the CD-ROM -- and a
  lot more actually, such as keyboard, mouse, VGA too --; in turn OVMF
  should not create boot options for these devices either, and should
  drop you directly to the shell.

  Assuming of course that your firmware binary includes the shell in
  the first place. If it doesn't, then you can only use "UefiShell.iso".

I recommend trying the following (32-bit command line), with Gerd's package:

  FW_BIN=/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd
  VARS_TMPL=/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd
  VARS=myvars.fd
  SHELL_ISO=/usr/share/edk2.git/ovmf-ia32/UefiShell.iso
  if ! [ -e "$VARS" ]; then
    cp -v -- "$VARS_TMPL" "$VARS"
  fi

  qemu-system-i386 \
    -m 2048 \
    -machine accel=kvm \
    \
    -drive if=pflash,format=raw,unit=0,readonly,file="$FW_BIN" \
    -drive if=pflash,format=raw,unit=1,file="$VARS" \
    \
    -device virtio-scsi-pci,id=scsi0 \
    -drive if=none,format=raw,readonly,file="$SHELL_ISO",id=shell \
    -device scsi-cd,bus=scsi0.0,drive=shell,bootindex=1 \
    \
    -chardev file,id=debugfile,path=ovmf.log \
    -device isa-debugcon,iobase=0x402,chardev=debugfile

This will boot the shell for you, and even save the OVMF debug log in
the file "ovmf.log".

> would you please specify a specific URL that contains the two files
> you speak of, whether it be a listing of those files, a .gz file, or
> even a .rpm file.

If you GNU/Linux distro of choice is RPM-based, simply follow the
instrutions at <https://www.kraxel.org/repos/>.

Otherwise, download the "edk2.git-ovmf-ia32" or "edk2.git-ovmf-x64" RPM
file (as needed) from <https://www.kraxel.org/repos/jenkins/edk2/>, and
extract it with:

  rpm2cpio FILENAME | pax -r -v

The extracted files you want are:

  usr/share/edk2.git/ovmf-*/OVMF_CODE-pure-efi.fd
  usr/share/edk2.git/ovmf-*/OVMF_VARS-pure-efi.fd
  usr/share/edk2.git/ovmf-*/UefiShell.iso

Thanks
Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Wednesday, October 5, 2016 8:28:46 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> On 10/05/16 04:47, spam collector wrote:
> > 
> > ----- Original Message -----
> >> From: "Laszlo Ersek" <lersek@redhat.com>
> >> To: "spam collector" <spamcollector@cableone.net>
> >> Cc: edk2-devel@ml01.01.org
> >> Sent: Tuesday, October 4, 2016 10:22:34 AM
> >> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> >> * The first stage improvement is the following command line:
> >>
> >>   qemu-system-x86_64 -pflash OVMF.fd
> > 
> > This did not work either with or without the NvVars file present.
> 
> I think I'll need a more complete issue riport then -- what is your
> exact QEMU command line, and what OVMF binary are you using?

"..\qemu-system-i386w.exe" -m 256 -localtime 
  -drive file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0 
  -pflash ..\OVMF.fd -parallel file:para.txt

Remember that I am running this in WinXPSP3.  Also, the version of
OVMF.fd is from http://www.tianocore.org/ovmf/ and I just noticed
that that page has been recently changed.  You use to be able to 
download the OVMF.fd directly from that page.  It was r15214, which 
I found out yesterday, is an older version.

> >  I did find a few RPM files and one of them contained:
> >   OVMF_CODE-pure-efi.fd
> >  and
> >   OVMF_VARS-pure-efi.fd
> 
> Yes, they are from Gerd's "edk2.git-ovmf-ia32" package (since you
> mention IA32), from <http://www.kraxel.org/repos/>, and they are
> recommended to use.

I also could not get these to work.  I will do a little more 
research into why not and be sure to let you know.

> Otherwise, download the "edk2.git-ovmf-ia32" or "edk2.git-ovmf-x64" RPM
> file (as needed) from <https://www.kraxel.org/repos/jenkins/edk2/>, and
> extract it with:
> 
>   rpm2cpio FILENAME | pax -r -v
> 
> The extracted files you want are:
> 
>   usr/share/edk2.git/ovmf-*/OVMF_CODE-pure-efi.fd
>   usr/share/edk2.git/ovmf-*/OVMF_VARS-pure-efi.fd
>   usr/share/edk2.git/ovmf-*/UefiShell.iso

These are the three files I used and could not get the shell to load.
I will try a few other things and get back with you.

Please note that I am using the Windows port of QEMU from 
https://qemu.weilnetz.de/ on a WinXP host, if that makes any 
difference.

Also, one more question.  I can make my GPT image either have a legacy
MBR with two entries pointing to one partition each or I can have a
Protected MBR that encompasses the whole GPT, both partitions, and the
backup items.

With the r15214 version and:

"..\qemu-system-i386w.exe" -m 256 -localtime 
  -drive file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0 
  -bios ..\OVMF.fd -parallel file:para.txt

it would boot to the shell either way and load and run my BOOT.efi file
just fine, whether I had a Legacy MBR or a PMBR.  Does this make a
difference with the new image(s) you mention above.

Also, please note that the hard drive image is a VHD image, a
raw image with a single sector at the end for the VHD info so that
Oracles VM Box will boot it.  This means that the backup GPT header
is *not* the last sector of the disk.  The

  -bios OVMF.fd

version boots it just fine in QEMU, but I thought I would mention
this in case the new code expects it to be in the last sector.

Thank you.  I will get back with you on my findings.

Ben

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/06/16 05:05, spam collector wrote:
> ----- Original Message -----
>> From: "Laszlo Ersek" <lersek@redhat.com>
>> To: "spam collector" <spamcollector@cableone.net>
>> Cc: edk2-devel@ml01.01.org
>> Sent: Wednesday, October 5, 2016 8:28:46 AM
>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>>
>> On 10/05/16 04:47, spam collector wrote:
>>>
>>> ----- Original Message -----
>>>> From: "Laszlo Ersek" <lersek@redhat.com>
>>>> To: "spam collector" <spamcollector@cableone.net>
>>>> Cc: edk2-devel@ml01.01.org
>>>> Sent: Tuesday, October 4, 2016 10:22:34 AM
>>>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>>
>>>> * The first stage improvement is the following command line:
>>>>
>>>>   qemu-system-x86_64 -pflash OVMF.fd
>>>
>>> This did not work either with or without the NvVars file present.
>>
>> I think I'll need a more complete issue riport then -- what is your
>> exact QEMU command line, and what OVMF binary are you using?
> 
> "..\qemu-system-i386w.exe" -m 256 -localtime 
>   -drive file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0 
>   -pflash ..\OVMF.fd -parallel file:para.txt
> 
> Remember that I am running this in WinXPSP3.

First, I can't "remember" it, because this is the first time you state
that. :)

Second, WinXPSP3 as host machine, really?... :)

Anyway, if both your QEMU and OVMF builds are okay / fresh enough, using
OVMF with TCG (i.e., not KVM) should work, yes.

> Also, the version of
> OVMF.fd is from http://www.tianocore.org/ovmf/ and I just noticed
> that that page has been recently changed.

That's a coincidence, but not a random one. Users have been repeatedly
confused by the r15214 binary, which at this point was more than 2.5
years old. Just before your query, I noticed
<https://bugs.launchpad.net/qemu/+bug/1622582> (and marked it invalid);
the reporter of that issue was also tripped up by the ancient r15214
build. So I asked Jordan to please update the page you reference, with a
link to Gerd's RPMs, which are bleeding edge builds.

> You use to be able to 
> download the OVMF.fd directly from that page.  It was r15214, which 
> I found out yesterday, is an older version.
> 
>>>  I did find a few RPM files and one of them contained:
>>>   OVMF_CODE-pure-efi.fd
>>>  and
>>>   OVMF_VARS-pure-efi.fd
>>
>> Yes, they are from Gerd's "edk2.git-ovmf-ia32" package (since you
>> mention IA32), from <http://www.kraxel.org/repos/>, and they are
>> recommended to use.
> 
> I also could not get these to work.  I will do a little more 
> research into why not and be sure to let you know.
> 
>> Otherwise, download the "edk2.git-ovmf-ia32" or "edk2.git-ovmf-x64" RPM
>> file (as needed) from <https://www.kraxel.org/repos/jenkins/edk2/>, and
>> extract it with:
>>
>>   rpm2cpio FILENAME | pax -r -v
>>
>> The extracted files you want are:
>>
>>   usr/share/edk2.git/ovmf-*/OVMF_CODE-pure-efi.fd
>>   usr/share/edk2.git/ovmf-*/OVMF_VARS-pure-efi.fd
>>   usr/share/edk2.git/ovmf-*/UefiShell.iso
> 
> These are the three files I used and could not get the shell to load.
> I will try a few other things and get back with you.
> 
> Please note that I am using the Windows port of QEMU from 
> https://qemu.weilnetz.de/ on a WinXP host, if that makes any 
> difference.

It could make a difference. I've never ever used Windows binaries of
QEMU. OTOH several people on this list have used such with success, so I
can't say definitely either way.

> 
> Also, one more question.  I can make my GPT image either have a legacy
> MBR with two entries pointing to one partition each or I can have a
> Protected MBR that encompasses the whole GPT, both partitions, and the
> backup items.
> 
> With the r15214 version and:
> 
> "..\qemu-system-i386w.exe" -m 256 -localtime 
>   -drive file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0 
>   -bios ..\OVMF.fd -parallel file:para.txt
> 
> it would boot to the shell either way and load and run my BOOT.efi file
> just fine, whether I had a Legacy MBR or a PMBR.  Does this make a
> difference with the new image(s) you mention above.

Sorry, I don't have any particular expertise with GPT. I believe the
PartitionDxe driver that's built into OVMF received some updates over
that 2.5y interval that I mentioned above.

> Also, please note that the hard drive image is a VHD image, a
> raw image with a single sector at the end for the VHD info so that
> Oracles VM Box will boot it.  This means that the backup GPT header
> is *not* the last sector of the disk.

As far as I know, QEMU supports VHDX. However, you pass a VHD image to
-drive with format=raw. That looks very broken. I guess I'd suggest
matching the actual image format with the format=... property. If you
pass format=raw, the VHD metadata at the end of the image will be
visible to the guest (as garbage).

>  The
> 
>   -bios OVMF.fd
> 
> version boots it just fine in QEMU, but I thought I would mention
> this in case the new code expects it to be in the last sector.

I'm quite certain all GPT drivers in the guest expect a valid disk
image, and format=raw, on a VHD disk image, seems to break that.

Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Thursday, October 6, 2016 12:39:39 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
> 
> > Remember that I am running this in WinXPSP3.
> 
> First, I can't "remember" it, because this is the first time you state
> that. :)

I apologize for that.  I thought I had mentioned it.
 
> Second, WinXPSP3 as host machine, really?... :)

What do you mean, "really?"  In my opinion, WinXP was the last usable, 
worth having OS from Microsoft.  As soon as they went to the platform
they have now, they lost all hope of ever getting anything back.
I haven't had to reboot this WinXP machine due to the OS for years.
It is less resource hungry, and much faster than the modern computers
I have around.  I would rather sit at this 10 year old machine running
WinXP any day of the week including Sunday over any of the more modern
Windows machine I have or seen.  Just my opinion though.  I, by no means,
intend to start a war over which Windows OS is better, nor do I want
to start a war over, "You should move to a Linux based machine."
If it isn't broke, I am not fixing it.

For that matter, I am not expecting a reply to that. :-)
 
> > Also, the version of
> > OVMF.fd is from http://www.tianocore.org/ovmf/ and I just noticed
> > that that page has been recently changed.
> 
> That's a coincidence, but not a random one. Users have been repeatedly
> confused by the r15214 binary, which at this point was more than 2.5
> years old. Just before your query, I noticed
> <https://bugs.launchpad.net/qemu/+bug/1622582> (and marked it invalid);
> the reporter of that issue was also tripped up by the ancient r15214
> build. So I asked Jordan to please update the page you reference, with a
> link to Gerd's RPMs, which are bleeding edge builds.

Noted.

> >>
> >>   usr/share/edk2.git/ovmf-*/OVMF_CODE-pure-efi.fd
> >>   usr/share/edk2.git/ovmf-*/OVMF_VARS-pure-efi.fd
> >>   usr/share/edk2.git/ovmf-*/UefiShell.iso

As per another message in this thread, I changed the Subsystem value 
from the 0x0B to 0x0A and it now boots as expected.  There is still
OVMF stuff in the way :-), but boots as expected to that point.
 
> > Also, please note that the hard drive image is a VHD image, a
> > raw image with a single sector at the end for the VHD info so that
> > Oracles VM Box will boot it.  This means that the backup GPT header
> > is *not* the last sector of the disk.
> 
> As far as I know, QEMU supports VHDX. However, you pass a VHD image to
> -drive with format=raw. That looks very broken. I guess I'd suggest
> matching the actual image format with the format=... property. If you
> pass format=raw, the VHD metadata at the end of the image will be
> visible to the guest (as garbage).

The latest version of the Windows port of QEMU and OVMF as listed
above, boot and treat the drive as expected, ignoring the last sector.
i.e.: OVMF uses the value in the GPT header to find the backup instead
of finding the last sector.  It doesn't complain that it isn't the
last sector.

This image is just a test image that I can use with both QEMU and
Oracle's VMBox so I don't have to have two separate images.

I believe I have all of the information and the new/correct files
and can boot my code with these new files.

Thank you so much for your help and advice.  I appreciate the
effort you placed on my behalf.  

Thanks,
Ben

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/07/16 05:11, spam collector wrote:
> ----- Original Message -----
>> From: "Laszlo Ersek" <lersek@redhat.com>
>> To: "spam collector" <spamcollector@cableone.net>
>> Cc: edk2-devel@ml01.01.org
>> Sent: Thursday, October 6, 2016 12:39:39 AM
>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>>
>>> Remember that I am running this in WinXPSP3.
>>
>> First, I can't "remember" it, because this is the first time you state
>> that. :)
> 
> I apologize for that.  I thought I had mentioned it.
>  
>> Second, WinXPSP3 as host machine, really?... :)
> 
> What do you mean, "really?" [...]
> If it isn't broke, I am not fixing it.
> 
> For that matter, I am not expecting a reply to that. :-)

No, I can explain it:

https://en.wikipedia.org/wiki/Windows_XP#End_of_support

    Windows XP is remotely exploitable by numerous security holes that
    were discovered after Microsoft stopped supporting Windows XP.
    [...]

> I believe I have all of the information and the new/correct files
> and can boot my code with these new files.

Great!

> Thank you so much for your help and advice.  I appreciate the
> effort you placed on my behalf.  

My pleasure :)

Cheers
Laszlo

Re: OVMF.fd and placement of EfiBootServicesData

[spam collector]

----- Original Message -----
> From: "Laszlo Ersek" <lersek@redhat.com>
> To: "spam collector" <spamcollector@cableone.net>
> Cc: edk2-devel@ml01.01.org
> Sent: Wednesday, October 5, 2016 8:28:46 AM
> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>
> I recommend trying the following (32-bit command line), with Gerd's package:
> 
>   FW_BIN=/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd
>   VARS_TMPL=/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd
>   VARS=myvars.fd
>   SHELL_ISO=/usr/share/edk2.git/ovmf-ia32/UefiShell.iso
>   if ! [ -e "$VARS" ]; then
>     cp -v -- "$VARS_TMPL" "$VARS"
>   fi
> 
>   qemu-system-i386 \
>     -m 2048 \
>     -machine accel=kvm \
>     \
>     -drive if=pflash,format=raw,unit=0,readonly,file="$FW_BIN" \
>     -drive if=pflash,format=raw,unit=1,file="$VARS" \
>     \
>     -device virtio-scsi-pci,id=scsi0 \
>     -drive if=none,format=raw,readonly,file="$SHELL_ISO",id=shell \
>     -device scsi-cd,bus=scsi0.0,drive=shell,bootindex=1 \
>     \
>     -chardev file,id=debugfile,path=ovmf.log \
>     -device isa-debugcon,iobase=0x402,chardev=debugfile
> 
> This will boot the shell for you, and even save the OVMF debug log in
> the file "ovmf.log".

I used the following command line (again, a WinXP DOS Session .BAT file):

"..\qemu-system-i386w.exe" -m 256 -drive if=pflash,format=raw,unit=0,readonly,file=..\OVMF_CODE.fd -drive 
if=pflash,format=raw,unit=1,file=..\OVMF_VARS.fd -device virtio-scsi-pci,id=scsi0 -drive 
if=none,format=raw,readonly,file=..\UefiShell.iso,id=shell -device scsi-cd,bus=scsi0.0,drive=shell,bootindex=1 
-chardev file,id=debugfile,path=ovmf.log -device isa-debugcon,iobase=0x402,chardev=debugfile -drive 
file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0

The shell booted up as expected.  However, my BOOT.efi file is now a:

  "image is not an application"

I don't think I mentioned before, my BOOT.efi is an OS loader, not 
a UEFI driver or other type application.  Sorry, I should have mentioned
that earlier, hope that did not confuse you.

My BOOT.efi file has a value of 0x0B as the Subsystem value.

   PE Optional Header: Subsystem: 0x000B

I removed the UefiShell.iso part:

"..\qemu-system-i386w.exe" -m 256 -drive if=pflash,format=raw,unit=0,readonly,file=..\OVMF_CODE.fd -drive 
if=pflash,format=raw,unit=1,file=..\OVMF_VARS.fd -device virtio-scsi-pci,id=scsi0 -chardev 
file,id=debugfile,path=ovmf.log -device isa-debugcon,iobase=0x402,chardev=debugfile -drive 
file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0

and the shell was loaded, with the same result, "not an application".

Anyway, thanks for your help.  I will figure this out, I am sure of it :-)

Ben

Re: OVMF.fd and placement of EfiBootServicesData

[Laszlo Ersek]

On 10/06/16 07:45, spam collector wrote:
> ----- Original Message -----
>> From: "Laszlo Ersek" <lersek@redhat.com>
>> To: "spam collector" <spamcollector@cableone.net>
>> Cc: edk2-devel@ml01.01.org
>> Sent: Wednesday, October 5, 2016 8:28:46 AM
>> Subject: Re: [edk2] OVMF.fd and placement of EfiBootServicesData
>>
>> I recommend trying the following (32-bit command line), with Gerd's package:
>>
>>   FW_BIN=/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd
>>   VARS_TMPL=/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd
>>   VARS=myvars.fd
>>   SHELL_ISO=/usr/share/edk2.git/ovmf-ia32/UefiShell.iso
>>   if ! [ -e "$VARS" ]; then
>>     cp -v -- "$VARS_TMPL" "$VARS"
>>   fi
>>
>>   qemu-system-i386 \
>>     -m 2048 \
>>     -machine accel=kvm \
>>     \
>>     -drive if=pflash,format=raw,unit=0,readonly,file="$FW_BIN" \
>>     -drive if=pflash,format=raw,unit=1,file="$VARS" \
>>     \
>>     -device virtio-scsi-pci,id=scsi0 \
>>     -drive if=none,format=raw,readonly,file="$SHELL_ISO",id=shell \
>>     -device scsi-cd,bus=scsi0.0,drive=shell,bootindex=1 \
>>     \
>>     -chardev file,id=debugfile,path=ovmf.log \
>>     -device isa-debugcon,iobase=0x402,chardev=debugfile
>>
>> This will boot the shell for you, and even save the OVMF debug log in
>> the file "ovmf.log".
> 
> I used the following command line (again, a WinXP DOS Session .BAT file):
> 
> "..\qemu-system-i386w.exe" -m 256 -drive if=pflash,format=raw,unit=0,readonly,file=..\OVMF_CODE.fd -drive 
> if=pflash,format=raw,unit=1,file=..\OVMF_VARS.fd -device virtio-scsi-pci,id=scsi0 -drive 
> if=none,format=raw,readonly,file=..\UefiShell.iso,id=shell -device scsi-cd,bus=scsi0.0,drive=shell,bootindex=1 
> -chardev file,id=debugfile,path=ovmf.log -device isa-debugcon,iobase=0x402,chardev=debugfile -drive 
> file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0
> 
> The shell booted up as expected.  However, my BOOT.efi file is now a:
> 
>   "image is not an application"
> 
> I don't think I mentioned before, my BOOT.efi is an OS loader, not 
> a UEFI driver or other type application.

(A UEFI_DRIVER module is *not* an application.)

> Sorry, I should have mentioned
> that earlier, hope that did not confuse you.
> 
> My BOOT.efi file has a value of 0x0B as the Subsystem value.
> 
>    PE Optional Header: Subsystem: 0x000B
> 
> I removed the UefiShell.iso part:
> 
> "..\qemu-system-i386w.exe" -m 256 -drive if=pflash,format=raw,unit=0,readonly,file=..\OVMF_CODE.fd -drive 
> if=pflash,format=raw,unit=1,file=..\OVMF_VARS.fd -device virtio-scsi-pci,id=scsi0 -chardev 
> file,id=debugfile,path=ovmf.log -device isa-debugcon,iobase=0x402,chardev=debugfile -drive 
> file=uefi.bin.vhd,format=raw,if=ide,media=disk,index=0
> 
> and the shell was loaded, with the same result, "not an application".

Based on "MdePkg/Include/IndustryStandard/PeImage.h":

- Subsystem type 0xb (11 decimal) seems to correspond to
EFI_IMAGE_SUBSYSTEM_EFI_BOOT_SERVICE_DRIVER. If your module is an
application however (and a boot loader is usually an application), it
should have EFI_IMAGE_SUBSYSTEM_EFI_APPLICATION (= 0xa, 10 decimal).

- You might want to double-check if the bitnesses match (you can't
launch an IMAGE_FILE_MACHINE_I386 (0x014c) image on the X64 and Ia32X64
builds of OVMF, nor an IMAGE_FILE_MACHINE_X64 (0x8664) image on the Ia32
build).

Laszlo