how to load drivers from additional FV's?

how to load drivers from additional FV's?

[Michael Zimmermann]

I'm trying to add another FV section FVMAIN_COMPACT so I can keep
Platform specific drivers in a separate, included fdf.

I did this:
  FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
    SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
PROCESSING_REQUIRED = TRUE {
      SECTION FV_IMAGE = FVMAIN
      SECTION FV_IMAGE = FVMAINPLATFORM
    }
  }

The image builds file and using uefitool I can verify that the new FV
is inside the compressed section.
But none of the drivers gets discovered/loaded and I get 'Protocol not
present!!' errors.

Thanks
Michael

Re: how to load drivers from additional FV's?

[Laszlo Ersek]

On 03/15/17 13:23, Michael Zimmermann wrote:
> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
> Platform specific drivers in a separate, included fdf.
> 
> I did this:
>   FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>     SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
> PROCESSING_REQUIRED = TRUE {
>       SECTION FV_IMAGE = FVMAIN
>       SECTION FV_IMAGE = FVMAINPLATFORM
>     }
>   }
> 
> The image builds file and using uefitool I can verify that the new FV
> is inside the compressed section.
> But none of the drivers gets discovered/loaded and I get 'Protocol not
> present!!' errors.

The FVs need to be exposed to the DXE core via FV HOBs. See
- 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
  spec,
- and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
  same.

You can use the BuildFvHob() function for this.

If the firmware volume contains PEIMs (... as well), then it has to be
exposed to the PEI core too, I think. I think the
PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
"PEI" in Volume 3 of the PI spec.)

... I used the PeiFvInitialization() function in
OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.

Thanks
Laszlo

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> 
> On 03/15/17 13:23, Michael Zimmermann wrote:
>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>> Platform specific drivers in a separate, included fdf.
>> 
>> I did this:
>>  FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>    SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>> PROCESSING_REQUIRED = TRUE {
>>      SECTION FV_IMAGE = FVMAIN
>>      SECTION FV_IMAGE = FVMAINPLATFORM
>>    }
>>  }
>> 
>> The image builds file and using uefitool I can verify that the new FV
>> is inside the compressed section.
>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>> present!!' errors.
> 
> The FVs need to be exposed to the DXE core via FV HOBs. See
> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>  spec,
> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>  same.
> 
> You can use the BuildFvHob() function for this.
> 
> If the firmware volume contains PEIMs (... as well), then it has to be
> exposed to the PEI core too, I think. I think the
> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
> "PEI" in Volume 3 of the PI spec.)
> 
> ... I used the PeiFvInitialization() function in
> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
> 

Laszlo,

I think this case is an FV that is compressed and nested in another FV that is discovered. I think the issues is multiple FV Sections in an FV file are not currently supported. 

Thanks,

Andrew Fish


> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: how to load drivers from additional FV's?

[Michael Zimmermann]

Laszlo: As Andrew said, the problem is nesting FV's. I've also tried
different ways and depths of nesting FV's in the main FV, but none of
them worked. Also, I'm using PrePi,

Andrew: Yea that seems to be the problem, but how would I access other
sections? ReadSection doesn't seem to take any section id or current
section pointer.

Thanks
Michael

On Wed, Mar 15, 2017 at 4:13 PM, Andrew Fish <afish@apple.com> wrote:
>
>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>
>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>> Platform specific drivers in a separate, included fdf.
>>>
>>> I did this:
>>>  FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>    SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>> PROCESSING_REQUIRED = TRUE {
>>>      SECTION FV_IMAGE = FVMAIN
>>>      SECTION FV_IMAGE = FVMAINPLATFORM
>>>    }
>>>  }
>>>
>>> The image builds file and using uefitool I can verify that the new FV
>>> is inside the compressed section.
>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>> present!!' errors.
>>
>> The FVs need to be exposed to the DXE core via FV HOBs. See
>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>  spec,
>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>  same.
>>
>> You can use the BuildFvHob() function for this.
>>
>> If the firmware volume contains PEIMs (... as well), then it has to be
>> exposed to the PEI core too, I think. I think the
>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>> "PEI" in Volume 3 of the PI spec.)
>>
>> ... I used the PeiFvInitialization() function in
>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>
>
> Laszlo,
>
> I think this case is an FV that is compressed and nested in another FV that is discovered. I think the issues is multiple FV Sections in an FV file are not currently supported.
>
> Thanks,
>
> Andrew Fish
>
>
>> Thanks
>> Laszlo
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 15, 2017, at 8:28 AM, Michael Zimmermann <sigmaepsilon92@gmail.com> wrote:
> 
> Laszlo: As Andrew said, the problem is nesting FV's. I've also tried
> different ways and depths of nesting FV's in the main FV, but none of
> them worked. Also, I'm using PrePi,
> 
> Andrew: Yea that seems to be the problem, but how would I access other
> sections? ReadSection doesn't seem to take any section id or current
> section pointer.
> 

Michael,

The 4th arg that is hard coded to zero is the SectionInstance.

https://github.com/tianocore/edk2/blob/master/MdePkg/Include/Protocol/FirmwareVolume2.h#L409
  
  @param SectionInstance      Indicates which instance of sections
                              with a type of SectionType to return.
                              SectionType in conjunction with
                              SectionInstance indicates which
                              section to return. SectionInstance is
                              zero based.

Thanks,

Andrew Fish

> Thanks
> Michael
> 
> On Wed, Mar 15, 2017 at 4:13 PM, Andrew Fish <afish@apple.com> wrote:
>> 
>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>> 
>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>> Platform specific drivers in a separate, included fdf.
>>>> 
>>>> I did this:
>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>> PROCESSING_REQUIRED = TRUE {
>>>>     SECTION FV_IMAGE = FVMAIN
>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>   }
>>>> }
>>>> 
>>>> The image builds file and using uefitool I can verify that the new FV
>>>> is inside the compressed section.
>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>> present!!' errors.
>>> 
>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>> spec,
>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>> same.
>>> 
>>> You can use the BuildFvHob() function for this.
>>> 
>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>> exposed to the PEI core too, I think. I think the
>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>> "PEI" in Volume 3 of the PI spec.)
>>> 
>>> ... I used the PeiFvInitialization() function in
>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>> 
>> 
>> Laszlo,
>> 
>> I think this case is an FV that is compressed and nested in another FV that is discovered. I think the issues is multiple FV Sections in an FV file are not currently supported.
>> 
>> Thanks,
>> 
>> Andrew Fish
>> 
>> 
>>> Thanks
>>> Laszlo
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: how to load drivers from additional FV's?

[Laszlo Ersek]

On 03/15/17 16:13, Andrew Fish wrote:
> 
>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>
>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>> Platform specific drivers in a separate, included fdf.
>>>
>>> I did this:
>>>  FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>    SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>> PROCESSING_REQUIRED = TRUE {
>>>      SECTION FV_IMAGE = FVMAIN
>>>      SECTION FV_IMAGE = FVMAINPLATFORM
>>>    }
>>>  }
>>>
>>> The image builds file and using uefitool I can verify that the new FV
>>> is inside the compressed section.
>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>> present!!' errors.
>>
>> The FVs need to be exposed to the DXE core via FV HOBs. See
>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>  spec,
>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>  same.
>>
>> You can use the BuildFvHob() function for this.
>>
>> If the firmware volume contains PEIMs (... as well), then it has to be
>> exposed to the PEI core too, I think. I think the
>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>> "PEI" in Volume 3 of the PI spec.)
>>
>> ... I used the PeiFvInitialization() function in
>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>
> 
> Laszlo,
> 
> I think this case is an FV that is compressed and nested in another FV that is discovered. I think the issues is multiple FV Sections in an FV file are not currently supported. 

In OVMF we do the same:

FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF PROCESSING_REQUIRED = TRUE {
     #
     # These firmware volumes will have files placed in them uncompressed,
     # and then both firmware volumes will be compressed in a single
     # compression operation in order to achieve better overall compression.
     #
     SECTION FV_IMAGE = PEIFV
     SECTION FV_IMAGE = DXEFV
   }
 }

"OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into memory -- which is special to OVMF, i.e. the fact that there is writeable memory in SEC --, then jumps to the PEI entry point. PlatformPei then exposes DXEFV to the PEI core (for loading the DXE core AIUI) and to the DXE core as well (for loading DXE modules).

The multiple FV sections in the FFS file are not discovered automatically. First, the OVMF build process saves the final (= decompressed) in-memory locations of the nested FVs into PCDs. Second, in DecompressMemFvs(), the SEC module decompresses the outer, compressed, FFS file, then locates the embedded FVs manually with FindFfsSectionInstance(), then moves the decompressed FVs individually to their pre-recorded locations.

In Michael's case, I think the same should be possible -- since FVMAIN is running for him, the outer, compressed file must already be decompressed (in full) to some temporary buffer in memory. If he can find, manually, the second section (with the embedded FVMAINPLATFORM firmware volume in it) in that area, then he can also expose it to the DXE core, I believe.

Thanks
Laszlo

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> 
> On 03/15/17 16:13, Andrew Fish wrote:
>> 
>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>> 
>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>> Platform specific drivers in a separate, included fdf.
>>>> 
>>>> I did this:
>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>> PROCESSING_REQUIRED = TRUE {
>>>>     SECTION FV_IMAGE = FVMAIN
>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>   }
>>>> }
>>>> 
>>>> The image builds file and using uefitool I can verify that the new FV
>>>> is inside the compressed section.
>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>> present!!' errors.
>>> 
>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>> spec,
>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>> same.
>>> 
>>> You can use the BuildFvHob() function for this.
>>> 
>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>> exposed to the PEI core too, I think. I think the
>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>> "PEI" in Volume 3 of the PI spec.)
>>> 
>>> ... I used the PeiFvInitialization() function in
>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>> 
>> 
>> Laszlo,
>> 
>> I think this case is an FV that is compressed and nested in another FV that is discovered. I think the issues is multiple FV Sections in an FV file are not currently supported. 
> 
> In OVMF we do the same:
> 
> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF PROCESSING_REQUIRED = TRUE {
>     #
>     # These firmware volumes will have files placed in them uncompressed,
>     # and then both firmware volumes will be compressed in a single
>     # compression operation in order to achieve better overall compression.
>     #
>     SECTION FV_IMAGE = PEIFV
>     SECTION FV_IMAGE = DXEFV
>   }
> }
> 
> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into memory -- which is special to OVMF, i.e. the fact that there is writeable memory in SEC --, then jumps to the PEI entry point. PlatformPei then exposes DXEFV to the PEI core (for loading the DXE core AIUI) and to the DXE core as well (for loading DXE modules).
> 
> The multiple FV sections in the FFS file are not discovered automatically. First, the OVMF build process saves the final (= decompressed) in-memory locations of the nested FVs into PCDs. Second, in DecompressMemFvs(), the SEC module decompresses the outer, compressed, FFS file, then locates the embedded FVs manually with FindFfsSectionInstance(), then moves the decompressed FVs individually to their pre-recorded locations.
> 
> In Michael's case, I think the same should be possible -- since FVMAIN is running for him, the outer, compressed file must already be decompressed (in full) to some temporary buffer in memory. If he can find, manually, the second section (with the embedded FVMAINPLATFORM firmware volume in it) in that area, then he can also expose it to the DXE core, I believe.
> 

Laszlo,

The DXE Core will dispatch FVs with Depex "automagically" but only one FV_IMAGE Section per FV_IMAGE file. 

So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd FV in the 1st FV_IMAGE Section. 

Also I'm not sure if the outer FV is compressed, but compressing compressed data usually does not end well as the algorithms leverage entropy. It is better to just compress the outer FV and have everything else uncompressed and let it all get compressed together. This also helps as there will be common dictionary entries so compressing as much data as possible in one shot usually works better. So for example if the outer FV is compressed it may have a dictionary entry that represents CopyMem, and it can represent that pattern with a small number of bits. Doing compression multiple times causes duplication as each compressed image has one copy of CopyMem (massive over simplification). Also if you compress a nested FV then the CopyMem pattern does not exist in the compressed data, and since the compressed data is more random it is hard to find patterns to compress, so it hurts the compression of the outer FV. 

Thanks,

Andrew Fish

> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>

Re: how to load drivers from additional FV's?

[Michael Zimmermann]

Andrew, using a fallback with 1 as the fourth argument didn't seem to work.

> So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd FV in the 1st FV_IMAGE Section.
I've actually tried that before and it didn't seem to work. BUt after
what you've said I tried using a uncompressed, nested FV like
this(inside [FV.FvMain]):

FILE FV_IMAGE = B67596E9-EA27-40E4-885C-5AE83D414BD4 {
  SECTION FV_IMAGE = FVMAINPLATFORM
}

And that actually worked! So is this a workaround or a clean solution?

Thanks a lot
Michael


On Wed, Mar 15, 2017 at 4:54 PM, Andrew Fish <afish@apple.com> wrote:
>
> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>
> On 03/15/17 16:13, Andrew Fish wrote:
>
>
> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>
> On 03/15/17 13:23, Michael Zimmermann wrote:
>
> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
> Platform specific drivers in a separate, included fdf.
>
> I did this:
> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
> PROCESSING_REQUIRED = TRUE {
>     SECTION FV_IMAGE = FVMAIN
>     SECTION FV_IMAGE = FVMAINPLATFORM
>   }
> }
>
> The image builds file and using uefitool I can verify that the new FV
> is inside the compressed section.
> But none of the drivers gets discovered/loaded and I get 'Protocol not
> present!!' errors.
>
>
> The FVs need to be exposed to the DXE core via FV HOBs. See
> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
> spec,
> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
> same.
>
> You can use the BuildFvHob() function for this.
>
> If the firmware volume contains PEIMs (... as well), then it has to be
> exposed to the PEI core too, I think. I think the
> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
> "PEI" in Volume 3 of the PI spec.)
>
> ... I used the PeiFvInitialization() function in
> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>
>
> Laszlo,
>
> I think this case is an FV that is compressed and nested in another FV that
> is discovered. I think the issues is multiple FV Sections in an FV file are
> not currently supported.
>
>
> In OVMF we do the same:
>
> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF PROCESSING_REQUIRED =
> TRUE {
>     #
>     # These firmware volumes will have files placed in them uncompressed,
>     # and then both firmware volumes will be compressed in a single
>     # compression operation in order to achieve better overall compression.
>     #
>     SECTION FV_IMAGE = PEIFV
>     SECTION FV_IMAGE = DXEFV
>   }
> }
>
> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into memory --
> which is special to OVMF, i.e. the fact that there is writeable memory in
> SEC --, then jumps to the PEI entry point. PlatformPei then exposes DXEFV to
> the PEI core (for loading the DXE core AIUI) and to the DXE core as well
> (for loading DXE modules).
>
> The multiple FV sections in the FFS file are not discovered automatically.
> First, the OVMF build process saves the final (= decompressed) in-memory
> locations of the nested FVs into PCDs. Second, in DecompressMemFvs(), the
> SEC module decompresses the outer, compressed, FFS file, then locates the
> embedded FVs manually with FindFfsSectionInstance(), then moves the
> decompressed FVs individually to their pre-recorded locations.
>
> In Michael's case, I think the same should be possible -- since FVMAIN is
> running for him, the outer, compressed file must already be decompressed (in
> full) to some temporary buffer in memory. If he can find, manually, the
> second section (with the embedded FVMAINPLATFORM firmware volume in it) in
> that area, then he can also expose it to the DXE core, I believe.
>
>
> Laszlo,
>
> The DXE Core will dispatch FVs with Depex "automagically" but only one
> FV_IMAGE Section per FV_IMAGE file.
>
> So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd
> FV in the 1st FV_IMAGE Section.
>
> Also I'm not sure if the outer FV is compressed, but compressing compressed
> data usually does not end well as the algorithms leverage entropy. It is
> better to just compress the outer FV and have everything else uncompressed
> and let it all get compressed together. This also helps as there will be
> common dictionary entries so compressing as much data as possible in one
> shot usually works better. So for example if the outer FV is compressed it
> may have a dictionary entry that represents CopyMem, and it can represent
> that pattern with a small number of bits. Doing compression multiple times
> causes duplication as each compressed image has one copy of CopyMem (massive
> over simplification). Also if you compress a nested FV then the CopyMem
> pattern does not exist in the compressed data, and since the compressed data
> is more random it is hard to find patterns to compress, so it hurts the
> compression of the outer FV.
>
> Thanks,
>
> Andrew Fish
>
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>
>

Re: how to load drivers from additional FV's?

[Gao, Liming]

Michael:
  I agree this is an issue in PeiCore and DxeCore, because PI spec has no limitation to nest FV number. Could you help submit one tracker in bugzillar? We will fix it. 

Thanks
Liming
> -----Original Message-----
> From: Michael Zimmermann [mailto:sigmaepsilon92@gmail.com]
> Sent: Thursday, March 16, 2017 12:08 AM
> To: Andrew Fish <afish@apple.com>
> Cc: Laszlo Ersek <lersek@redhat.com>; Tian, Feng <feng.tian@intel.com>; Gao, Liming <liming.gao@intel.com>;
> edk2-devel@lists.01.org <edk2-devel@ml01.01.org>; Zeng, Star <star.zeng@intel.com>
> Subject: Re: [edk2] how to load drivers from additional FV's?
> 
> Andrew, using a fallback with 1 as the fourth argument didn't seem to work.
> 
> > So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd FV in the 1st FV_IMAGE Section.
> I've actually tried that before and it didn't seem to work. BUt after
> what you've said I tried using a uncompressed, nested FV like
> this(inside [FV.FvMain]):
> 
> FILE FV_IMAGE = B67596E9-EA27-40E4-885C-5AE83D414BD4 {
>   SECTION FV_IMAGE = FVMAINPLATFORM
> }
> 
> And that actually worked! So is this a workaround or a clean solution?
> 
> Thanks a lot
> Michael
> 
> 
> On Wed, Mar 15, 2017 at 4:54 PM, Andrew Fish <afish@apple.com> wrote:
> >
> > On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> >
> > On 03/15/17 16:13, Andrew Fish wrote:
> >
> >
> > On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> >
> > On 03/15/17 13:23, Michael Zimmermann wrote:
> >
> > I'm trying to add another FV section FVMAIN_COMPACT so I can keep
> > Platform specific drivers in a separate, included fdf.
> >
> > I did this:
> > FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
> >   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
> > PROCESSING_REQUIRED = TRUE {
> >     SECTION FV_IMAGE = FVMAIN
> >     SECTION FV_IMAGE = FVMAINPLATFORM
> >   }
> > }
> >
> > The image builds file and using uefitool I can verify that the new FV
> > is inside the compressed section.
> > But none of the drivers gets discovered/loaded and I get 'Protocol not
> > present!!' errors.
> >
> >
> > The FVs need to be exposed to the DXE core via FV HOBs. See
> > - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
> > spec,
> > - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
> > same.
> >
> > You can use the BuildFvHob() function for this.
> >
> > If the firmware volume contains PEIMs (... as well), then it has to be
> > exposed to the PEI core too, I think. I think the
> > PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
> > "PEI" in Volume 3 of the PI spec.)
> >
> > ... I used the PeiFvInitialization() function in
> > OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
> >
> >
> > Laszlo,
> >
> > I think this case is an FV that is compressed and nested in another FV that
> > is discovered. I think the issues is multiple FV Sections in an FV file are
> > not currently supported.
> >
> >
> > In OVMF we do the same:
> >
> > FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
> >   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF PROCESSING_REQUIRED =
> > TRUE {
> >     #
> >     # These firmware volumes will have files placed in them uncompressed,
> >     # and then both firmware volumes will be compressed in a single
> >     # compression operation in order to achieve better overall compression.
> >     #
> >     SECTION FV_IMAGE = PEIFV
> >     SECTION FV_IMAGE = DXEFV
> >   }
> > }
> >
> > "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into memory --
> > which is special to OVMF, i.e. the fact that there is writeable memory in
> > SEC --, then jumps to the PEI entry point. PlatformPei then exposes DXEFV to
> > the PEI core (for loading the DXE core AIUI) and to the DXE core as well
> > (for loading DXE modules).
> >
> > The multiple FV sections in the FFS file are not discovered automatically.
> > First, the OVMF build process saves the final (= decompressed) in-memory
> > locations of the nested FVs into PCDs. Second, in DecompressMemFvs(), the
> > SEC module decompresses the outer, compressed, FFS file, then locates the
> > embedded FVs manually with FindFfsSectionInstance(), then moves the
> > decompressed FVs individually to their pre-recorded locations.
> >
> > In Michael's case, I think the same should be possible -- since FVMAIN is
> > running for him, the outer, compressed file must already be decompressed (in
> > full) to some temporary buffer in memory. If he can find, manually, the
> > second section (with the embedded FVMAINPLATFORM firmware volume in it) in
> > that area, then he can also expose it to the DXE core, I believe.
> >
> >
> > Laszlo,
> >
> > The DXE Core will dispatch FVs with Depex "automagically" but only one
> > FV_IMAGE Section per FV_IMAGE file.
> >
> > So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd
> > FV in the 1st FV_IMAGE Section.
> >
> > Also I'm not sure if the outer FV is compressed, but compressing compressed
> > data usually does not end well as the algorithms leverage entropy. It is
> > better to just compress the outer FV and have everything else uncompressed
> > and let it all get compressed together. This also helps as there will be
> > common dictionary entries so compressing as much data as possible in one
> > shot usually works better. So for example if the outer FV is compressed it
> > may have a dictionary entry that represents CopyMem, and it can represent
> > that pattern with a small number of bits. Doing compression multiple times
> > causes duplication as each compressed image has one copy of CopyMem (massive
> > over simplification). Also if you compress a nested FV then the CopyMem
> > pattern does not exist in the compressed data, and since the compressed data
> > is more random it is hard to find patterns to compress, so it hurts the
> > compression of the outer FV.
> >
> > Thanks,
> >
> > Andrew Fish
> >
> > Thanks
> > Laszlo
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> >
> >

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 15, 2017, at 7:29 PM, Gao, Liming <liming.gao@intel.com> wrote:
> 
> Michael:
>  I agree this is an issue in PeiCore and DxeCore, because PI spec has no limitation to nest FV number. Could you help submit one tracker in bugzillar? We will fix it. 
> 

Liming,

Thanks for double checking the spec, I did not have time to do that today.

Thanks,

Andrew Fish

> Thanks
> Liming
>> -----Original Message-----
>> From: Michael Zimmermann [mailto:sigmaepsilon92@gmail.com]
>> Sent: Thursday, March 16, 2017 12:08 AM
>> To: Andrew Fish <afish@apple.com>
>> Cc: Laszlo Ersek <lersek@redhat.com>; Tian, Feng <feng.tian@intel.com>; Gao, Liming <liming.gao@intel.com>;
>> edk2-devel@lists.01.org <edk2-devel@ml01.01.org>; Zeng, Star <star.zeng@intel.com>
>> Subject: Re: [edk2] how to load drivers from additional FV's?
>> 
>> Andrew, using a fallback with 1 as the fourth argument didn't seem to work.
>> 
>>> So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd FV in the 1st FV_IMAGE Section.
>> I've actually tried that before and it didn't seem to work. BUt after
>> what you've said I tried using a uncompressed, nested FV like
>> this(inside [FV.FvMain]):
>> 
>> FILE FV_IMAGE = B67596E9-EA27-40E4-885C-5AE83D414BD4 {
>>  SECTION FV_IMAGE = FVMAINPLATFORM
>> }
>> 
>> And that actually worked! So is this a workaround or a clean solution?
>> 
>> Thanks a lot
>> Michael
>> 
>> 
>> On Wed, Mar 15, 2017 at 4:54 PM, Andrew Fish <afish@apple.com> wrote:
>>> 
>>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>> 
>>> On 03/15/17 16:13, Andrew Fish wrote:
>>> 
>>> 
>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
>>> 
>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>> 
>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>> Platform specific drivers in a separate, included fdf.
>>> 
>>> I did this:
>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>  SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>> PROCESSING_REQUIRED = TRUE {
>>>    SECTION FV_IMAGE = FVMAIN
>>>    SECTION FV_IMAGE = FVMAINPLATFORM
>>>  }
>>> }
>>> 
>>> The image builds file and using uefitool I can verify that the new FV
>>> is inside the compressed section.
>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>> present!!' errors.
>>> 
>>> 
>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>> spec,
>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>> same.
>>> 
>>> You can use the BuildFvHob() function for this.
>>> 
>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>> exposed to the PEI core too, I think. I think the
>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>> "PEI" in Volume 3 of the PI spec.)
>>> 
>>> ... I used the PeiFvInitialization() function in
>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>> 
>>> 
>>> Laszlo,
>>> 
>>> I think this case is an FV that is compressed and nested in another FV that
>>> is discovered. I think the issues is multiple FV Sections in an FV file are
>>> not currently supported.
>>> 
>>> 
>>> In OVMF we do the same:
>>> 
>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>  SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF PROCESSING_REQUIRED =
>>> TRUE {
>>>    #
>>>    # These firmware volumes will have files placed in them uncompressed,
>>>    # and then both firmware volumes will be compressed in a single
>>>    # compression operation in order to achieve better overall compression.
>>>    #
>>>    SECTION FV_IMAGE = PEIFV
>>>    SECTION FV_IMAGE = DXEFV
>>>  }
>>> }
>>> 
>>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into memory --
>>> which is special to OVMF, i.e. the fact that there is writeable memory in
>>> SEC --, then jumps to the PEI entry point. PlatformPei then exposes DXEFV to
>>> the PEI core (for loading the DXE core AIUI) and to the DXE core as well
>>> (for loading DXE modules).
>>> 
>>> The multiple FV sections in the FFS file are not discovered automatically.
>>> First, the OVMF build process saves the final (= decompressed) in-memory
>>> locations of the nested FVs into PCDs. Second, in DecompressMemFvs(), the
>>> SEC module decompresses the outer, compressed, FFS file, then locates the
>>> embedded FVs manually with FindFfsSectionInstance(), then moves the
>>> decompressed FVs individually to their pre-recorded locations.
>>> 
>>> In Michael's case, I think the same should be possible -- since FVMAIN is
>>> running for him, the outer, compressed file must already be decompressed (in
>>> full) to some temporary buffer in memory. If he can find, manually, the
>>> second section (with the embedded FVMAINPLATFORM firmware volume in it) in
>>> that area, then he can also expose it to the DXE core, I believe.
>>> 
>>> 
>>> Laszlo,
>>> 
>>> The DXE Core will dispatch FVs with Depex "automagically" but only one
>>> FV_IMAGE Section per FV_IMAGE file.
>>> 
>>> So one workaround would be to add a 2nd FV_IMAGE file that contains the 2nd
>>> FV in the 1st FV_IMAGE Section.
>>> 
>>> Also I'm not sure if the outer FV is compressed, but compressing compressed
>>> data usually does not end well as the algorithms leverage entropy. It is
>>> better to just compress the outer FV and have everything else uncompressed
>>> and let it all get compressed together. This also helps as there will be
>>> common dictionary entries so compressing as much data as possible in one
>>> shot usually works better. So for example if the outer FV is compressed it
>>> may have a dictionary entry that represents CopyMem, and it can represent
>>> that pattern with a small number of bits. Doing compression multiple times
>>> causes duplication as each compressed image has one copy of CopyMem (massive
>>> over simplification). Also if you compress a nested FV then the CopyMem
>>> pattern does not exist in the compressed data, and since the compressed data
>>> is more random it is hard to find patterns to compress, so it hurts the
>>> compression of the outer FV.
>>> 
>>> Thanks,
>>> 
>>> Andrew Fish
>>> 
>>> Thanks
>>> Laszlo
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>> 
>>> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: how to load drivers from additional FV's?

[Laszlo Ersek]

On 03/15/17 16:54, Andrew Fish wrote:
> 
>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com
>> <mailto:lersek@redhat.com>> wrote:
>>
>> On 03/15/17 16:13, Andrew Fish wrote:
>>>
>>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com
>>>> <mailto:lersek@redhat.com>> wrote:
>>>>
>>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>>> Platform specific drivers in a separate, included fdf.
>>>>>
>>>>> I did this:
>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>     SECTION FV_IMAGE = FVMAIN
>>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>>   }
>>>>> }
>>>>>
>>>>> The image builds file and using uefitool I can verify that the new FV
>>>>> is inside the compressed section.
>>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>>> present!!' errors.
>>>>
>>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>>> spec,
>>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>>> same.
>>>>
>>>> You can use the BuildFvHob() function for this.
>>>>
>>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>>> exposed to the PEI core too, I think. I think the
>>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>>> "PEI" in Volume 3 of the PI spec.)
>>>>
>>>> ... I used the PeiFvInitialization() function in
>>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>>>
>>>
>>> Laszlo,
>>>
>>> I think this case is an FV that is compressed and nested in another
>>> FV that is discovered. I think the issues is multiple FV Sections in
>>> an FV file are not currently supported. 
>>
>> In OVMF we do the same:
>>
>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>> PROCESSING_REQUIRED = TRUE {
>>     #
>>     # These firmware volumes will have files placed in them uncompressed,
>>     # and then both firmware volumes will be compressed in a single
>>     # compression operation in order to achieve better overall
>> compression.
>>     #
>>     SECTION FV_IMAGE = PEIFV
>>     SECTION FV_IMAGE = DXEFV
>>   }
>> }
>>
>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into
>> memory -- which is special to OVMF, i.e. the fact that there is
>> writeable memory in SEC --, then jumps to the PEI entry point.
>> PlatformPei then exposes DXEFV to the PEI core (for loading the DXE
>> core AIUI) and to the DXE core as well (for loading DXE modules).
>>
>> The multiple FV sections in the FFS file are not discovered
>> automatically. First, the OVMF build process saves the final (=
>> decompressed) in-memory locations of the nested FVs into PCDs. Second,
>> in DecompressMemFvs(), the SEC module decompresses the outer,
>> compressed, FFS file, then locates the embedded FVs manually with
>> FindFfsSectionInstance(), then moves the decompressed FVs individually
>> to their pre-recorded locations.
>>
>> In Michael's case, I think the same should be possible -- since FVMAIN
>> is running for him, the outer, compressed file must already be
>> decompressed (in full) to some temporary buffer in memory. If he can
>> find, manually, the second section (with the embedded FVMAINPLATFORM
>> firmware volume in it) in that area, then he can also expose it to the
>> DXE core, I believe.
>>
> 
> Laszlo,
> 
> The DXE Core will dispatch FVs with Depex "automagically" but only one
> FV_IMAGE Section per FV_IMAGE file. 
> 
> So one workaround would be to add a 2nd FV_IMAGE file that contains the
> 2nd FV in the 1st FV_IMAGE Section. 
> 
> Also I'm not sure if the outer FV is compressed, but compressing
> compressed data usually does not end well as the algorithms leverage
> entropy.

I apologize if my previous emails were confusing; I didn't mean to imply
nested compression. OVMF doesn't do that.

> It is better to just compress the outer FV and have everything
> else uncompressed and let it all get compressed together.

Yes, that's what OVMF does.

Did you perhaps misread the comment above that I quoted from the FDF?

"These firmware volumes will have files placed in them *uncompressed*,
and then both firmware volumes will be compressed in a single
compression operation in order to achieve better overall compression."

(Emphasis mine.)

Thanks!
Laszlo

> This also
> helps as there will be common dictionary entries so compressing as much
> data as possible in one shot usually works better. So for example if the
> outer FV is compressed it may have a dictionary entry that represents
> CopyMem, and it can represent that pattern with a small number of bits.
> Doing compression multiple times causes duplication as each compressed
> image has one copy of CopyMem (massive over simplification). Also if you
> compress a nested FV then the CopyMem pattern does not exist in the
> compressed data, and since the compressed data is more random it is hard
> to find patterns to compress, so it hurts the compression of the outer FV. 
> 
> Thanks,
> 
> Andrew Fish
> 
>> Thanks
>> Laszlo
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>> https://lists.01.org/mailman/listinfo/edk2-devel
> 

Re: how to load drivers from additional FV's?

[Michael Zimmermann]

So if I understand this correctly there's no bug to fix because
nested, uncompressed FV's are already supported and using compressed
ones is bad practice anyway?

Thanks
Michael

On Thu, Mar 16, 2017 at 12:20 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> On 03/15/17 16:54, Andrew Fish wrote:
>>
>>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com
>>> <mailto:lersek@redhat.com>> wrote:
>>>
>>> On 03/15/17 16:13, Andrew Fish wrote:
>>>>
>>>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com
>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>
>>>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>>>> Platform specific drivers in a separate, included fdf.
>>>>>>
>>>>>> I did this:
>>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>>     SECTION FV_IMAGE = FVMAIN
>>>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>>>   }
>>>>>> }
>>>>>>
>>>>>> The image builds file and using uefitool I can verify that the new FV
>>>>>> is inside the compressed section.
>>>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>>>> present!!' errors.
>>>>>
>>>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>>>> spec,
>>>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>>>> same.
>>>>>
>>>>> You can use the BuildFvHob() function for this.
>>>>>
>>>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>>>> exposed to the PEI core too, I think. I think the
>>>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>>>> "PEI" in Volume 3 of the PI spec.)
>>>>>
>>>>> ... I used the PeiFvInitialization() function in
>>>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>>>>
>>>>
>>>> Laszlo,
>>>>
>>>> I think this case is an FV that is compressed and nested in another
>>>> FV that is discovered. I think the issues is multiple FV Sections in
>>>> an FV file are not currently supported.
>>>
>>> In OVMF we do the same:
>>>
>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>> PROCESSING_REQUIRED = TRUE {
>>>     #
>>>     # These firmware volumes will have files placed in them uncompressed,
>>>     # and then both firmware volumes will be compressed in a single
>>>     # compression operation in order to achieve better overall
>>> compression.
>>>     #
>>>     SECTION FV_IMAGE = PEIFV
>>>     SECTION FV_IMAGE = DXEFV
>>>   }
>>> }
>>>
>>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into
>>> memory -- which is special to OVMF, i.e. the fact that there is
>>> writeable memory in SEC --, then jumps to the PEI entry point.
>>> PlatformPei then exposes DXEFV to the PEI core (for loading the DXE
>>> core AIUI) and to the DXE core as well (for loading DXE modules).
>>>
>>> The multiple FV sections in the FFS file are not discovered
>>> automatically. First, the OVMF build process saves the final (=
>>> decompressed) in-memory locations of the nested FVs into PCDs. Second,
>>> in DecompressMemFvs(), the SEC module decompresses the outer,
>>> compressed, FFS file, then locates the embedded FVs manually with
>>> FindFfsSectionInstance(), then moves the decompressed FVs individually
>>> to their pre-recorded locations.
>>>
>>> In Michael's case, I think the same should be possible -- since FVMAIN
>>> is running for him, the outer, compressed file must already be
>>> decompressed (in full) to some temporary buffer in memory. If he can
>>> find, manually, the second section (with the embedded FVMAINPLATFORM
>>> firmware volume in it) in that area, then he can also expose it to the
>>> DXE core, I believe.
>>>
>>
>> Laszlo,
>>
>> The DXE Core will dispatch FVs with Depex "automagically" but only one
>> FV_IMAGE Section per FV_IMAGE file.
>>
>> So one workaround would be to add a 2nd FV_IMAGE file that contains the
>> 2nd FV in the 1st FV_IMAGE Section.
>>
>> Also I'm not sure if the outer FV is compressed, but compressing
>> compressed data usually does not end well as the algorithms leverage
>> entropy.
>
> I apologize if my previous emails were confusing; I didn't mean to imply
> nested compression. OVMF doesn't do that.
>
>> It is better to just compress the outer FV and have everything
>> else uncompressed and let it all get compressed together.
>
> Yes, that's what OVMF does.
>
> Did you perhaps misread the comment above that I quoted from the FDF?
>
> "These firmware volumes will have files placed in them *uncompressed*,
> and then both firmware volumes will be compressed in a single
> compression operation in order to achieve better overall compression."
>
> (Emphasis mine.)
>
> Thanks!
> Laszlo
>
>> This also
>> helps as there will be common dictionary entries so compressing as much
>> data as possible in one shot usually works better. So for example if the
>> outer FV is compressed it may have a dictionary entry that represents
>> CopyMem, and it can represent that pattern with a small number of bits.
>> Doing compression multiple times causes duplication as each compressed
>> image has one copy of CopyMem (massive over simplification). Also if you
>> compress a nested FV then the CopyMem pattern does not exist in the
>> compressed data, and since the compressed data is more random it is hard
>> to find patterns to compress, so it hurts the compression of the outer FV.
>>
>> Thanks,
>>
>> Andrew Fish
>>
>>> Thanks
>>> Laszlo
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>
>

Re: how to load drivers from additional FV's?

[Laszlo Ersek]

On 03/16/17 13:20, Michael Zimmermann wrote:
> So if I understand this correctly there's no bug to fix because
> nested, uncompressed FV's are already supported

I'm not sure how to answer this; I guess the answer depends on what
meaning you attribute to "supported". In OVMF it works, but it depends
on two things: OVMF's SEC driver having access to writeable RAM, and
OVMF manually messing with the firmware volumes (locating them,
decompressing the outer file, moving stuff around, and so on).

On a physical platform, I guess you can do the same, if you are willing
to mess with the FVs manually, and you delay that code until at least
the PEI phase sees the permanent RAM installed.

Thanks
Laszlo

> and using compressed
> ones is bad practice anyway?
> 
> Thanks
> Michael
> 
> On Thu, Mar 16, 2017 at 12:20 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>> On 03/15/17 16:54, Andrew Fish wrote:
>>>
>>>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com
>>>> <mailto:lersek@redhat.com>> wrote:
>>>>
>>>> On 03/15/17 16:13, Andrew Fish wrote:
>>>>>
>>>>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com
>>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>>
>>>>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>>>>> Platform specific drivers in a separate, included fdf.
>>>>>>>
>>>>>>> I did this:
>>>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>>>     SECTION FV_IMAGE = FVMAIN
>>>>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>>>>   }
>>>>>>> }
>>>>>>>
>>>>>>> The image builds file and using uefitool I can verify that the new FV
>>>>>>> is inside the compressed section.
>>>>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>>>>> present!!' errors.
>>>>>>
>>>>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>>>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>>>>> spec,
>>>>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>>>>> same.
>>>>>>
>>>>>> You can use the BuildFvHob() function for this.
>>>>>>
>>>>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>>>>> exposed to the PEI core too, I think. I think the
>>>>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>>>>> "PEI" in Volume 3 of the PI spec.)
>>>>>>
>>>>>> ... I used the PeiFvInitialization() function in
>>>>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>>>>>
>>>>>
>>>>> Laszlo,
>>>>>
>>>>> I think this case is an FV that is compressed and nested in another
>>>>> FV that is discovered. I think the issues is multiple FV Sections in
>>>>> an FV file are not currently supported.
>>>>
>>>> In OVMF we do the same:
>>>>
>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>> PROCESSING_REQUIRED = TRUE {
>>>>     #
>>>>     # These firmware volumes will have files placed in them uncompressed,
>>>>     # and then both firmware volumes will be compressed in a single
>>>>     # compression operation in order to achieve better overall
>>>> compression.
>>>>     #
>>>>     SECTION FV_IMAGE = PEIFV
>>>>     SECTION FV_IMAGE = DXEFV
>>>>   }
>>>> }
>>>>
>>>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into
>>>> memory -- which is special to OVMF, i.e. the fact that there is
>>>> writeable memory in SEC --, then jumps to the PEI entry point.
>>>> PlatformPei then exposes DXEFV to the PEI core (for loading the DXE
>>>> core AIUI) and to the DXE core as well (for loading DXE modules).
>>>>
>>>> The multiple FV sections in the FFS file are not discovered
>>>> automatically. First, the OVMF build process saves the final (=
>>>> decompressed) in-memory locations of the nested FVs into PCDs. Second,
>>>> in DecompressMemFvs(), the SEC module decompresses the outer,
>>>> compressed, FFS file, then locates the embedded FVs manually with
>>>> FindFfsSectionInstance(), then moves the decompressed FVs individually
>>>> to their pre-recorded locations.
>>>>
>>>> In Michael's case, I think the same should be possible -- since FVMAIN
>>>> is running for him, the outer, compressed file must already be
>>>> decompressed (in full) to some temporary buffer in memory. If he can
>>>> find, manually, the second section (with the embedded FVMAINPLATFORM
>>>> firmware volume in it) in that area, then he can also expose it to the
>>>> DXE core, I believe.
>>>>
>>>
>>> Laszlo,
>>>
>>> The DXE Core will dispatch FVs with Depex "automagically" but only one
>>> FV_IMAGE Section per FV_IMAGE file.
>>>
>>> So one workaround would be to add a 2nd FV_IMAGE file that contains the
>>> 2nd FV in the 1st FV_IMAGE Section.
>>>
>>> Also I'm not sure if the outer FV is compressed, but compressing
>>> compressed data usually does not end well as the algorithms leverage
>>> entropy.
>>
>> I apologize if my previous emails were confusing; I didn't mean to imply
>> nested compression. OVMF doesn't do that.
>>
>>> It is better to just compress the outer FV and have everything
>>> else uncompressed and let it all get compressed together.
>>
>> Yes, that's what OVMF does.
>>
>> Did you perhaps misread the comment above that I quoted from the FDF?
>>
>> "These firmware volumes will have files placed in them *uncompressed*,
>> and then both firmware volumes will be compressed in a single
>> compression operation in order to achieve better overall compression."
>>
>> (Emphasis mine.)
>>
>> Thanks!
>> Laszlo
>>
>>> This also
>>> helps as there will be common dictionary entries so compressing as much
>>> data as possible in one shot usually works better. So for example if the
>>> outer FV is compressed it may have a dictionary entry that represents
>>> CopyMem, and it can represent that pattern with a small number of bits.
>>> Doing compression multiple times causes duplication as each compressed
>>> image has one copy of CopyMem (massive over simplification). Also if you
>>> compress a nested FV then the CopyMem pattern does not exist in the
>>> compressed data, and since the compressed data is more random it is hard
>>> to find patterns to compress, so it hurts the compression of the outer FV.
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>> Thanks
>>>> Laszlo
>>>> _______________________________________________
>>>> edk2-devel mailing list
>>>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>>
>>

Re: how to load drivers from additional FV's?

[Michael Zimmermann]

Laszlo, I don't know if I was clear enough but I've been able to load
drivers from a nested FV without any manual loading or decompressing
and without any changes to my code.

All I needed to do was putting this into FVMAIN:
  FILE FV_IMAGE = $(GUID) {
    SECTION FV_IMAGE = FVMAINPLATFORM
  }

Thanks
Michael

On Thu, Mar 16, 2017 at 1:27 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> On 03/16/17 13:20, Michael Zimmermann wrote:
>> So if I understand this correctly there's no bug to fix because
>> nested, uncompressed FV's are already supported
>
> I'm not sure how to answer this; I guess the answer depends on what
> meaning you attribute to "supported". In OVMF it works, but it depends
> on two things: OVMF's SEC driver having access to writeable RAM, and
> OVMF manually messing with the firmware volumes (locating them,
> decompressing the outer file, moving stuff around, and so on).
>
> On a physical platform, I guess you can do the same, if you are willing
> to mess with the FVs manually, and you delay that code until at least
> the PEI phase sees the permanent RAM installed.
>
> Thanks
> Laszlo
>
>> and using compressed
>> ones is bad practice anyway?
>>
>> Thanks
>> Michael
>>
>> On Thu, Mar 16, 2017 at 12:20 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>>> On 03/15/17 16:54, Andrew Fish wrote:
>>>>
>>>>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com
>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>
>>>>> On 03/15/17 16:13, Andrew Fish wrote:
>>>>>>
>>>>>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com
>>>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>>>
>>>>>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>>>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>>>>>> Platform specific drivers in a separate, included fdf.
>>>>>>>>
>>>>>>>> I did this:
>>>>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>>>>     SECTION FV_IMAGE = FVMAIN
>>>>>>>>     SECTION FV_IMAGE = FVMAINPLATFORM
>>>>>>>>   }
>>>>>>>> }
>>>>>>>>
>>>>>>>> The image builds file and using uefitool I can verify that the new FV
>>>>>>>> is inside the compressed section.
>>>>>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>>>>>> present!!' errors.
>>>>>>>
>>>>>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>>>>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>>>>>> spec,
>>>>>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>>>>>> same.
>>>>>>>
>>>>>>> You can use the BuildFvHob() function for this.
>>>>>>>
>>>>>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>>>>>> exposed to the PEI core too, I think. I think the
>>>>>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>>>>>> "PEI" in Volume 3 of the PI spec.)
>>>>>>>
>>>>>>> ... I used the PeiFvInitialization() function in
>>>>>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>>>>>>
>>>>>>
>>>>>> Laszlo,
>>>>>>
>>>>>> I think this case is an FV that is compressed and nested in another
>>>>>> FV that is discovered. I think the issues is multiple FV Sections in
>>>>>> an FV file are not currently supported.
>>>>>
>>>>> In OVMF we do the same:
>>>>>
>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>   SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>     #
>>>>>     # These firmware volumes will have files placed in them uncompressed,
>>>>>     # and then both firmware volumes will be compressed in a single
>>>>>     # compression operation in order to achieve better overall
>>>>> compression.
>>>>>     #
>>>>>     SECTION FV_IMAGE = PEIFV
>>>>>     SECTION FV_IMAGE = DXEFV
>>>>>   }
>>>>> }
>>>>>
>>>>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into
>>>>> memory -- which is special to OVMF, i.e. the fact that there is
>>>>> writeable memory in SEC --, then jumps to the PEI entry point.
>>>>> PlatformPei then exposes DXEFV to the PEI core (for loading the DXE
>>>>> core AIUI) and to the DXE core as well (for loading DXE modules).
>>>>>
>>>>> The multiple FV sections in the FFS file are not discovered
>>>>> automatically. First, the OVMF build process saves the final (=
>>>>> decompressed) in-memory locations of the nested FVs into PCDs. Second,
>>>>> in DecompressMemFvs(), the SEC module decompresses the outer,
>>>>> compressed, FFS file, then locates the embedded FVs manually with
>>>>> FindFfsSectionInstance(), then moves the decompressed FVs individually
>>>>> to their pre-recorded locations.
>>>>>
>>>>> In Michael's case, I think the same should be possible -- since FVMAIN
>>>>> is running for him, the outer, compressed file must already be
>>>>> decompressed (in full) to some temporary buffer in memory. If he can
>>>>> find, manually, the second section (with the embedded FVMAINPLATFORM
>>>>> firmware volume in it) in that area, then he can also expose it to the
>>>>> DXE core, I believe.
>>>>>
>>>>
>>>> Laszlo,
>>>>
>>>> The DXE Core will dispatch FVs with Depex "automagically" but only one
>>>> FV_IMAGE Section per FV_IMAGE file.
>>>>
>>>> So one workaround would be to add a 2nd FV_IMAGE file that contains the
>>>> 2nd FV in the 1st FV_IMAGE Section.
>>>>
>>>> Also I'm not sure if the outer FV is compressed, but compressing
>>>> compressed data usually does not end well as the algorithms leverage
>>>> entropy.
>>>
>>> I apologize if my previous emails were confusing; I didn't mean to imply
>>> nested compression. OVMF doesn't do that.
>>>
>>>> It is better to just compress the outer FV and have everything
>>>> else uncompressed and let it all get compressed together.
>>>
>>> Yes, that's what OVMF does.
>>>
>>> Did you perhaps misread the comment above that I quoted from the FDF?
>>>
>>> "These firmware volumes will have files placed in them *uncompressed*,
>>> and then both firmware volumes will be compressed in a single
>>> compression operation in order to achieve better overall compression."
>>>
>>> (Emphasis mine.)
>>>
>>> Thanks!
>>> Laszlo
>>>
>>>> This also
>>>> helps as there will be common dictionary entries so compressing as much
>>>> data as possible in one shot usually works better. So for example if the
>>>> outer FV is compressed it may have a dictionary entry that represents
>>>> CopyMem, and it can represent that pattern with a small number of bits.
>>>> Doing compression multiple times causes duplication as each compressed
>>>> image has one copy of CopyMem (massive over simplification). Also if you
>>>> compress a nested FV then the CopyMem pattern does not exist in the
>>>> compressed data, and since the compressed data is more random it is hard
>>>> to find patterns to compress, so it hurts the compression of the outer FV.
>>>>
>>>> Thanks,
>>>>
>>>> Andrew Fish
>>>>
>>>>> Thanks
>>>>> Laszlo
>>>>> _______________________________________________
>>>>> edk2-devel mailing list
>>>>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>>>
>>>
>

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 16, 2017, at 5:33 AM, Michael Zimmermann <sigmaepsilon92@gmail.com> wrote:
> 
> Laszlo, I don't know if I was clear enough but I've been able to load
> drivers from a nested FV without any manual loading or decompressing
> and without any changes to my code.
> 
> All I needed to do was putting this into FVMAIN:
>  FILE FV_IMAGE = $(GUID) {
>    SECTION FV_IMAGE = FVMAINPLATFORM
>  }
> 

Michael,

Load FVs from FVs has been a feature for a very long time in DXE (pre dates edk2) and eventually got added to PEI as well. 

My comments on compression are just generic optimization advice, it should work regardless of the pattern you use. 
My basic compression advice is:
1) Compress as much code together as possible 
2) Files with high entropy like compressed files, and sometime graphics images can make compression not work well. So compressing a compressed file does not usually work well.  
3) Like performance, the only way to know for sure what gets you the best compression is to try things out. 

Thanks,

Andrew Fish

> Thanks
> Michael
> 
> On Thu, Mar 16, 2017 at 1:27 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>> On 03/16/17 13:20, Michael Zimmermann wrote:
>>> So if I understand this correctly there's no bug to fix because
>>> nested, uncompressed FV's are already supported
>> 
>> I'm not sure how to answer this; I guess the answer depends on what
>> meaning you attribute to "supported". In OVMF it works, but it depends
>> on two things: OVMF's SEC driver having access to writeable RAM, and
>> OVMF manually messing with the firmware volumes (locating them,
>> decompressing the outer file, moving stuff around, and so on).
>> 
>> On a physical platform, I guess you can do the same, if you are willing
>> to mess with the FVs manually, and you delay that code until at least
>> the PEI phase sees the permanent RAM installed.
>> 
>> Thanks
>> Laszlo
>> 
>>> and using compressed
>>> ones is bad practice anyway?
>>> 
>>> Thanks
>>> Michael
>>> 
>>> On Thu, Mar 16, 2017 at 12:20 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>>>> On 03/15/17 16:54, Andrew Fish wrote:
>>>>> 
>>>>>> On Mar 15, 2017, at 8:38 AM, Laszlo Ersek <lersek@redhat.com
>>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>> 
>>>>>> On 03/15/17 16:13, Andrew Fish wrote:
>>>>>>> 
>>>>>>>> On Mar 15, 2017, at 8:07 AM, Laszlo Ersek <lersek@redhat.com
>>>>>>>> <mailto:lersek@redhat.com>> wrote:
>>>>>>>> 
>>>>>>>> On 03/15/17 13:23, Michael Zimmermann wrote:
>>>>>>>>> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
>>>>>>>>> Platform specific drivers in a separate, included fdf.
>>>>>>>>> 
>>>>>>>>> I did this:
>>>>>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>>>>>  SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>>>>>    SECTION FV_IMAGE = FVMAIN
>>>>>>>>>    SECTION FV_IMAGE = FVMAINPLATFORM
>>>>>>>>>  }
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> The image builds file and using uefitool I can verify that the new FV
>>>>>>>>> is inside the compressed section.
>>>>>>>>> But none of the drivers gets discovered/loaded and I get 'Protocol not
>>>>>>>>> present!!' errors.
>>>>>>>> 
>>>>>>>> The FVs need to be exposed to the DXE core via FV HOBs. See
>>>>>>>> - 9.8.5 "Firmware Volume HOBs" in Volume 2 of the Platform Init 1.5
>>>>>>>> spec,
>>>>>>>> - and more importantly, 5.7 "Firmware Volume HOB" in Volume 3 of the
>>>>>>>> same.
>>>>>>>> 
>>>>>>>> You can use the BuildFvHob() function for this.
>>>>>>>> 
>>>>>>>> If the firmware volume contains PEIMs (... as well), then it has to be
>>>>>>>> exposed to the PEI core too, I think. I think the
>>>>>>>> PeiServicesInstallFvInfoPpi() function can be used for that. (See 3.3
>>>>>>>> "PEI" in Volume 3 of the PI spec.)
>>>>>>>> 
>>>>>>>> ... I used the PeiFvInitialization() function in
>>>>>>>> OvmfPkg/PlatformPei/Fv.c as a "cheat sheet" for the above.
>>>>>>>> 
>>>>>>> 
>>>>>>> Laszlo,
>>>>>>> 
>>>>>>> I think this case is an FV that is compressed and nested in another
>>>>>>> FV that is discovered. I think the issues is multiple FV Sections in
>>>>>>> an FV file are not currently supported.
>>>>>> 
>>>>>> In OVMF we do the same:
>>>>>> 
>>>>>> FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>>>>>>  SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
>>>>>> PROCESSING_REQUIRED = TRUE {
>>>>>>    #
>>>>>>    # These firmware volumes will have files placed in them uncompressed,
>>>>>>    # and then both firmware volumes will be compressed in a single
>>>>>>    # compression operation in order to achieve better overall
>>>>>> compression.
>>>>>>    #
>>>>>>    SECTION FV_IMAGE = PEIFV
>>>>>>    SECTION FV_IMAGE = DXEFV
>>>>>>  }
>>>>>> }
>>>>>> 
>>>>>> "OvmfPkg/Sec/SecMain.c" decompresses the compressed FFS file into
>>>>>> memory -- which is special to OVMF, i.e. the fact that there is
>>>>>> writeable memory in SEC --, then jumps to the PEI entry point.
>>>>>> PlatformPei then exposes DXEFV to the PEI core (for loading the DXE
>>>>>> core AIUI) and to the DXE core as well (for loading DXE modules).
>>>>>> 
>>>>>> The multiple FV sections in the FFS file are not discovered
>>>>>> automatically. First, the OVMF build process saves the final (=
>>>>>> decompressed) in-memory locations of the nested FVs into PCDs. Second,
>>>>>> in DecompressMemFvs(), the SEC module decompresses the outer,
>>>>>> compressed, FFS file, then locates the embedded FVs manually with
>>>>>> FindFfsSectionInstance(), then moves the decompressed FVs individually
>>>>>> to their pre-recorded locations.
>>>>>> 
>>>>>> In Michael's case, I think the same should be possible -- since FVMAIN
>>>>>> is running for him, the outer, compressed file must already be
>>>>>> decompressed (in full) to some temporary buffer in memory. If he can
>>>>>> find, manually, the second section (with the embedded FVMAINPLATFORM
>>>>>> firmware volume in it) in that area, then he can also expose it to the
>>>>>> DXE core, I believe.
>>>>>> 
>>>>> 
>>>>> Laszlo,
>>>>> 
>>>>> The DXE Core will dispatch FVs with Depex "automagically" but only one
>>>>> FV_IMAGE Section per FV_IMAGE file.
>>>>> 
>>>>> So one workaround would be to add a 2nd FV_IMAGE file that contains the
>>>>> 2nd FV in the 1st FV_IMAGE Section.
>>>>> 
>>>>> Also I'm not sure if the outer FV is compressed, but compressing
>>>>> compressed data usually does not end well as the algorithms leverage
>>>>> entropy.
>>>> 
>>>> I apologize if my previous emails were confusing; I didn't mean to imply
>>>> nested compression. OVMF doesn't do that.
>>>> 
>>>>> It is better to just compress the outer FV and have everything
>>>>> else uncompressed and let it all get compressed together.
>>>> 
>>>> Yes, that's what OVMF does.
>>>> 
>>>> Did you perhaps misread the comment above that I quoted from the FDF?
>>>> 
>>>> "These firmware volumes will have files placed in them *uncompressed*,
>>>> and then both firmware volumes will be compressed in a single
>>>> compression operation in order to achieve better overall compression."
>>>> 
>>>> (Emphasis mine.)
>>>> 
>>>> Thanks!
>>>> Laszlo
>>>> 
>>>>> This also
>>>>> helps as there will be common dictionary entries so compressing as much
>>>>> data as possible in one shot usually works better. So for example if the
>>>>> outer FV is compressed it may have a dictionary entry that represents
>>>>> CopyMem, and it can represent that pattern with a small number of bits.
>>>>> Doing compression multiple times causes duplication as each compressed
>>>>> image has one copy of CopyMem (massive over simplification). Also if you
>>>>> compress a nested FV then the CopyMem pattern does not exist in the
>>>>> compressed data, and since the compressed data is more random it is hard
>>>>> to find patterns to compress, so it hurts the compression of the outer FV.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Andrew Fish
>>>>> 
>>>>>> Thanks
>>>>>> Laszlo
>>>>>> _______________________________________________
>>>>>> edk2-devel mailing list
>>>>>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>>>>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>>>> 
>>>> 
>> 

Re: how to load drivers from additional FV's?

[Andrew Fish]

> On Mar 15, 2017, at 5:23 AM, Michael Zimmermann <sigmaepsilon92@gmail.com> wrote:
> 
> I'm trying to add another FV section FVMAIN_COMPACT so I can keep
> Platform specific drivers in a separate, included fdf.
> 
> I did this:
>  FILE FV_IMAGE = 9E21FD93-9C72-4c15-8C4B-E77F1DB2D792 {
>    SECTION GUIDED EE4E5898-3914-4259-9D6E-DC7BD79403CF
> PROCESSING_REQUIRED = TRUE {
>      SECTION FV_IMAGE = FVMAIN
>      SECTION FV_IMAGE = FVMAINPLATFORM
>    }
>  }
> 
> The image builds file and using uefitool I can verify that the new FV
> is inside the compressed section.
> But none of the drivers gets discovered/loaded and I get 'Protocol not
> present!!' errors.
> 

Michael,

This looks like an implementation restriction. Please file a bug.

CoreProcessFvImageFile() will only grab the 1st instance (hard coded to 0) of EFI_SECTION_FIRMWARE_VOLUME_IMAGE and your example has 2 sections. 

Maybe the CoreProcessFvImageFile() should loop on instance. I assume the device path will be OK, but some one should check on that. 
https://github.com/tianocore/edk2/blob/master/MdeModulePkg/Core/Dxe/Dispatcher/Dispatcher.c#L975


  //
  // Read the first (and only the first) firmware volume section
  //
  SectionType   = EFI_SECTION_FIRMWARE_VOLUME_IMAGE;
  FvHeader      = NULL;
  FvAlignment   = 0;
  Buffer        = NULL;
  BufferSize    = 0;
  AlignedBuffer = NULL;
  Status = Fv->ReadSection (
                 Fv,
                 DriverName,
                 SectionType,
                 0,
                 &Buffer,
                 &BufferSize,
                 &AuthenticationStatus
                 );



Thanks,

Andrew Fish

> Thanks
> Michael
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel