TE relocations

TE relocations

[valerij zaporogeci]

Hi everyone. I have a problem with understanding the description of TE
files and its relocations.
The specifications says this:
"17.2 XIP Images
For execute-in-place (XIP) images that do not require relocations,
loading a TE image simply
requires that the loader adjust the image’s entry point from the value
specified in the
EFI_TE_IMAGE_HEADER. For example, if the image (and thus the TE
header) resides at memory
location LoadedImageAddress, then the actual entry for the driver is
computed as follows:
EntryPoint = LoadedImageAddress + sizeof (EFI_TE_IMAGE_HEADER)
+
((EFI_TE_IMAGE_HEADER *)LoadedImageAddress)–>
AddressOfEntryPoint – ((EFI_TE_IMAGE_HEADER *)
LoadedImageAddress)–>StrippedSize;"

But it looks not as simple as "simply" adjusting the only
AddressOfEntryPoint. What "do not require relocations means" here?
Suppose an Image has ImageBase set to X in its header, and it IS at
the address X in XIP memory. Will it need relocations? For PE it
won't, but it seems it is not the case for TE. Since not only
AddressOfEntryPoint is affected, also CodeBase, sections RVAs and all
base relocations too. They all are lying not at their RVA in XIP
memory, because TE has a smaller header and screws up even
FileAlignment of sections. Doesn't it?
The adjustment in the quote uses StrippedSize and it is described as
what was removed from PE:

"The StrippedSize should be set to the number of bytes removed from
the start of the original
image, which will typically include the MS-DOS, COFF, and optional
headers, as well as the section headers."

But does it take into account section alignment by FileAlignment?
(SectionAlignment==FileAlignment in this case). Because for example if
FileAlignment = 200h, then in PE, section for code would be sitting at
n*200h from the beginning of the file and from ImageBase in XIP
memory.
Here is how specification describes PE->TE tranformation process:
"• Create an EFI_TE_IMAGE_HEADER in memory
• Parse the PE/COFF headers in an existing image and extract the
necessary fields to fill in  the EFI_TE_IMAGE_HEADER
• Fill in the signature and stripped size fields in the EFI_TE_IMAGE_HEADER
• Write out the EFI_TE_IMAGE_HEADER to a new binary file
• Write out the contents of the original image, less the stripped
headers, to the output file"

This process would place any section at different file offset
(dispecting alignment) than in PE, and as a result for XIP - different
RVA from ImageBase. For XIP this would mean that such a file requires
relocation, even though its actual ImageBase is exactly what is
written in the file. For non-XIP it would mean that the loader would
need section headers to know what displacement to put the section to
in memory to not break addresses. But they are stripped! Otherwise, if
it just blindly copies what is after TE header, RVAs would be broken,
all addresses would be broken, as for XIP.
In fact this means any TE file loaded at its preferred ImageBase will
need relocations. And for non-XIP. section header cannot be stripped.
Then what TE files would not need relocations?
Did I understand right?
Thank you for answers.

Re: TE relocations

[Andrew Fish]

> On Oct 5, 2016, at 8:08 AM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> 
> Hi everyone. I have a problem with understanding the description of TE
> files and its relocations.

TE is basically PE/COFF with a smaller header. Unlike ELF (Mach-O, etc.) PE/COFF includes the headers in the executable image. The TE header is basically the bits required by EFI and the adjustments (so you can calculate the location the PE/COFF header would have started, since the sections are relative to that in the PE/COFF debug info) needed by the debugger to turn it into a PE/COFF. 

The only reason TE exists is to save size in the ROM for XIP code. The build tools zero out the unused parts of the PE/COFF header so they tend to compress well and TE is not needed. 

> The specifications says this:
> "17.2 XIP Images
> For execute-in-place (XIP) images that do not require relocations,
> loading a TE image simply
> requires that the loader adjust the image’s entry point from the value
> specified in the
> EFI_TE_IMAGE_HEADER. For example, if the image (and thus the TE
> header) resides at memory
> location LoadedImageAddress, then the actual entry for the driver is
> computed as follows:
> EntryPoint = LoadedImageAddress + sizeof (EFI_TE_IMAGE_HEADER)
> +
> ((EFI_TE_IMAGE_HEADER *)LoadedImageAddress)–>
> AddressOfEntryPoint – ((EFI_TE_IMAGE_HEADER *)
> LoadedImageAddress)–>StrippedSize;"
> 
> But it looks not as simple as "simply" adjusting the only
> AddressOfEntryPoint. What "do not require relocations means" here?

PE/COFF images are linked at a specific address and can execute directly if they are run from that address. If a PE/COFF image is loaded/executed from a different address the relocations must be applied to adjust all the absolute references in the image to the new addresses. 

The edk2 links PE/COFF images at zero (or for things like ELF they add a pad for the PE/COFF header ~0x220). When an FV is constructed and XIP (SEC, PEIM, and PEI CORE) images are placed in the FV these PE/COFF images get relocated to the address in the FV (ROM address). 

If the code is XIP then there is no need for relocations as that code can run from any address. 

I think this section is making the point that even if the image does not contain any relocations (all code is PC relative for example) you still have to adjust the EntryPoint to point to ROM address of the entry point. 

> Suppose an Image has ImageBase set to X in its header, and it IS at
> the address X in XIP memory. Will it need relocations?

No. 

> For PE it
> won't, but it seems it is not the case for TE. Since not only
> AddressOfEntryPoint is affected, also CodeBase, sections RVAs and all
> base relocations too. They all are lying not at their RVA in XIP
> memory, because TE has a smaller header and screws up even
> FileAlignment of sections. Doesn't it?

No. The TE header lets you compute the location of the PE/COFF header as this is required to make debuggers happy. So part of the TE construction is making sure the virtual start of the PE/COFF header is aligned correctly. You also need to make sure that the file alignment and section alignment are the same. For most toolchains 0x20 is uses as the file and section alignment to save space.

Note: 0x20 was picked as that was the smallest value we could get to work with Visual Studio "back in the day."

> The adjustment in the quote uses StrippedSize and it is described as
> what was removed from PE:
> 
> "The StrippedSize should be set to the number of bytes removed from
> the start of the original
> image, which will typically include the MS-DOS, COFF, and optional
> headers, as well as the section headers."
> 
> But does it take into account section alignment by FileAlignment?

Yes as I pointed out section alignment and FileAlignment has to be the same for XIP to work. 

> (SectionAlignment==FileAlignment in this case). Because for example if
> FileAlignment = 200h, then in PE, section for code would be sitting at
> n*200h from the beginning of the file and from ImageBase in XIP
> memory.
> Here is how specification describes PE->TE tranformation process:
> "• Create an EFI_TE_IMAGE_HEADER in memory
> • Parse the PE/COFF headers in an existing image and extract the
> necessary fields to fill in  the EFI_TE_IMAGE_HEADER
> • Fill in the signature and stripped size fields in the EFI_TE_IMAGE_HEADER
> • Write out the EFI_TE_IMAGE_HEADER to a new binary file
> • Write out the contents of the original image, less the stripped
> headers, to the output file"
> 
> This process would place any section at different file offset
> (dispecting alignment) than in PE, and as a result for XIP - different
> RVA from ImageBase. For XIP this would mean that such a file requires
> relocation, even though its actual ImageBase is exactly what is
> written in the file. For non-XIP it would mean that the loader would
> need section headers to know what displacement to put the section to
> in memory to not break addresses. But they are stripped! Otherwise, if
> it just blindly copies what is after TE header, RVAs would be broken,
> all addresses would be broken, as for XIP.
> In fact this means any TE file loaded at its preferred ImageBase will
> need relocations. And for non-XIP. section header cannot be stripped.
> Then what TE files would not need relocations?

Any TE that executes from the ROM directly does not need relocations. That is how it works today. As long as SectionAlignment == FileAlignment then everything works from an XIP point of view. 

I guess we could have some bugs in the entire stack (PE/COFF -> TE -> FV) of not honoring the original PE/COFF alignment? But as my Niece would say Meh. The point of TE is to save space in the ROM vs. PE/COFF so if the section alignment is 0x200 there is no point in using TE, you should just use PE/COFF. 

The rules in the FDF file allow you to control if XIP code is TE or PE/COFF. It looks like different platforms pick different rules. 
https://github.com/tianocore/edk2/blob/master/QuarkPlatformPkg/QuarkMin.fdf#L517 <https://github.com/tianocore/edk2/blob/master/QuarkPlatformPkg/QuarkMin.fdf#L517>
https://github.com/tianocore/edk2/blob/master/OvmfPkg/OvmfPkgX64.fdf#L420 <https://github.com/tianocore/edk2/blob/master/OvmfPkg/OvmfPkgX64.fdf#L420>

It is also possible to control on a per module basis if an XIP image in TE or PE/COFF using RuleOverride in the FDF. 

Thanks,

Andrew Fish

PS. I was looking at the TE Image Header and I was reminded that Vincent Zimmer used his initials for the TE signature. I'm sure Vincent will claim this was an ode to Mark Zbikowski, since MZ is the signature in the DOS header :). 
 https://github.com/tianocore/edk2/blob/master/MdePkg/Include/IndustryStandard/PeImage.h#L719 <https://github.com/tianocore/edk2/blob/master/MdePkg/Include/IndustryStandard/PeImage.h#L719>
  UINT16                    Signature;            ///< The signature for TE format = "VZ".


> Did I understand right?
> Thank you for answers.
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: TE relocations

[valerij zaporogeci]

Thank you, Andrew, I guess I understood. PE/TE images get relocated at
FV creation time, they execute directly from FV and they are linked at
where they really will lie in XIP memory. And the alignment value
chosen is not that minimum of 512 byte, mentioned in the PE
specification. Now it's clear. The only thing I still don't get is
that:
Suppose you have your input PE image, which you need to translate in
TE. It's linked at 0. It has a code section and it is lying (in the
file) at the first available 20h aligned offset right after all the
headers. Now you make translation into TE following the specification
algorithm. It says you should strip away all not needed stuff from the
beginning of the PE file, then put TE header instead (28h byte) and
right after it you should place all the remaining PE content. But
constructed this way TE, will break absolute referencies in the code.
Suppose some code references some data, using its address: address =
<ImageBase>:0 + <DataSectionBase> + <SectionOffset>. Where
DataSectionBase is the offset of the beginning of the section from the
ImageBase, thus its RVA. ImageBase remains the same - 0, SectionOffset
too, but _actual_ DataSectionBase, would not be the same for the newly
created TE, because stripping got the data section closer to the file
beginning. Every reference to the data would be broken. Base
relocation at the FV creation won't help, since it changes ImageBase
(from 0 to "ROM address"), it doesn't fix DataSectionBase mismatch. If
after PE->TE translation, "somehow" DataSectionBase remains the same,
then what space saving it is? :)

Re: TE relocations

[Andrew Fish]

> On Oct 5, 2016, at 12:24 PM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> 
> Thank you, Andrew, I guess I understood. PE/TE images get relocated at
> FV creation time, they execute directly from FV and they are linked at
> where they really will lie in XIP memory.

Pedantically speaking PE/TE images are not linked at FV creation time they are relocated to the XIP address. For the most part this is the same thing that EFI does when it loads the PE/TE image into malloced memory. 

> And the alignment value
> chosen is not that minimum of 512 byte, mentioned in the PE
> specification. Now it's clear. The only thing I still don't get is
> that:
> Suppose you have your input PE image, which you need to translate in
> TE. It's linked at 0. It has a code section and it is lying (in the
> file) at the first available 20h aligned offset right after all the
> headers. Now you make translation into TE following the specification
> algorithm. It says you should strip away all not needed stuff from the
> beginning of the PE file, then put TE header instead (28h byte) and
> right after it you should place all the remaining PE content. But
> constructed this way TE, will break absolute referencies in the code.
> Suppose some code references some data, using its address: address =
> <ImageBase>:0 + <DataSectionBase> + <SectionOffset>. Where
> DataSectionBase is the offset of the beginning of the section from the
> ImageBase, thus its RVA. ImageBase remains the same - 0, SectionOffset
> too, but _actual_ DataSectionBase, would not be the same for the newly
> created TE, because stripping got the data section closer to the file
> beginning. Every reference to the data would be broken. Base
> relocation at the FV creation won't help, since it changes ImageBase
> (from 0 to "ROM address"), it doesn't fix DataSectionBase mismatch. If
> after PE->TE translation, "somehow" DataSectionBase remains the same,
> then what space saving it is? :)


I don't think you are Groking the simplicity inherent in the TE design. The image layout of a PE/COFF or TE image is the same.

>From low address to hight address:
IMAGE_HEADER
SECTION_TABLE
.text SECTION
.data SECTION
.reloc SECTION
.debug SECTION

The 1st section after the SECTION_TABLE is going to start on a section alignment boundary, and each subsequent section will start on the aligned boundary. 

>From the PE/COFF point of view it is only the system that cares about the PE/COFF header not really the code that executes. So the image does not really care that it is TE or PE/COFF. I'll use P for PE/COFF header and T for TE header to make my point. 

P
P
P
PT
----
SECTION_TABLE
---
.text SECTION
.data SECTION
.reloc SECTION
.debug SECTION

This layout is the basic reason that debuggers work even though they don't understand TE. When you tell a debugger the start location of a TE image you actually give the debugger the start of P. There is info in the TE header that lets you calculate the location that would have been the start of the PE/COFF header. Conceptually it is the space before the TE image (P but no T) that is being saved in the ROM by using the TE format. 

Feel free to look at the code, https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BasePeCoffLib/BasePeCoff.c <https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BasePeCoffLib/BasePeCoff.c> , that decodes PE/COFF and TE.

Thanks,

Andrew Fish

PS It is a little tricky to actually look at TE images in the build output. All the  modules get constructed as PE/COFF. If you search the Build/ directories for .te files these are a TE Section in the FV, thus they start with a 4 byte section header. If you remove the 1st 4 bytes you get a TE Image. 

Re: TE relocations

[valerij zaporogeci]

Thank you very much, Andrew.
The last question. Regarding that scheme you depicted, which entity
will have ImageBase address when TE image is loaded in the memory? A
'conceptual' stripped Pe headers, or Te header?
Which is placed (even conceptually) at ImageBase address?
P <--- This? (would have been here if presented)
P
P
PT <--- Or this? (really is put at ImageBase)
----
SECTION_TABLE
---
.text SECTION
.data SECTION
.reloc SECTION
.debug SECTION

Because looking at the AddressOfEntryPoint adjustment shows TE header
is placed at ImageBase address. But then again all sections (and the
addresses of referenced symbols) get shifted at the delta value
(StrippedSize - SizeOf(TE_HEADER)). The same as AddressOfEntryPoint.
If ImageBase points to non-existent Pe headers, before TE header, then
everything is fine with adresses in sections, but AddressOfEntryPoint
doesn't need to be adjusted as well contrary to what specification
says.

Re: TE relocations

[Andrew Fish]

> On Oct 5, 2016, at 2:45 PM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> 
> Thank you very much, Andrew.
> The last question. Regarding that scheme you depicted, which entity
> will have ImageBase address when TE image is loaded in the memory? A
> 'conceptual' stripped Pe headers, or Te header?
> Which is placed (even conceptually) at ImageBase address?
> P <--- This? (would have been here if presented)
> P
> P
> PT <--- Or this? (really is put at ImageBase)
> ----
> SECTION_TABLE
> ---
> .text SECTION
> .data SECTION
> .reloc SECTION
> .debug SECTION
> 

The ImageBase is the same for PE/COFF and TE. 
In the code ImageAddress points to the start of T or P (well P can have a DOS header prepended etc). I think a lot of the code operates on ImageAddress and thus needs the adjustment. 

Thanks,

Andrew Fish

> Because looking at the AddressOfEntryPoint adjustment shows TE header
> is placed at ImageBase address. But then again all sections (and the
> addresses of referenced symbols) get shifted at the delta value
> (StrippedSize - SizeOf(TE_HEADER)). The same as AddressOfEntryPoint.
> If ImageBase points to non-existent Pe headers, before TE header, then
> everything is fine with adresses in sections, but AddressOfEntryPoint
> doesn't need to be adjusted as well contrary to what specification
> says.

Re: TE relocations

[valerij zaporogeci]

>> The ImageBase is the same for PE/COFF and TE.
>> In the code ImageAddress points to the start of T or P (well P can have a DOS header
>> prepended etc). I think a lot of the code operates on ImageAddress and thus needs the
>> adjustment.

Well, I don't want to abuse your attention. Just last try to explain
the incosistency here I can not resolve.
Suppose we have some imaginable ISA instruction somewhere in code:
LOAD r1, [0x402f04bc]
and 0x402f04bc is the address of some symbol, resolved by linker.
let's parse this address. Let ImageBase be 0x402f0000, and data
section offset be 0x400 and finally data item offset in the section be
0xbc. Data section is also at 0x400 from the file beginning, since
sectionalignment==filealignment, which means the layout is the same.
When it is loaded at 0x402f000, everything works. Data section is at
0x400 from there, and our variable is at 0xbc from the section start.
Now, we make TE from it. Now, the data section in the TE file is NOT
at 0x400 from the file beginning (it is closer). And when (and if) TE
is loaded such that TE header is placed in memory at THE SAME
ImageBase address as the original PE would have been, the referenced
variable will not be at 402f04bc. And the code, referencing address
0x402f4bc, would get something else instead of this variable content.
This is not the case in the reality. But why? The PI specification
recipe, with only AddressOfEntryPoint adjustment and without
adjustment of anything else referenced (in the code) should result in
this incostistency.

Re: TE relocations

[Andrew Fish]

> On Oct 5, 2016, at 4:25 PM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> 
>>> The ImageBase is the same for PE/COFF and TE.
>>> In the code ImageAddress points to the start of T or P (well P can have a DOS header
>>> prepended etc). I think a lot of the code operates on ImageAddress and thus needs the
>>> adjustment.
> 
> Well, I don't want to abuse your attention.

Well I'm doing it off the top of my head so I'm not doing a lot of research. If I get a chance at some point I'll re-read the PI spec and see if is a bug in the specification since I can file an ECR to get it fixed. 

> Just last try to explain
> the incosistency here I can not resolve.
> Suppose we have some imaginable ISA instruction somewhere in code:
> LOAD r1, [0x402f04bc]
> and 0x402f04bc is the address of some symbol, resolved by linker.
> let's parse this address. Let ImageBase be 0x402f0000, and data
> section offset be 0x400 and finally data item offset in the section be
> 0xbc. Data section is also at 0x400 from the file beginning, since
> sectionalignment==filealignment, which means the layout is the same.
> When it is loaded at 0x402f000, everything works. Data section is at
> 0x400 from there, and our variable is at 0xbc from the section start.
> Now, we make TE from it. Now, the data section in the TE file is NOT
> at 0x400 from the file beginning (it is closer). And when (and if) TE
> is loaded such that TE header is placed in memory at THE SAME
> ImageBase address as the original PE would have been, the referenced
> variable will not be at 402f04bc. And the code, referencing address
> 0x402f4bc, would get something else instead of this variable content.
> This is not the case in the reality. But why? The PI specification
> recipe, with only AddressOfEntryPoint adjustment and without
> adjustment of anything else referenced (in the code) should result in
> this incostistency.

Quick answer is sectionalignment==filealignment for XIP is from the PE/COFF image point of view. From a TE point of view the FileAlignment has to get adjusted. But please remember TE is really just a shortened version of the PE/COFF header, so other than references to those header values it is still the PE/COFF image. 

So I have to ask why are you so interested in TE?

Thanks,

Andrew Fish


> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: TE relocations

[valerij zaporogeci]

Thank you.

 > So I have to ask why are you so interested in TE?

I wanted to use it for the Pei Core file and Peim's. One of the
machines I want to target my UEFI fw on has only 14KB SRAM available,
so I thought it would be nice there. But ok, that machine is MIPS SBC
and I did nothing yet for it. :) Other ARM targets have more SRAM. By
the way, are there chances that UEFI specification will add official
support for the MIPS architecture?

Re: TE relocations

[Gao, Liming]

If TE is loaded such that TE header is placed in memory at THE SAME ImageBase address as the original PE would have been,  its mapped original PE ImageBase will be adjusted to the begin of the TE image + sizeof (EFI_TE_IMAGE_HEADER) - StrippedSize. Then, its mapped PE image will be relocated based on this address. So, there is no issue here.  You can see the code edk2\MdeModulePkg\Core\Pei\Image\Image.c line 474. When load TeImage, the allocated memory address is for PE ImageBase, then PE ImageBase will skip the reserved space for the stripped PeHeader and be changed to TE image base. 

As Andrew point, some platform has enabled TE image for PeiCore and PEIM. In build process, GenFw tool will be used to convert PE image to TE image, GenFfs tool integrate TE image into FFS, and GenFv tool will combine all FFS into FV image. GenFv tool will rebase all TE/PE image as their address in FV image to make them become XIP. In this process, GenFw will check whether PE image has the same section and file alignment. If not, GenFw can't convert it to TE image. 

Thanks
Liming
> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Andrew Fish
> Sent: Thursday, October 06, 2016 8:08 AM
> To: valerij zaporogeci <vlrzprgts@gmail.com>
> Cc: edk2-devel <edk2-devel@lists.01.org>
> Subject: Re: [edk2] TE relocations
> 
> 
> > On Oct 5, 2016, at 4:25 PM, valerij zaporogeci <vlrzprgts@gmail.com> wrote:
> >
> >>> The ImageBase is the same for PE/COFF and TE.
> >>> In the code ImageAddress points to the start of T or P (well P can have a
> DOS header
> >>> prepended etc). I think a lot of the code operates on ImageAddress and
> thus needs the
> >>> adjustment.
> >
> > Well, I don't want to abuse your attention.
> 
> Well I'm doing it off the top of my head so I'm not doing a lot of research. If I
> get a chance at some point I'll re-read the PI spec and see if is a bug in the
> specification since I can file an ECR to get it fixed.
> 
> > Just last try to explain
> > the incosistency here I can not resolve.
> > Suppose we have some imaginable ISA instruction somewhere in code:
> > LOAD r1, [0x402f04bc]
> > and 0x402f04bc is the address of some symbol, resolved by linker.
> > let's parse this address. Let ImageBase be 0x402f0000, and data
> > section offset be 0x400 and finally data item offset in the section be
> > 0xbc. Data section is also at 0x400 from the file beginning, since
> > sectionalignment==filealignment, which means the layout is the same.
> > When it is loaded at 0x402f000, everything works. Data section is at
> > 0x400 from there, and our variable is at 0xbc from the section start.
> > Now, we make TE from it. Now, the data section in the TE file is NOT
> > at 0x400 from the file beginning (it is closer). And when (and if) TE
> > is loaded such that TE header is placed in memory at THE SAME
> > ImageBase address as the original PE would have been, the referenced
> > variable will not be at 402f04bc. And the code, referencing address
> > 0x402f4bc, would get something else instead of this variable content.
> > This is not the case in the reality. But why? The PI specification
> > recipe, with only AddressOfEntryPoint adjustment and without
> > adjustment of anything else referenced (in the code) should result in
> > this incostistency.
> 
> Quick answer is sectionalignment==filealignment for XIP is from the PE/COFF
> image point of view. From a TE point of view the FileAlignment has to get
> adjusted. But please remember TE is really just a shortened version of the
> PE/COFF header, so other than references to those header values it is still
> the PE/COFF image.
> 
> So I have to ask why are you so interested in TE?
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> > _______________________________________________
> > edk2-devel mailing list
> > edk2-devel@lists.01.org
> > https://lists.01.org/mailman/listinfo/edk2-devel
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel