Re: [edk2-devel] Additional configuration options on Armada/Cn913x

["Ard Biesheuvel" <ard.biesheuvel@arm.com>]

On 6/11/20 4:33 PM, greg@unrelenting.technology wrote:
> June 11, 2020 5:17 PM, "Ard Biesheuvel" <ard.biesheuvel@arm.com> wrote:
> 
>> On 6/11/20 4:07 PM, greg@unrelenting.technology wrote:
>>
>>> June 11, 2020 4:19 PM, "Ard Biesheuvel" <ard.biesheuvel@arm.com> wrote:
>>> On 6/5/20 5:19 PM, Marcin Wojtas via groups.io wrote:
>>>> Disabling ECAM shift is just a matter of exposing the iATU controls to the OS, right? Why do you
>>>> need to disable it?
>>>
>>> I'm not sure what iATU controls are (and we don't want to do anything in the OS),
>>> but basically the current address is shifted by 0x8000 to only expose the last device to the OS,
>>> to work around the silicon bug (lack of some filtering thing) that causes devices to appear many
>>> many times.
>>> But actually most modern devices (e.g. AMD RX 480, Mellanox CX2) *do not* get duplicated at all,
>>> they show up
>>> in the first position, and this shift moves the memory way past that position and the OS sees
>>> no PCIe devices at all. The only device that was duplicated into all slots for me was a cheap SATA
>>> card.
>>> In my experience whether the device is duplicated seems to correlate with the "Legacy" field
>>> in the UEFI Shell's pci command. IIRC Marcin has explained the actual technical characteristic
>>> of these devices in some mail before. So it might actually be possible to decide whether to do the
>>> shift
>>> automatically at runtime depending on the inserted device (?)
>>> But a setting in the setup menu is easier to do and less magical.
>>> I've just been running with the shift reverted:
>>> https://github.com/myfreeweb/edk2-platforms/commit/36395be2a8707f6d396e07405eb9fe47b64cf964
>>> to make my Radeon GPU work.
>>
>> OK, the shift is definitely a hack, and your assertion that 'most modern devices do not get
>> duplicated at all' does not match my experience, tbh.
>> Are these all devices that support ARI by any chance?
> 
> Mellanox (ConnectX-2) and Intel (82599ES and good old 82576) NICs do.
> 
> But AMD GPUs actually don't! Still the RX480 (POLARIS10) only works without the shift.
> 

Interesting. It all depends on whether the endpoint decodes the device 
field to begin with: it doesn't have to, since the root port at the 
other end should be doing the filtering.

> An older GPU (HD7970 I think) I've tried once was duplicated, but only literally duplicated (into two),
> not into *all* slots.
> 

This is because the iATU granule size is 64k, and so we are only 
exposing 64k of ECAM space to the CPU. (Other implementations of this IP 
that use smaller granule sizes don't need this shift at all)

>> The problem is that the PCIe IP is truly broken, and the lack of a root port means that TLPs get
>> emitted that should never reach the device in the first place, and it is not the job of the device
>> to filter these TLPs, or reason about their own device # in the first place.
> 
> Well, it must be that AMD Radeons do reason about this :)
> and that sadly breaks the offset hack.
> 

This is interesting. I wouldn't expect the endpoint to have any 
awareness of how the ECAM space is exposed to the CPU. How does it fail 
exactly?