Re: [edk2-devel] Additional configuration options on Armada/Cn913x

[greg@unrelenting.technology]

June 11, 2020 5:38 PM, "Ard Biesheuvel" <ard.biesheuvel@arm.com> wrote:

> On 6/11/20 4:33 PM, greg@unrelenting.technology wrote:
> 
>> June 11, 2020 5:17 PM, "Ard Biesheuvel" <ard.biesheuvel@arm.com> wrote:
>> On 6/11/20 4:07 PM, greg@unrelenting.technology wrote:
>>> 
>> 
>> June 11, 2020 4:19 PM, "Ard Biesheuvel" <ard.biesheuvel@arm.com> wrote:
>> On 6/5/20 5:19 PM, Marcin Wojtas via groups.io wrote:
>> Disabling ECAM shift is just a matter of exposing the iATU controls to the OS, right? Why do you
>> need to disable it?
>> 
>> I'm not sure what iATU controls are (and we don't want to do anything in the OS),
>> but basically the current address is shifted by 0x8000 to only expose the last device to the OS,
>> to work around the silicon bug (lack of some filtering thing) that causes devices to appear many
>> many times.
>> But actually most modern devices (e.g. AMD RX 480, Mellanox CX2) *do not* get duplicated at all,
>> they show up
>> in the first position, and this shift moves the memory way past that position and the OS sees
>> no PCIe devices at all. The only device that was duplicated into all slots for me was a cheap SATA
>> card.
>> In my experience whether the device is duplicated seems to correlate with the "Legacy" field
>> in the UEFI Shell's pci command. IIRC Marcin has explained the actual technical characteristic
>> of these devices in some mail before. So it might actually be possible to decide whether to do the
>> shift
>> automatically at runtime depending on the inserted device (?)
>> But a setting in the setup menu is easier to do and less magical.
>> I've just been running with the shift reverted:
>> https://github.com/myfreeweb/edk2-platforms/commit/36395be2a8707f6d396e07405eb9fe47b64cf964
>> to make my Radeon GPU work.
>>> OK, the shift is definitely a hack, and your assertion that 'most modern devices do not get
>>> duplicated at all' does not match my experience, tbh.
>>> Are these all devices that support ARI by any chance?
>> 
>> Mellanox (ConnectX-2) and Intel (82599ES and good old 82576) NICs do.
>> But AMD GPUs actually don't! Still the RX480 (POLARIS10) only works without the shift.
> 
> Interesting. It all depends on whether the endpoint decodes the device field to begin with: it
> doesn't have to, since the root port at the other end should be doing the filtering.
> 
>> An older GPU (HD7970 I think) I've tried once was duplicated, but only literally duplicated (into
>> two),
>> not into *all* slots.
> 
> This is because the iATU granule size is 64k, and so we are only exposing 64k of ECAM space to the
> CPU. (Other implementations of this IP that use smaller granule sizes don't need this shift at all)
> 
>>> The problem is that the PCIe IP is truly broken, and the lack of a root port means that TLPs get
>>> emitted that should never reach the device in the first place, and it is not the job of the device
>>> to filter these TLPs, or reason about their own device # in the first place.
>> 
>> Well, it must be that AMD Radeons do reason about this :)
>> and that sadly breaks the offset hack.
> 
> This is interesting. I wouldn't expect the endpoint to have any awareness of how the ECAM space is
> exposed to the CPU. How does it fail exactly?

As I said: it just doesn't get duplicated, so with the shift, the OS does not see it at all.

I have no idea why exactly it doesn't get duplicated.