From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mx.groups.io with SMTP id smtpd.web10.8843.1591886332015072215 for ; Thu, 11 Jun 2020 07:38:52 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: arm.com, ip: 217.140.110.172, mailfrom: ard.biesheuvel@arm.com) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 980311F1; Thu, 11 Jun 2020 07:38:51 -0700 (PDT) Received: from [192.168.1.69] (unknown [10.37.8.184]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99EF13F6CF; Thu, 11 Jun 2020 07:38:50 -0700 (PDT) Subject: Re: [edk2-devel] Additional configuration options on Armada/Cn913x To: greg@unrelenting.technology, devel@edk2.groups.io, mw@semihalf.com Cc: Leif Lindholm References: <7860963d-c540-1577-600c-106d7236f921@arm.com> <05412f93-d4ca-3ae8-744f-6ba7f1ce7e8e@arm.com> <38e05018a4b41becb9d6a37f9defef38@unrelenting.technology> From: "Ard Biesheuvel" Message-ID: Date: Thu, 11 Jun 2020 16:38:48 +0200 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <38e05018a4b41becb9d6a37f9defef38@unrelenting.technology> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit On 6/11/20 4:33 PM, greg@unrelenting.technology wrote: > June 11, 2020 5:17 PM, "Ard Biesheuvel" wrote: > >> On 6/11/20 4:07 PM, greg@unrelenting.technology wrote: >> >>> June 11, 2020 4:19 PM, "Ard Biesheuvel" wrote: >>> On 6/5/20 5:19 PM, Marcin Wojtas via groups.io wrote: >>>> Disabling ECAM shift is just a matter of exposing the iATU controls to the OS, right? Why do you >>>> need to disable it? >>> >>> I'm not sure what iATU controls are (and we don't want to do anything in the OS), >>> but basically the current address is shifted by 0x8000 to only expose the last device to the OS, >>> to work around the silicon bug (lack of some filtering thing) that causes devices to appear many >>> many times. >>> But actually most modern devices (e.g. AMD RX 480, Mellanox CX2) *do not* get duplicated at all, >>> they show up >>> in the first position, and this shift moves the memory way past that position and the OS sees >>> no PCIe devices at all. The only device that was duplicated into all slots for me was a cheap SATA >>> card. >>> In my experience whether the device is duplicated seems to correlate with the "Legacy" field >>> in the UEFI Shell's pci command. IIRC Marcin has explained the actual technical characteristic >>> of these devices in some mail before. So it might actually be possible to decide whether to do the >>> shift >>> automatically at runtime depending on the inserted device (?) >>> But a setting in the setup menu is easier to do and less magical. >>> I've just been running with the shift reverted: >>> https://github.com/myfreeweb/edk2-platforms/commit/36395be2a8707f6d396e07405eb9fe47b64cf964 >>> to make my Radeon GPU work. >> >> OK, the shift is definitely a hack, and your assertion that 'most modern devices do not get >> duplicated at all' does not match my experience, tbh. >> Are these all devices that support ARI by any chance? > > Mellanox (ConnectX-2) and Intel (82599ES and good old 82576) NICs do. > > But AMD GPUs actually don't! Still the RX480 (POLARIS10) only works without the shift. > Interesting. It all depends on whether the endpoint decodes the device field to begin with: it doesn't have to, since the root port at the other end should be doing the filtering. > An older GPU (HD7970 I think) I've tried once was duplicated, but only literally duplicated (into two), > not into *all* slots. > This is because the iATU granule size is 64k, and so we are only exposing 64k of ECAM space to the CPU. (Other implementations of this IP that use smaller granule sizes don't need this shift at all) >> The problem is that the PCIe IP is truly broken, and the lack of a root port means that TLPs get >> emitted that should never reach the device in the first place, and it is not the job of the device >> to filter these TLPs, or reason about their own device # in the first place. > > Well, it must be that AMD Radeons do reason about this :) > and that sadly breaks the offset hack. > This is interesting. I wouldn't expect the endpoint to have any awareness of how the ECAM space is exposed to the CPU. How does it fail exactly?