From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out1.migadu.com (out1.migadu.com [91.121.223.63]) by mx.groups.io with SMTP id smtpd.web11.9069.1591886788379520602 for ; Thu, 11 Jun 2020 07:46:29 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@unrelenting.technology header.s=default header.b=YriWTXdZ; spf=pass (domain: unrelenting.technology, ip: 91.121.223.63, mailfrom: greg@unrelenting.technology) MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=unrelenting.technology; s=default; t=1591886786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9bySKuB0jyX+ngKkZ6VnSF6daKlYj4wcrVsvoM5Ym60=; b=YriWTXdZOGYKPUUKERaQfnBRl12rkQnmPI0OG6qq5Ouq8E8fwRgKqeY2buf5I7HvjV4LRf QDDKbevSS2SOXSD3as7y7iGEX/F0WjqxuevbAiTNyFBOGjiJcjMMKk7+SNdwJH6Vecinw6 vOls7XTPISwdMgrYCn4NPiiYBtpfebI= Date: Thu, 11 Jun 2020 14:46:24 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: greg@unrelenting.technology Message-ID: <5a3d50a00a6596f0a9e5d8de7f5d56b5@unrelenting.technology> Subject: Re: [edk2-devel] Additional configuration options on Armada/Cn913x To: "Ard Biesheuvel" , devel@edk2.groups.io, mw@semihalf.com Cc: "Leif Lindholm" In-Reply-To: References: <7860963d-c540-1577-600c-106d7236f921@arm.com> <05412f93-d4ca-3ae8-744f-6ba7f1ce7e8e@arm.com> <38e05018a4b41becb9d6a37f9defef38@unrelenting.technology> X-Spam-Score: -0.10 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable June 11, 2020 5:38 PM, "Ard Biesheuvel" wrote:= =0A=0A> On 6/11/20 4:33 PM, greg@unrelenting.technology wrote:=0A> =0A>> = June 11, 2020 5:17 PM, "Ard Biesheuvel" wrote:= =0A>> On 6/11/20 4:07 PM, greg@unrelenting.technology wrote:=0A>>> =0A>> = =0A>> June 11, 2020 4:19 PM, "Ard Biesheuvel" wr= ote:=0A>> On 6/5/20 5:19 PM, Marcin Wojtas via groups.io wrote:=0A>> Disa= bling ECAM shift is just a matter of exposing the iATU controls to the OS= , right? Why do you=0A>> need to disable it?=0A>> =0A>> I'm not sure what= iATU controls are (and we don't want to do anything in the OS),=0A>> but= basically the current address is shifted by 0x8000 to only expose the la= st device to the OS,=0A>> to work around the silicon bug (lack of some fi= ltering thing) that causes devices to appear many=0A>> many times.=0A>> B= ut actually most modern devices (e.g. AMD RX 480, Mellanox CX2) *do not* = get duplicated at all,=0A>> they show up=0A>> in the first position, and = this shift moves the memory way past that position and the OS sees=0A>> n= o PCIe devices at all. The only device that was duplicated into all slots= for me was a cheap SATA=0A>> card.=0A>> In my experience whether the dev= ice is duplicated seems to correlate with the "Legacy" field=0A>> in the = UEFI Shell's pci command. IIRC Marcin has explained the actual technical = characteristic=0A>> of these devices in some mail before. So it might act= ually be possible to decide whether to do the=0A>> shift=0A>> automatical= ly at runtime depending on the inserted device (?)=0A>> But a setting in = the setup menu is easier to do and less magical.=0A>> I've just been runn= ing with the shift reverted:=0A>> https://github.com/myfreeweb/edk2-platf= orms/commit/36395be2a8707f6d396e07405eb9fe47b64cf964=0A>> to make my Rade= on GPU work.=0A>>> OK, the shift is definitely a hack, and your assertion= that 'most modern devices do not get=0A>>> duplicated at all' does not m= atch my experience, tbh.=0A>>> Are these all devices that support ARI by = any chance?=0A>> =0A>> Mellanox (ConnectX-2) and Intel (82599ES and good = old 82576) NICs do.=0A>> But AMD GPUs actually don't! Still the RX480 (PO= LARIS10) only works without the shift.=0A> =0A> Interesting. It all depen= ds on whether the endpoint decodes the device field to begin with: it=0A>= doesn't have to, since the root port at the other end should be doing th= e filtering.=0A> =0A>> An older GPU (HD7970 I think) I've tried once was = duplicated, but only literally duplicated (into=0A>> two),=0A>> not into = *all* slots.=0A> =0A> This is because the iATU granule size is 64k, and s= o we are only exposing 64k of ECAM space to the=0A> CPU. (Other implement= ations of this IP that use smaller granule sizes don't need this shift at= all)=0A> =0A>>> The problem is that the PCIe IP is truly broken, and the= lack of a root port means that TLPs get=0A>>> emitted that should never = reach the device in the first place, and it is not the job of the device= =0A>>> to filter these TLPs, or reason about their own device # in the fi= rst place.=0A>> =0A>> Well, it must be that AMD Radeons do reason about t= his :)=0A>> and that sadly breaks the offset hack.=0A> =0A> This is inter= esting. I wouldn't expect the endpoint to have any awareness of how the E= CAM space is=0A> exposed to the CPU. How does it fail exactly?=0A=0AAs I = said: it just doesn't get duplicated, so with the shift, the OS does not = see it at all.=0A=0AI have no idea why exactly it doesn't get duplicated.