From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=2607:f8b0:4001:c06::236; helo=mail-io0-x236.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=edk2-devel@lists.01.org Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id AEB2E22198F44 for ; Thu, 21 Dec 2017 02:03:04 -0800 (PST) Received: by mail-io0-x236.google.com with SMTP id v186so20476023iod.7 for ; Thu, 21 Dec 2017 02:07:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=AeZPOYQjwuBjooJNJUOKQ1fUSlLVUe3Eju/7PabtrKo=; b=AxcGZSGOkgUXClLzZAMYK/DvfdpHmMsrebeuYU3+5uWH1se4CZjW1VBcgY1WynBeNw FP3NDcRaprK5qDGHRYwgcc/FRzrmi/qRwDzyO1LRR1m2Tl9w9bdyqzoxVcIzTO4ZiY7c b3zFYZoRm4FLNxwfN5R4+NIwCKDOEbsY2ZzgM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=AeZPOYQjwuBjooJNJUOKQ1fUSlLVUe3Eju/7PabtrKo=; b=skWNUbV8ZhrtOjQ8rxHDvDx+YhWSZoYRB0bm7aSLQRL9VmZA8mmrsPJPzJ58rFAOc/ FFhBEipGAMVtA4YXmjGvfLUkSkk5G8vCpelTm9DnnUZNimekwcLCNJMvcNodeuHMl2qf ThrWCOgX4+WbYqHbmVTx3Zv3Nmh1ZFQIeXqbTdmh6APOGY0NoOzQV5hGXN9j4G92CCOI o2Xbcd72Fy47ub9rr5Yrfo6mQDd9xkuRykzvUrvP+k5V7y+aur1iN43n7Kmc2QMaMew4 RtsMySChoss4egAgyN2b9LKHFcV4ipFk7FWzkCILBnM7q2c71sEEqwqiknPP33nYaieX 1Nnw== X-Gm-Message-State: AKGB3mJs4lDx1mld7pxGNsdZeVBqVWf1TTMShuA35FOFurm4WIQEaM+b eYYWcWGywRdvlhQDcy3wGQfcJxFFY9ivAxBFnYnXjg== X-Google-Smtp-Source: ACJfBovf48DkxNF5QN4/4pODJ/+iy87SjtjU0w3rn3NhQJa7/vj2qvhZPXmftKRmqZF2vrCoRPcccMf9sAXQBxEqZRI= X-Received: by 10.107.137.15 with SMTP id l15mr12412066iod.52.1513850872338; Thu, 21 Dec 2017 02:07:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.52.14 with HTTP; Thu, 21 Dec 2017 02:07:51 -0800 (PST) In-Reply-To: <979e60dc-13b3-48fd-fa08-58d570791b0e@Intel.com> References: <1513758078-99534-1-git-send-email-heyi.guo@linaro.org> <20171220151704.GA2482@iwish> <20171221082726.GA100076@SZX1000114654> <20171221091453.GB100076@SZX1000114654> <9272c2e4-cd44-b299-480a-bde9946dc3f4@Intel.com> <979e60dc-13b3-48fd-fa08-58d570791b0e@Intel.com> From: Ard Biesheuvel Date: Thu, 21 Dec 2017 10:07:51 +0000 Message-ID: To: "Ni, Ruiyu" Cc: Eric Dong , Jason Zhang , "edk2-devel@lists.01.org" , Guo Heyi , linaro-uefi , Star Zeng Subject: Re: [RFC] MdeModulePkg/PciHostBridge: Add address translation support X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 10:03:05 -0000 Content-Type: text/plain; charset="UTF-8" On 21 December 2017 at 09:59, Ni, Ruiyu wrote: > On 12/21/2017 5:52 PM, Ard Biesheuvel wrote: >> >> On 21 December 2017 at 09:48, Ni, Ruiyu wrote: >>> >>> On 12/21/2017 5:14 PM, Guo Heyi wrote: >>>> >>>> >>>> On Thu, Dec 21, 2017 at 08:32:37AM +0000, Ard Biesheuvel wrote: >>>>> >>>>> >>>>> On 21 December 2017 at 08:27, Guo Heyi wrote: >>>>>> >>>>>> >>>>>> On Wed, Dec 20, 2017 at 03:26:45PM +0000, Ard Biesheuvel wrote: >>>>>>> >>>>>>> >>>>>>> On 20 December 2017 at 15:17, gary guo wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Dec 20, 2017 at 09:13:58AM +0000, Ard Biesheuvel wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Heyi, >>>>>>>>> >>>>>>>>> On 20 December 2017 at 08:21, Heyi Guo wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> PCIe on some ARM platforms requires address translation, not only >>>>>>>>>> for >>>>>>>>>> legacy IO access, but also for 32bit memory BAR access as well. >>>>>>>>>> There >>>>>>>>>> will be "Address Translation Unit" or something similar in PCI >>>>>>>>>> host >>>>>>>>>> bridges to translation CPU address to PCI address and vice versa. >>>>>>>>>> So >>>>>>>>>> we think it may be useful to add address translation support to >>>>>>>>>> the >>>>>>>>>> generic PCI host bridge driver. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I agree. While unusual on a PC, it is quite common on other >>>>>>>>> architectures to have more complex non 1:1 topologies, which >>>>>>>>> currently >>>>>>>>> require a forked PciHostBridgeDxe driver with local changes >>>>>>>>> applied. >>>>>>>>> >>>>>>>>>> This RFC only contains one minor change to the definition of >>>>>>>>>> PciHostBridgeLib, and there certainly will be a lot of other >>>>>>>>>> changes >>>>>>>>>> to make it work, including: >>>>>>>>>> >>>>>>>>>> 1. Use CPU address for GCD space add and allocate operations, >>>>>>>>>> instead >>>>>>>>>> of PCI address; also IO space will be changed to memory space if >>>>>>>>>> translation exists. >>>>>>>>>> >>>>>>>>> >>>>>>>>> For I/O space, the translation should simply be applied to the I/O >>>>>>>>> range. I don't think it makes sense to use memory space here, given >>>>>>>>> that it is specific to architectures that lack native port I/O. >>>>>>>>> >>>>>>>> >>>>>>>> I made an assumption here that platforms supporting real port IO >>>>>>>> space >>>>>>>> do not need address translation, like IA32 and X64, and port IO >>>>>>>> translation implies the platform does not support real port IO >>>>>>>> space. >>>>>>>> >>>>>>> >>>>>>> This may be a reasonable assumption. But I still think it is better >>>>>>> not to encode any assumptions in the first place. >>>>>>> >>>>>>>> Indeed the assumption is not so "generic", so I'll agree if you >>>>>>>> recommend to support IO to IO translation as well. But I still hope >>>>>>>> to >>>>>>>> have IO to memory translation support in PCI host bridge driver, >>>>>>>> rather than in CPU IO protocol, since the faked IO space might only >>>>>>>> be >>>>>>>> used for PCI host bridge and we may have overlapping IO ranges for >>>>>>>> each host bridge, which is compatible with PCIe specification and >>>>>>>> PCIe >>>>>>>> ACPI description. >>>>>>>> >>>>>>> >>>>>>> That is fine. Under UEFI, these will translate to non-overlapping I/O >>>>>>> spaces in the CPU's view. Under the OS, this could be totally >>>>>>> different. >>>>>>> >>>>>>> >>>>>>> For example, >>>>>>> >>>>>>> RC0 IO 0x0000 .. 0xffff -> CPU 0x00000 .. 0x0ffff >>>>>>> RC1 IO 0x0000 .. 0xffff -> CPU 0x10000 .. 0x1ffff >>>>>>> >>>>>>> This is very similar to how MMIO translation works, and makes I/O >>>>>>> devices behind the host bridges uniquely addressable for drivers. >>>>>>> >>>>>>> For our understanding, could you share the host bridge configuration >>>>>>> that you are targetting? >>>>>> >>>>>> >>>>>> >>>>>> IO translation on one of our platforms is like below: >>>>>> >>>>>> PCI IO space CPU memory space >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >>>>>> ...... >>>>>> >>>>>> The translated addresses may be beyond 32bit address, will this >>>>>> violate IO space limitation? From EDK2 code, I didn't see such >>>>>> limitation for IO space. >>>>>> >>>>> >>>>> The MMIO address will not be used for I/O port addressing by the CPU. >>>>> The MMIO to IO translation is an implementation detail of your CpuIo2 >>>>> protocol implementation. >>>>> >>>>> So there will be two stacked translations, one for PCI I/O to CPU I/O, >>>>> and one for CPU I/O to CPU MMIO. The latter is transparent to the PCI >>>>> host bridge driver. >>>> >>>> >>>> >>>> Yes this should work. >>>> >>>> Hi Star, Eric and Ruiyu, >>>> >>>> Any comments on this RFC? >>> >>> >>> >>> Let me confirm my understanding: >>> The PciHostBridge core driver/library interface changes only >>> take care of the MMIO translation. >>> >>> Heyi you will implement a special CpuIo driver in your >>> platform code to take care of the IO to MMIO translation. >>> >>> But let me confirm, will you need to additional translate >>> the MMIO (translated from IO) to another MMIO using an offset? >>> If yes, will you handle that translation in your CpuIo driver? >>> >> >> Hi Ray, >> >> The issue is that several PCIe root complexes have colliding I/O ranges: > > > Ard, > The IO-MMIO mapping needs CPU support. I am not sure whether IA32 or > x64 supports. > But I guess ARM supports. right? > Yes. Exposing PCI I/O ranges via MMIO translation is specific to the CpuIo2 implementations we have for ARM. > Will all the IO part be implemented in ARM CpuIo2 protocol? > No. The CPU to PCI I/O translation needs to be in PciHostBridgeDxe, because it is in charge of allocating the GCD space, and without translation, those allocations will collide. Also, while perhaps non-existent in reality, it is imaginable that a host bridge could translate port I/O addresses between the two sides of the bridge, similar to how non 1:1 mapped PCI MMIO is handled. So just like MMIO, the port I/O address used by the CPU, and the port I/O address programmed into the device BAR could be subject to translation. This is not solvable in the CpuIo2 protocol, because without translation at the host bridge driver level, a CPU port I/O address is ambiguous: e.g., address 0x1000 may apply to each of the RCs. >> >>>>>> PCI IO space CPU memory space >>>>>> 0x0000 .. 0xffff -> 0xafff0000 .. 0xafffffff >>>>>> (The sizes are always 0x10000 so I will omit the limit for others) >>>>>> 0x0000 .. 0xffff -> 0x8abff0000 >>>>>> 0x0000 .. 0xffff -> 0x8b7ff0000 >> >> >> So the CPU view is different from the PCI view, and to create a single >> CPU I/O space where all I/O port ranges behind all host bridges are >> accessible, we need I/O translation for the CPU. This will result in >> an intermediate representation >> >>>>>> PCI IO space CPU IO space >>>>>> 0x0000 .. 0xffff -> 0x00000 .. 0x0ffff >>>>>> 0x0000 .. 0xffff -> 0x10000 .. 0x1ffff >>>>>> 0x0000 .. 0xffff -> 0x20000 .. 0x2ffff >> >> >> On top of that, given that ARM has no native port I/O instructions, we >> will need to implement MMIO to IO translation, but this can be >> implemented in the CpuIo2 protocol. >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel >> > > > -- > Thanks, > Ray