From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id CAACE941731 for ; Wed, 1 Nov 2023 12:25:48 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=eMCct2SKqK+WaYZf7PJ7DSj1jF4nKeI5hNzJeLuErLA=; c=relaxed/simple; d=groups.io; h=Message-ID:Date:MIME-Version:User-Agent:To:Cc:References:From:Subject:In-Reply-To:Feedback-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1698841547; v=1; b=lalaEhrmh/0u2ctOfWnRrKnzpDwysqs8HrKT7oPNJ5o+w8hN2+G8gkQFor+8V2I8SBeduHHq 7bUawhaMmSuuUdAFydkw9aFOfJ1J3sGMnHcH63ZrlvUj+CmcnBDygKIpIhzcAxIsWrn4IbpveTF owPqXg58TwQDz13der1qJZag= X-Received: by 127.0.0.2 with SMTP id MA0iYY7687511xNh9UWANUwo; Wed, 01 Nov 2023 05:25:47 -0700 X-Received: from a7-19.smtp-out.eu-west-1.amazonses.com (a7-19.smtp-out.eu-west-1.amazonses.com [54.240.7.19]) by mx.groups.io with SMTP id smtpd.web10.5658.1698841545354307421 for ; Wed, 01 Nov 2023 05:25:45 -0700 Message-ID: <0102018b8ad8c2e7-187ec6ab-01ca-4b47-8c19-aa2748065b06-000000@eu-west-1.amazonses.com> Date: Wed, 1 Nov 2023 12:25:43 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: devel@edk2.groups.io, ardb@kernel.org, pedro.falcato@gmail.com Cc: jlotwo@gmail.com, Leif Lindholm , Sami Mujawar , Ray Ni References: From: "Michael Brown" Subject: Re: [edk2-devel] [RFC] Ordering of Arm PCI ECAM and MMIO operations In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on blyat.fensystems.co.uk Feedback-ID: 1.eu-west-1.fspj4M/5bzJ9NLRzJP0PaxRwxrpZqiDQJ1IF94CF2TA=:AmazonSES X-SES-Outgoing: 2023.11.01-54.240.7.19 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mcb30@ipxe.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: sbokvKDv8J9wtgxP05KHuRSOx7686176AA= Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=lalaEhrm; dmarc=none; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io On 01/11/2023 09:56, Ard Biesheuvel wrote: > On Wed, 1 Nov 2023 at 03:09, Pedro Falcato wrote: >> On Wed, Nov 1, 2023 at 12:40 AM Joe L wrote: >>> Our CMN TRM showcases an example where ECAM and MMIO are two different regions in the HN-I SAM. The implication is that we would expect a DSB between the ECAM write and MMIO read. I'm asking our Open Source Software group to confirm that standard PCIe software is generally expected to be aware of the need for a DSB--but my impression from talking to some of our hardware engineers is that that is indeed the expectation. > > > > 1) as per the architecture (as interpreted by the ARM architects), a > DSB is required to ensure that the side effects of enabling a MMIO BAR > in the PCI config space are sufficiently observable to memory accesses > to that BAR that appear after the PCI config space access in the > program. It's possibly worth mentioning what the PCIe specification requires in terms of ECAM ordering: "As an example, software may wish to configure a device Function’s Base Address register by writing to the device using the ECAM, and then read a location in the memory-mapped range described by this Base Address register. If the software were to issue the memory-mapped read before the ECAM write was completed, it would be possible for the memory-mapped read to be re-ordered and arrive at the device before the Configuration Write Request, thus causing unpredictable results. To avoid this problem, processor and host bridge implementations must ensure that a method exists for the software to determine when the write using the ECAM is completed by the completer." By my reading, the PCIe specification seems to therefore require something stronger than an ordering guarantee: it requires the ability for software to make a standalone determination that the write has *completed*, independent of the existence of any subsequent I/O operations. As a practical example of when this might be relevant: software could be writing to device configuration space to disable bus mastering as part of a reset or shutdown sequence, in order to guarantee that the device will initiate no further DMA operations and that any DMA buffers allocated to the device can be freed and reused. In this situation, there may be no subsequent MMIO read or write to the device, and so there is no way to rely upon an ordering guarantee to satisfy the requirement. Any solution involving ordering guarantees can therefore mask the problem in some situations, but cannot solve it. The PCIe specification does not mandate that any particular mechanism be used, but it does require that the processor and/or host bridge provides *some* mechanism for software to determine that the ECAM write has completed. What mechanism does ARM (or the host bridge) provide to determine completion of an ECAM write? Thanks, Michael -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110477): https://edk2.groups.io/g/devel/message/110477 Mute This Topic: https://groups.io/mt/102310377/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-