From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 52C12AC13B0 for ; Fri, 3 Nov 2023 16:57:10 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=BfsjSPxlwXoGZ3v2gyNIdQrmbK/agj7/QCs612pIbho=; c=relaxed/simple; d=groups.io; h=Message-ID:Date:MIME-Version:User-Agent:To:Cc:References:From:Subject:In-Reply-To:Feedback-ID:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1699030629; v=1; b=pmQ03XuYBIjIbZYErjFgiSlJFACNf7TvepbdjDxcM/fLqz5CyQK7CkdLeueKRz/IlkgoPV1v LA1R6EDy/R5IemT2oGxQjPseFE2nDllDybA/V1FZC+sZ5gLnnDABLmPoIN9DISub6+CQMYX5uSa 6YzdPuSVOcH9SQaD1XhGwdO4= X-Received: by 127.0.0.2 with SMTP id pqgMYY7687511xxD7fxa4tTm; Fri, 03 Nov 2023 09:57:09 -0700 X-Received: from a7-12.smtp-out.eu-west-1.amazonses.com (a7-12.smtp-out.eu-west-1.amazonses.com [54.240.7.12]) by mx.groups.io with SMTP id smtpd.web10.59187.1699030627551941705 for ; Fri, 03 Nov 2023 09:57:08 -0700 Message-ID: <0102018b961decc1-e615fc66-39cb-4826-9f61-bdabeb43e73e-000000@eu-west-1.amazonses.com> Date: Fri, 3 Nov 2023 16:57:05 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: "Kinney, Michael D" , "devel@edk2.groups.io" , "lersek@redhat.com" , "jlotwo@gmail.com" Cc: Leif Lindholm , Ard Biesheuvel , Sami Mujawar , "Wang, Jian J" , "Gao, Liming" , "Wu, Hao A" , "Ni, Ray" , Pedro Falcato References: <742009f523e8645102f784a3c0df6af870c68804.1698966883.git.jlotwo@gmail.com> From: "Michael Brown" Subject: Re: [edk2-devel] [PATCH] MdeModulePkg/PciHostBridgeDxe: Add readback after final Cfg-Write In-Reply-To: X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on blyat.fensystems.co.uk Feedback-ID: 1.eu-west-1.fspj4M/5bzJ9NLRzJP0PaxRwxrpZqiDQJ1IF94CF2TA=:AmazonSES X-SES-Outgoing: 2023.11.03-54.240.7.12 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mcb30@ipxe.org List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: 4I4xNAmW0QiQTWSswKO0QoO0x7686176AA= Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=pmQ03XuY; dmarc=none; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io On 03/11/2023 16:03, Kinney, Michael D wrote: > PCI Configuration read/write operations are non-posted, so the PCI > Configuration write operation should complete without the need for > an additional transaction. If you are seeing an issue, then it may > be in the implementation of the PciLib/PciSegmentLib that is not > guaranteeing this non-posted behavior. From the PCIe spec: "The ECAM converts memory transactions from the host CPU into Configuration Requests on the PCI Express fabric. This conversion potentially creates ordering problems for the software, because writes to memory addresses are typically posted transactions but writes to Configuration Space are not posted on the PCI Express fabric." My understanding from the above is that the posted write occurs at the=20 level of the CPU performing a memory write to the ECAM window. By the=20 time this write reaches the ECAM and becomes a non-posted write=20 transaction within the downstream PCIe fabric, the CPU has already moved=20 on and is not waiting for completion of the ECAM memory write. Reading back from the PCI configuration register will cause the CPU to=20 wait until the read from the ECAM window has completed, which cannot=20 happen until the corresponding downstream PCI configuration read has=20 completed. Since the CPU will (with appropriate barrier operations) not=20 reorder the memory read ahead of the preceding memory write, the overall=20 effect is to guarantee that the memory write has reached the ECAM, and=20 that the memory write reached the ECAM before the subsequent memory read=20 from the same location. There is an implicit assumption in the above that the ECAM and PCIe=20 fabric will themselves not reorder the PCI configuration read ahead of=20 the PCI configuration write. Unfortunately, on a closer reading of the=20 spec, this may not be a valid assumption: other parts of the PCIe spec=20 state that non-posted transactions (e.g. configuration reads and writes)=20 may be freely reordered. It seems slightly insane that PCIe bridges would be allowed to=20 arbitrarily reorder configuration cycles to downstream devices, but=20 that's what the spec seems to state. The upshot seems to be that: a) software is entirely responsible for ensuring that PCI configuration=20 writes via ECAM have completed, and b) software has no available mechanism for ensuring that PCI=20 configuration writes via ECAM have completed. Ard: any alternative suggestions on ways we can wait for completion,=20 given that even reading back from the PCI configuration register is=20 apparently not guaranteed? Thanks, Michael -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#110646): https://edk2.groups.io/g/devel/message/110646 Mute This Topic: https://groups.io/mt/102354842/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-