From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by mx.groups.io with SMTP id smtpd.web11.2343.1615638113832979413 for ; Sat, 13 Mar 2021 04:21:54 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@measite-de.20150623.gappssmtp.com header.s=20150623 header.b=b0Drzuku; spf=none, err=permanent DNS error (domain: measite.de, ip: 209.85.218.48, mailfrom: treffer@measite.de) Received: by mail-ej1-f48.google.com with SMTP id ci14so58299465ejc.7 for ; Sat, 13 Mar 2021 04:21:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=measite-de.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=0qQdlUReSAvGCymaveL4+z9NiNEqTbSJhLXWEpGIVsQ=; b=b0Drzukuoe2WPe2BBDSBpF4wjhuZ0WwrEGkoSCqaif7Q3y1lhd98XFuUhLD7dEyFJD eP7uDhxEjsSHmt8ScAAfus7zkmQ1VLLSYyl3UX0pwhajWXPC2cRCw8NK7Ge2QksEY7fh N9U5uOcKLmXQY1oqskzpvEVu1zzeOm5UB2GhRhZabO4iYT6YbDua533i9l7EcvsV/SmD 8pQhVhy4lPBj7V2GQ5Za7qwatGjjPWjMCMYiabx/Qu9so7C8ocEnWAs7vE7m60ahiTxg V9weVs2vZQP3eK2/U4Z0zgnjrO+geAswdt6efRdFv+xWF44OIgbC8z5RPZrLNQaGxBKs +TYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=0qQdlUReSAvGCymaveL4+z9NiNEqTbSJhLXWEpGIVsQ=; b=W82JE8tNK7ZeLfAeouXKm304g/Qj4DEMQHuzvvKHXRcdozl2OY9rIkrDCnjGW0B6I6 xHI0YwgdZ4dwAVaK/Vwgn+BMNJBurCdIqI636xC98zY7PFwyWEpiqxr+aXqRIb7iEzGe hWpL+20EHCvJJ8p8Jr0bs/a3Uf6Vup9TEGHrUklUlmEN86jdY3Qb8unKGe0emrDJfCqa A/MWg0Pfe42kEcAVR6wNlauRH0IT9Egy1ooNiyxzZMWZX3iyZAmh8DrtjEGHI5X2hlmy QiDsff4S8t/6Z/hXvMA5cAPsPPVkFbgvFljNm/NAxwtdqYYNeTrdaoIWi1xyj4ej22sv KVMw== X-Gm-Message-State: AOAM533RWeNxZMDpUZMO+8OgO7CSLUDjD69vFlv0dqNXHyXWuFcxWepb LH3tCZW7fvdWNR0xBMULhXq4WQ== X-Google-Smtp-Source: ABdhPJySCUS7WN9/oMYZeDwOmF6Up0CN6b/nCWMHwWWtTT+LQQQ6gJHrQB+34fGAgHCOsJR8BkEGsw== X-Received: by 2002:a17:906:f9cc:: with SMTP id lj12mr13405330ejb.544.1615638112233; Sat, 13 Mar 2021 04:21:52 -0800 (PST) Return-Path: Received: from ?IPv6:2003:d2:6708:9100:ea6e:fc46:fefb:bd8? (p200300d267089100ea6efc46fefb0bd8.dip0.t-ipconnect.de. [2003:d2:6708:9100:ea6e:fc46:fefb:bd8]) by smtp.gmail.com with ESMTPSA id g25sm4370815edp.95.2021.03.13.04.21.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 13 Mar 2021 04:21:51 -0800 (PST) Subject: Re: [edk2-platforms][PATCH 1/1] Silicon/Broadcom/Bcm27xx: Allow more than one device on pcie busses >1 To: Jeremy Linton , devel@edk2.groups.io Cc: Pete Batard , Leif Lindholm , Ard Biesheuvel , "Andrei Warkentin (awarkentin@vmware.com)" , Samer El-Haj-Mahmoud References: <74065f91-c57c-522f-055e-ff0a09463a17@arm.com> From: treffer@measite.de Message-ID: Date: Sat, 13 Mar 2021 13:21:48 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <74065f91-c57c-522f-055e-ff0a09463a17@arm.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US On 13.03.21 06:08, Jeremy Linton wrote: > Hi, > > > First, thanks for the patches, this really helps! > > On 3/12/21 12:32 PM, René Treffer wrote: >> There is only a single pcie port on the bcm2711 so limiting the >> number of >> devices to 1 worked as long as there is no way to add a pcie switch. >> >> On the compute module 4 it is possible to add a pcie switch (tested with >> asm1184e) which adds 5 new pcie busses. >> >> In the current state the pci enumeration fails for the pcie switch >> internal bus (bus 2, device 1,3,5,7). The root port gets configured with >> subordniate=0x2, blocking e.g. a booting linux from discovering devices >> behind the switch. >> >> Devices behind the switch work after lifting the device limit on busses >> other than 0 and 1. >> >> Signed-off-by: René Treffer >> --- >>   .../Library/Bcm2711PciSegmentLib/PciSegmentLib.c   | 14 +++++++------- >>   1 file changed, 7 insertions(+), 7 deletions(-) >> >> diff --git >> a/Silicon/Broadcom/Bcm27xx/Library/Bcm2711PciSegmentLib/PciSegmentLib.c >> b/Silicon/Broadcom/Bcm27xx/Library/Bcm2711PciSegmentLib/PciSegmentLib.c >> index 44ce3b4b99..4af9374d23 100644 >> --- >> a/Silicon/Broadcom/Bcm27xx/Library/Bcm2711PciSegmentLib/PciSegmentLib.c >> +++ >> b/Silicon/Broadcom/Bcm27xx/Library/Bcm2711PciSegmentLib/PciSegmentLib.c >> @@ -78,6 +78,7 @@ PciSegmentLibGetConfigBase ( >>     UINT64        Base; >>     UINT64        Offset; >>     UINT32        Dev; >> +  UINT32        Bus; >>       Base = PCIE_REG_BASE; >>     Offset = Address & 0xFFF;         /* Pick off the 4k register >> offset */ >> @@ -89,17 +90,16 @@ PciSegmentLibGetConfigBase ( >>       Base += PCIE_EXT_CFG_DATA; >>       if (mPciSegmentLastAccess != Address) { >>         Dev = EFI_PCI_ADDR_DEV (Address); >> +      Bus = EFI_PCI_ADDR_BUS (Address); >>         /* >> -       * Scan things out directly rather than translating the "bus" to >> a device, etc.. >> -       * only we need to limit each bus to a single device. >> +       * There can only be a single device on bus 1 (downstream of >> root). >> +       * Subsequent busses (behind a PCIe switch) could have more. >>          */ >> -      if (Dev < 1) { >> -          MmioWrite32 (PCIE_REG_BASE + PCIE_EXT_CFG_INDEX, Address); >> -          mPciSegmentLastAccess = Address; >> -      } else { >> -          mPciSegmentLastAccess = 0; >> +      if (Dev > 0 && (Bus == 1 || Bus == 0)) { > > Did you try this with another switch plugged in downstream the first one? I don't have a second switch yet. The topology I tried is: # lspci -vnnt -[0000:00]---00.0-[01-06]----00.0-[02-06]--+-01.0-[03]----00.0  Toshiba Corporation XG4 NVMe SSD Controller [1179:0115]                                            +-03.0-[04]----00.0  Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014]                                            +-05.0-[05]--                                            \-07.0-[06]-- # lspci -nn 00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge [14e4:2711] (rev 20) 01:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 02:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 02:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 02:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 02:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184] 03:00.0 Non-Volatile memory controller [0108]: Toshiba Corporation XG4 NVMe SSD Controller [1179:0115] (rev 01) 04:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) So far I have tried (behind the switch): - NVMe card. Works, even as a boot device after enabling the NVMe package for the RPI4 - USB card. Works. upd72020x-load works for uploading firmware (I have misflashed the ROM though) - SATA controller ASM1062 + 2xJMB575 based (10 ports) - got detected, haven't tried more - LSI HBA, IT mode. Didn't work due to "Resource conflict", breaks NVMe booting / PCIe bus The last setup looks like a bug. The LSI HBA resource conflict shouldn't stop other cards from functioning, right? I can give that another try and upload the debug logs if anyone is interested. The 64MB BAR space default seems to hurt in this setup, but I haven't been able to get more working. Devices ordered for future testing: - An old RADEON HD 3450 256MB (I don't expect that one to work) - An old Intel I350 dual port network card (hoping to test option ROM emulation) - Another PCIe switch (will take 2+ weeks) I hope to test most of that next week and would probably send another patch if these things work as intended. Regards,   René > This looks right, because presumably the first new switch is sending > the UR's properly for its subordinate buses. > > Dumping the segment last access reset is fine too. > > Although, this set has unicode whitespace characters, which presumably > Ard fixed up in the previous patch since I see it there too. > > >>             return 0xFFFFFFFF; >>         } >> +      MmioWrite32 (PCIE_REG_BASE + PCIE_EXT_CFG_INDEX, Address); >> +      mPciSegmentLastAccess = Address; >>       } >>     } >>     return Base + Offset; >> > > So, > > Reviewed-by: Jeremy Linton > > Thanks again!