Re: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

[Ashish Singhal <ashishsingha@nvidia.com>]

Eugene,

Small question. Did the issue appear after the V4 patch went in? Looking at the code before that patch, we were enabling 64b dma in pci based on capability register already despite of driver supporting only 32b dma.

Thanks
Ashish

Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________
From: Cohen, Eugene <eugene@hp.com>
Sent: Thursday, February 28, 2019 5:11 PM
To: Ashish Singhal; Wu, Hao A; edk2-devel@lists.01.org; Ard Biesheuvel
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Ashish,

Agreed - #2 would be better in the long run since this will have better performance by eliminating the bounce buffering.  My original intent in submitting the patch was to fix the logic the current implementation with minimal impact.

Thanks,

Eugene
From: Ashish Singhal <ashishsingha@nvidia.com>
Sent: Thursday, February 28, 2019 4:58 PM
To: Cohen, Eugene <eugene@hp.com>; Wu, Hao A <hao.a.wu@intel.com>; edk2-devel@lists.01.org; Ard Biesheuvel <ard.biesheuvel@linaro.org>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Eugene,

Thanks for pointing that out. This is a use case we are not covering as of now. I see two options to this:


  1.  Do not enable 64b DMA support in PCI based on V3 as driver does not support V3 64b ADMA. This is a quick fix.
  2.  Enable V3 64b ADMA support to add the missing feature. This will take maybe a day or two and can be done.

Thanks
Ashish

From: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>
Sent: Thursday, February 28, 2019 3:40 PM
To: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org<mailto:ard.biesheuvel@linaro.org>>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Ashish,

I think that code will still fail for our use case.  We are version 3 with 64-bit support so Private->Capability[Slot].SysBus64V3 == 0 will evaluate to FALSE.  Since we are V3 Private->ControllerVersion[Slot] >= SD_MMC_HC_CTRL_VER_400 will also evaluate to FALSE.  Therefore Support64BitDma will still be TRUE resulting in DUAL_ADDRESS_CYCLE being set which disables bounce buffering.

Since no code is in place to do V3 64b DMA we will still hit the same problem, specifically namely that buffers that are not DMAable will be allocated and we will still fail the check here<https://github.com/tianocore/edk2/blob/ece4c1de3e7b2340d351c2054c79ea689a954ed6/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHci.c#L1426>.

Until such time that V3 64b DMA support is in place I believe only the V4 bit should be evaluated.

Eugene

From: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>
Sent: Thursday, February 28, 2019 3:21 PM
To: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org<mailto:ard.biesheuvel@linaro.org>>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Eugene,

Thanks for the explanation. The problem is valid and is more clear to me now. How about we do this:

Instead of:

    if (Private->Capability[Slot].SysBus64V3 == 0 &&
        Private->Capability[Slot].SysBus64V4 == 0) {
      Support64BitDma = FALSE;
    }

What do you think about:

    if ((Private->ControllerVersion[Slot] == SD_MMC_HC_CTRL_VER_300 &&
         Private->Capability[Slot].SysBus64V3 == 0) ||
        (Private->ControllerVersion[Slot] >= SD_MMC_HC_CTRL_VER_400 &&
         Private->Capability[Slot].SysBus64V4 == 0)) {
      Support64BitDma = FALSE;
    }

With this, we would be checking 64b capability based on the version we are using and not for something we may not be using despite of being advertised in the controller.

Thanks
Ashish

From: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>
Sent: Thursday, February 28, 2019 2:59 PM
To: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>; Ard Biesheuvel <ard.biesheuvel@linaro.org<mailto:ard.biesheuvel@linaro.org>>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Ashish,

Ø  Right now, we disable 64b DMA Support in PCI if the controller cannot support 64b DMA in V3 as well as V4.  If either of these support 64b DMA, we do not disable it. In the code, we set Support64BitDma to TRUE by default and change it to FALSE only if any of the controller does not support it in V3 as well as V4. If all controllers support it in V3 or V4 we keep 64b DMA support enabled.

That is precisely the problem.  An SDHC v3 controller might support 64b DMA in V3 but not in V4 mode.  The current code will leave 64b DMA support enabled resulting in the issuing of the PCI DUAL_ADDRESS_CYCLE attribute ( see https://github.com/tianocore/edk2/blob/ece4c1de3e7b2340d351c2054c79ea689a954ed6/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c#L738 ) which then causes buffers to be allocated that cannot be DMAed.

For reference look at this snippet of the NonDiscoverablePciDeviceIo driver: https://github.com/tianocore/edk2/blob/ece4c1de3e7b2340d351c2054c79ea689a954ed6/MdeModulePkg/Bus/Pci/NonDiscoverablePciDeviceDxe/NonDiscoverablePciDeviceIo.c#L622 and you can see that bounce buffering will only occur if DUAL_ADDRESS_CYCLE is clear.

So since we do not have V3 64b DMA (96-bit descriptor) support in place we must not allow the DUAL_ADDRESS_CYCLE attribute to be set or we will fail with this check: https://github.com/tianocore/edk2/blob/ece4c1de3e7b2340d351c2054c79ea689a954ed6/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHci.c#L1426


I've added Ard who updated the driver with DUAL_ADDRESS_CYCLE support.

Eugene


From: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>
Sent: Thursday, February 28, 2019 2:28 PM
To: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Eugene,

We do not have support for V4 64b DMA right now but it can be added later if needed. I am trying to understand the reason behind changing the check from AND to OR. Right now, we disable 64b DMA Support in PCI if the controller cannot support 64b DMA in V3 as well as V4. If either of these support 64b DMA, we do not disable it. In the code, we set Support64BitDma to TRUE by default and change it to FALSE only if any of the controller does not support it in V3 as well as V4. If all controllers support it in V3 or V4 we keep 64b DMA support enabled.

  //
  // Enable 64-bit DMA support in the PCI layer if this controller
  // supports it.
  //
  if (Support64BitDma) {
    Status = PciIo->Attributes (
                      PciIo,
                      EfiPciIoAttributeOperationEnable,
                      EFI_PCI_IO_ATTRIBUTE_DUAL_ADDRESS_CYCLE,
                      NULL
                      );
    if (EFI_ERROR (Status)) {
      DEBUG ((DEBUG_WARN, "SdMmcPciHcDriverBindingStart: failed to enable 64-bit DMA (%r)\n", Status));
    }
  }

Thanks
Ashish

From: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>
Sent: Thursday, February 28, 2019 12:56 PM
To: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Ashish,

Ø  With my change, if any of the controller did not support 64b DMA in V3 as well as V4 capability, we are not enabling it in PCI layer.

The logic is:

   if (Private->Capability[Slot].SysBus64V3 == 0 &&
       Private->Capability[Slot].SysBus64V4 == 0) {
        Support64BitDma = FALSE;
    }

which means that for a SDHC v3 controller you have SysBus64V3=1 and SysBus64V4=0 the FALSE assignment is never done - this is not correct.  Perhaps you intended an OR instead of an AND?  Either way changing this to an || or using my patch is the same logical result because a V3 controller will use 32-bit DMA and V4 controller will use 64-bit DMA (a V4 controller should have the V3 bit set).  I really saw no reason to be checking the V3 bit since the driver was unprepared to do V3 64-bit DMA operations anyways.

Eugene


From: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>
Sent: Thursday, February 28, 2019 12:15 PM
To: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>; Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Hello Eugene,

My patch enabled support for SDHC 4.0 and above in general including support for 64b ADMA descriptor. The check for V3 capability for 64b DMA was already there and similar check was implemented for V4 capability for 64b DMA. Earlier, if any of the V3 controller did not support 64b DMA, we were not enabling it in PCI layer. With my change, if any of the controller did not support 64b DMA in V3 as well as V4 capability, we are not enabling it in PCI layer.

This check in my opinion is better because we only disable 64b DMA PCI support when both V3 and V4 have it disabled.

Thanks
Ashish

-----Original Message-----
From: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>
Sent: Thursday, February 28, 2019 4:24 AM
To: Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Cc: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Hao,

> I remember the commit b5547b9ce97e80c3127682a2a5d4b9bd14af353e from
> Ashish only handles the controllers with version greater or equal to 4.00.

Right - that commit added support for SDHC 4.0 and above. The original driver supported SDHC 3.0 albeit only with SDMA and 32-bit ADMA support.

With that commit two descriptor types are supported the 32-bit ADMA descriptor (SD_MMC_HC_ADMA_32_DESC_LINE which is 64-bits in size) and the V4 64-bit ADMA descriptor (SD_MMC_HC_ADMA_64_DESC_LINE which is 128-bits in size).

However the commit mistakenly added a check for the V3 capability for 64-bit DMA and used it to set the PCI DUAL_ADDRESS_CYCLE attributre which then does not the 32-bit compatible bounce buffer mechanism. Later, when we attempt an ADMA data transfer we hit an ASSERT because the PCI DMA subsystem is not using bounce buffers to provide 32-bit DMA compatible memory. So the patch I submitted simply removes the unnecessary check of the V3 64-bit DMA capability check so the PCI DUAL_ADDRESS_CYCLE attribute is not set allowing 32-bit DMA to succeed on these platforms.

> And the ADMA2 (96-bit Descriptor) mode for V3 controllers is selected
> by setting the 'DMA Select' filed in the Host Control 1 Register to
> 11b. But the currently behavior of the driver is setting the field to
> 10b, which I think will not switch to the ADMA2 (96-bit Descriptor) mode for V3.

Correct, right now for a V3 controller only 32-bit DMA is supported. An enhancement for V3 64-bit ADMA would improve performance on controllers that support that mode by eliminating the bounce buffer and associated memory copies. I think we should file a BZ for SD HCI V3 64-bit ADMA support - if you agree I would be happy to do that.

I should point out that we have done extensive testing of this change on our host controller.

Thanks,

Eugene

---

From: Wu, Hao A <hao.a.wu@intel.com<mailto:hao.a.wu@intel.com>>
Sent: Wednesday, February 27, 2019 8:25 PM
To: Cohen, Eugene <eugene@hp.com<mailto:eugene@hp.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Cc: Ashish Singhal <ashishsingha@nvidia.com<mailto:ashishsingha@nvidia.com>>
Subject: RE: [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC v3 64-bit systems

Loop Ashish in.
Some comments below.

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Cohen, Eugene
> Sent: Wednesday, February 27, 2019 6:59 PM
> To: mailto:edk2-devel@lists.01.org; Wu, Hao A
> Subject: [edk2] [PATCH] MdeModulePkg/SdMmcPciHcDxe: Fix DMA on SDHC
> v3 64-bit systems
>
> The SdMmcPciHcDriverBindingStart function was checking two different
> capability bits in determining whether 64-bit DMA modes were
> supported, one mode is defined in the SDHC version
> 3 specification (using 96-bit descriptors) and another is defined in
> the SDHC version 4 specification (using 128-bit descriptors). Since
> the currently implementation of 64-bit
> ADMA2 only supports the SDHC version 4 implementation it is incorrect
> to check the V3 64-bit capability bit since this will activate V4
> ADMA2 on V3 controllers.

I remember the commit b5547b9ce97e80c3127682a2a5d4b9bd14af353e from Ashish only handles the controllers with version greater or equal to 4.00.

And the ADMA2 (96-bit Descriptor) mode for V3 controllers is selected by setting the 'DMA Select' filed in the Host Control 1 Register to 11b. But the currently behavior of the driver is setting the field to 10b, which I think will not switch to the ADMA2 (96-bit Descriptor) mode for V3.

Maybe there is something I miss here. Could you help to provide some more detail on the issue you met? Thanks.

Best Regards,
Hao Wu

>
> Cc: Hao Wu <mailto:hao.a.wu@intel.com>
> Contributed-under: TianoCore Contribution Agreement 1.1
> Signed-off-by: Eugene Cohen <mailto:eugene@hp.com>
> ---
> MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c
> b/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c
> index b474f8d..5bc91c5 100644
> --- a/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c
> +++ b/MdeModulePkg/Bus/Pci/SdMmcPciHcDxe/SdMmcPciHcDxe.c
> @@ -666,8 +666,7 @@ SdMmcPciHcDriverBindingStart ( // If any of the
> slots does not support 64b system bus // do not enable 64b DMA in the
> PCI layer.
> //
> - if (Private->Capability[Slot].SysBus64V3 == 0 &&
> - Private->Capability[Slot].SysBus64V4 == 0) {
> + if (Private->Capability[Slot].SysBus64V4 == 0) {
> Support64BitDma = FALSE;
> }
>
> --
> 2.7.4
> _______________________________________________
> edk2-devel mailing list
> mailto:edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------