Mapping of PrpList in NvmExpressDxe

Mapping of PrpList in NvmExpressDxe

[Arka Sharma]

I am wondering what is the reason for mapping the PrpList buffer with
EfiPciIoOperationBusMasterCommonBuffer, as host will fill the Prp
entries and after updating the submission queue doorbell device will
start processing the command and fetch the Prp entries. So I am
thinking the Prplist buffer could have been mapped as
EfiPciIoOperationBusMasterRead. Is there any reason for mapping it as
CommonBuffer that I am not able to figure out ?

Regards,
Arka

Re: Mapping of PrpList in NvmExpressDxe

[Andrew Fish]

> On Feb 17, 2017, at 5:54 AM, Arka Sharma <arka.sw1988@gmail.com> wrote:
> 
> I am wondering what is the reason for mapping the PrpList buffer with
> EfiPciIoOperationBusMasterCommonBuffer, as host will fill the Prp
> entries and after updating the submission queue doorbell device will
> start processing the command and fetch the Prp entries. So I am
> thinking the Prplist buffer could have been mapped as
> EfiPciIoOperationBusMasterRead. Is there any reason for mapping it as
> CommonBuffer that I am not able to figure out ?
> 

Arka,

Good question. Historically there have been a lot of bugs in DMA code in EFI. The reason being if you don't follow the rules the code still works since DMA is so coherent in hardware on an x86 PC. This is not the case for a lot of ARM platforms. 

There is a good overview of UEFI DMA operations in the UEFI Spec PCI Bus Support Chapter. 

DMA Bus Master Read Operation
• Call Map() for EfiPciOperationBusMasterRead or EfiPciOperationBusMasterRead64.
• Program the DMA Bus Master with the DeviceAddress returned by Map().
• Start the DMA Bus Master.
• Wait for DMA Bus Master to complete the read operation.
• Call Unmap().

DMA Bus Master Write Operation
• Call Map() for EfiPciOperationBusMasterWrite or EfiPciOperationBusMasterRead64.
• Program the DMA Bus Master with the DeviceAddress returned by Map().
• Start the DMA Bus Master.
• Wait for DMA Bus Master to complete the write operation.
• Perform a PCI controller specific read transaction to flush all PCI write buffers (See PCI Specification Section 3.2.5.2) .
• Call Flush().
• Call Unmap().

DMA Bus Master Common Buffer Operation
• Call AllocateBuffer() to allocate a common buffer.
• Call Map() for EfiPciOperationBusMasterCommonBuffer or EfiPciOperationBusMasterCommonBuffer64.
• Program the DMA Bus Master with the DeviceAddress returned by Map().
• The common buffer can now be accessed equally by the processor and the DMA bus master.
• Call Unmap().
• Call FreeBuffer().

So to answer your question. The Read and Write operations are one shot on the buffer, while Common Buffer is a buffer that is DMA coherent and can be reused. Thats probably why Common Buffer is used by the NVMe driver. 

Basically the common usage for the Read and Write mappings are the caller passing a buffer (like a block on the disk). The Queues that run DMA are common buffer as the PCI hardware and the CPU both need to access them intermittently. To be clear for the read and write case the CPU only has a coherent view of the buffer after the Unmap() is called. 

I think the only way you can enforce most of the UEFI driver DMA rules on x86 is to turn on an IOMMU that would cause faults if you don't follow the rules. Basically you have the IOMMU fault on DMA transactions to a buffer that is not following the rules above. 

Thanks,

Andrew Fish

> Regards,
> Arka
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Mapping of PrpList in NvmExpressDxe

[Arka Sharma]

Thanks Andrew for your response. Actually my question related to
EfiPciOperationBusMasterCommonBuffer was only in case of PrpList
buffer which is allocated and mapped in NvmeCreatePrpList function. In
case of queues the mapping with EfiPciOperationBusMasterCommonBuffer
is understood as in case of Completion queue the device will post the
response of a command and driver will also poll the phase tag to
detect a new completion entry so coherent buffer is needed. But in
case of PrpList buffer such simultaneous access is not done. The host
will update the prp entries and device will pick them up only after
host update submission queue doorbell. So my question is in case of
particularly PrpList buffer why it is mapped as
EfiPciOperationBusMasterCommonBuffer, where as per my understanding
EfiPciOperationBusMasterRead would have sufficed.

Regards,
Arka

On Fri, Feb 17, 2017 at 10:13 PM, Andrew Fish <afish@apple.com> wrote:
>
>> On Feb 17, 2017, at 5:54 AM, Arka Sharma <arka.sw1988@gmail.com> wrote:
>>
>> I am wondering what is the reason for mapping the PrpList buffer with
>> EfiPciIoOperationBusMasterCommonBuffer, as host will fill the Prp
>> entries and after updating the submission queue doorbell device will
>> start processing the command and fetch the Prp entries. So I am
>> thinking the Prplist buffer could have been mapped as
>> EfiPciIoOperationBusMasterRead. Is there any reason for mapping it as
>> CommonBuffer that I am not able to figure out ?
>>
>
> Arka,
>
> Good question. Historically there have been a lot of bugs in DMA code in EFI. The reason being if you don't follow the rules the code still works since DMA is so coherent in hardware on an x86 PC. This is not the case for a lot of ARM platforms.
>
> There is a good overview of UEFI DMA operations in the UEFI Spec PCI Bus Support Chapter.
>
> DMA Bus Master Read Operation
> • Call Map() for EfiPciOperationBusMasterRead or EfiPciOperationBusMasterRead64.
> • Program the DMA Bus Master with the DeviceAddress returned by Map().
> • Start the DMA Bus Master.
> • Wait for DMA Bus Master to complete the read operation.
> • Call Unmap().
>
> DMA Bus Master Write Operation
> • Call Map() for EfiPciOperationBusMasterWrite or EfiPciOperationBusMasterRead64.
> • Program the DMA Bus Master with the DeviceAddress returned by Map().
> • Start the DMA Bus Master.
> • Wait for DMA Bus Master to complete the write operation.
> • Perform a PCI controller specific read transaction to flush all PCI write buffers (See PCI Specification Section 3.2.5.2) .
> • Call Flush().
> • Call Unmap().
>
> DMA Bus Master Common Buffer Operation
> • Call AllocateBuffer() to allocate a common buffer.
> • Call Map() for EfiPciOperationBusMasterCommonBuffer or EfiPciOperationBusMasterCommonBuffer64.
> • Program the DMA Bus Master with the DeviceAddress returned by Map().
> • The common buffer can now be accessed equally by the processor and the DMA bus master.
> • Call Unmap().
> • Call FreeBuffer().
>
> So to answer your question. The Read and Write operations are one shot on the buffer, while Common Buffer is a buffer that is DMA coherent and can be reused. Thats probably why Common Buffer is used by the NVMe driver.
>
> Basically the common usage for the Read and Write mappings are the caller passing a buffer (like a block on the disk). The Queues that run DMA are common buffer as the PCI hardware and the CPU both need to access them intermittently. To be clear for the read and write case the CPU only has a coherent view of the buffer after the Unmap() is called.
>
> I think the only way you can enforce most of the UEFI driver DMA rules on x86 is to turn on an IOMMU that would cause faults if you don't follow the rules. Basically you have the IOMMU fault on DMA transactions to a buffer that is not following the rules above.
>
> Thanks,
>
> Andrew Fish
>
>> Regards,
>> Arka
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: Mapping of PrpList in NvmExpressDxe

[Andrew Fish]

> On Feb 17, 2017, at 9:30 AM, Arka Sharma <arka.sw1988@gmail.com> wrote:
> 
> Thanks Andrew for your response. Actually my question related to
> EfiPciOperationBusMasterCommonBuffer was only in case of PrpList
> buffer which is allocated and mapped in NvmeCreatePrpList function. In
> case of queues the mapping with EfiPciOperationBusMasterCommonBuffer
> is understood as in case of Completion queue the device will post the
> response of a command and driver will also poll the phase tag to
> detect a new completion entry so coherent buffer is needed. But in
> case of PrpList buffer such simultaneous access is not done. The host
> will update the prp entries and device will pick them up only after
> host update submission queue doorbell. So my question is in case of
> particularly PrpList buffer why it is mapped as
> EfiPciOperationBusMasterCommonBuffer, where as per my understanding
> EfiPciOperationBusMasterRead would have sufficed.
> 

Arka,

The EfiPciOperationBusMasterRead implies the driver would have to call Map() prior to every DMA, and Unmap() after every DMA before the CPU looks at the data. For EfiPciOperationBusMasterCommonBuffer the Map() is part of the malloc, and the unmap is part of the free of the buffer. So in general EfiPciOperationBusMasterCommonBuffer is more efficient and there are less coherency race conditions, so it is easier to write the code. 

On systems where DMA is not coherent a Map()/Unmap() call may have to do caches flushes and other operations on every call (if you miss a call you can get data corruption in your buffer), while a Common Buffer might just be uncached memory. 

Thanks,

Andrew Fish

> Regards,
> Arka
> 
> On Fri, Feb 17, 2017 at 10:13 PM, Andrew Fish <afish@apple.com> wrote:
>> 
>>> On Feb 17, 2017, at 5:54 AM, Arka Sharma <arka.sw1988@gmail.com> wrote:
>>> 
>>> I am wondering what is the reason for mapping the PrpList buffer with
>>> EfiPciIoOperationBusMasterCommonBuffer, as host will fill the Prp
>>> entries and after updating the submission queue doorbell device will
>>> start processing the command and fetch the Prp entries. So I am
>>> thinking the Prplist buffer could have been mapped as
>>> EfiPciIoOperationBusMasterRead. Is there any reason for mapping it as
>>> CommonBuffer that I am not able to figure out ?
>>> 
>> 
>> Arka,
>> 
>> Good question. Historically there have been a lot of bugs in DMA code in EFI. The reason being if you don't follow the rules the code still works since DMA is so coherent in hardware on an x86 PC. This is not the case for a lot of ARM platforms.
>> 
>> There is a good overview of UEFI DMA operations in the UEFI Spec PCI Bus Support Chapter.
>> 
>> DMA Bus Master Read Operation
>> • Call Map() for EfiPciOperationBusMasterRead or EfiPciOperationBusMasterRead64.
>> • Program the DMA Bus Master with the DeviceAddress returned by Map().
>> • Start the DMA Bus Master.
>> • Wait for DMA Bus Master to complete the read operation.
>> • Call Unmap().
>> 
>> DMA Bus Master Write Operation
>> • Call Map() for EfiPciOperationBusMasterWrite or EfiPciOperationBusMasterRead64.
>> • Program the DMA Bus Master with the DeviceAddress returned by Map().
>> • Start the DMA Bus Master.
>> • Wait for DMA Bus Master to complete the write operation.
>> • Perform a PCI controller specific read transaction to flush all PCI write buffers (See PCI Specification Section 3.2.5.2) .
>> • Call Flush().
>> • Call Unmap().
>> 
>> DMA Bus Master Common Buffer Operation
>> • Call AllocateBuffer() to allocate a common buffer.
>> • Call Map() for EfiPciOperationBusMasterCommonBuffer or EfiPciOperationBusMasterCommonBuffer64.
>> • Program the DMA Bus Master with the DeviceAddress returned by Map().
>> • The common buffer can now be accessed equally by the processor and the DMA bus master.
>> • Call Unmap().
>> • Call FreeBuffer().
>> 
>> So to answer your question. The Read and Write operations are one shot on the buffer, while Common Buffer is a buffer that is DMA coherent and can be reused. Thats probably why Common Buffer is used by the NVMe driver.
>> 
>> Basically the common usage for the Read and Write mappings are the caller passing a buffer (like a block on the disk). The Queues that run DMA are common buffer as the PCI hardware and the CPU both need to access them intermittently. To be clear for the read and write case the CPU only has a coherent view of the buffer after the Unmap() is called.
>> 
>> I think the only way you can enforce most of the UEFI driver DMA rules on x86 is to turn on an IOMMU that would cause faults if you don't follow the rules. Basically you have the IOMMU fault on DMA transactions to a buffer that is not following the rules above.
>> 
>> Thanks,
>> 
>> Andrew Fish
>> 
>>> Regards,
>>> Arka
>>> _______________________________________________
>>> edk2-devel mailing list
>>> edk2-devel@lists.01.org
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>