title: Linux PCIe DMA Driver (Xilinx XDMA)
body: <p>I am currently working with the Xilinx XDMA driver (see here for source code: <a href="https://www.xilinx.com/support/answers/65444.html" rel="noreferrer" title="XDMA Source">XDMA Source</a>), and am attempting to get it to run (before you ask: I have contacted my technical support point of contact and the Xilinx forum is riddled with people having the same issue). However, I may have found a snag in Xilinx's code that might be a deal breaker for me.  I am hoping there is something that I'm not considering.</p>
  
  <p>First off, there are two primary modes of the driver, AXI-Memory Mapped (AXI-MM) and AXI-Streaming (AXI-ST).  For my particular application, I require AXI-ST, since data will continuously be flowing from the device.</p>
  
  <p>The driver is written to take advantage of scatter-gather lists.  In AXI-MM mode, this works because reads are rather random events (i.e., there isn't a flow of data out of the device, instead the userspace application simply requests data when it needs to).  As such, the DMA transfer is built up, the data is transfered, and the transfer is then torn down.  This is a combination of <code>get_user_pages()</code>, <code>pci_map_sg()</code>, and <code>pci_unmap_sg()</code>.</p>
  
  <p>For AXI-ST, things get weird, and the source code is far from orthodox.  The driver allocates a circular buffer where the data is meant to continuously flow into.  This buffer is generally sized to be somewhat large (mine is set on the order of 32MB), since you want to be able to handle transient events where the userspace application forgot about the driver and can then later work off the incoming data.</p>
  
  <p>Here's where things get wonky... the circular buffer is allocated using <code>vmalloc32()</code> and the pages from that allocation are mapped in the same way as the userspace buffer is in AXI-MM mode (i.e., using the <code>pci_map_sg()</code> interface).  As a result, because the circular buffer is shared between the device and CPU, every <code>read()</code> call requires me to call <code>pci_dma_sync_sg_for_cpu()</code> and <code>pci_dma_sync_sg_for_device()</code>, which absolutely destroys my performance (I can not keep up with the device!), since this works on the entire buffer.  Funny enough, Xilinx never included these sync calls in their code, so I first knew I had a problem when I edited their test script to attempt more than one DMA transfer before exiting and the resulting data buffer was corrupted.</p>
  
  <p>As a result, I'm wondering how I can fix this.  I've considered rewriting the code to build up my own buffer allocated using <code>pci_alloc_consistent()/dma_alloc_coherent()</code>, but this is easier said than done.  Namely, the code is architected to assume using scatter-gather lists everywhere (there appears to be a strange, proprietary mapping between the scatter-gather list and the memory descriptors that the FPGA understands).</p>
  
  <p>Are there any other API calls I should be made aware of?  Can I use the "single" variants (i.e., <code>pci dma_sync_single_for_cpu()</code>) via some translation mechanism to not sync the entire buffer?  Alternatively, is there perhaps some function that can make the circular buffer allocated with <code>vmalloc()</code> coherent?</p>