title: Ordering of DMA reads and writes in PCI Express devices
body: <p><strong>Short version:</strong></p>
  
  <p>I have two main questions:</p>
  
  <ul>
  <li><p>Are DMA writes allowed to pass DMA reads? If so, is there any way to stop them from doing so (e.g. by setting a flag, etc.)?</p></li>
  <li><p>Do DMA reads respect any byte-ordering? e.g. left-to-right, right-to-left, or is it non-deterministic? Is there any way to enforce a left-to-right ordering, like what DMA writes have?</p></li>
  </ul>
  
  <p><strong>Long version:</strong></p>
  
  <p>I am not really familiar with the terminology of this subject, so please excuse any mistakes.</p>
  
  <p>The case I have is as follows:</p>
  
  <p>An array of numbers is stored in the main memory, and the network card is issuing a sequence of DMA reads (to the entire array) and DMA writes (to increment a cell of the array). Let's say the array is initially set to zero, like this:</p>
  
  <pre><code>| index |  value |
  ------------------
  |   A   |    0   |
  |   B   |    0   |
  </code></pre>
  
  <p>And the network DMAs are the only ways that this array is read or written to (i.e. no CPU involvement).</p>
  
  <p>Let's say we have this schedule:</p>
  
  <pre><code>R, R, W(A), W(B)
  </code></pre>
  
  <p>I was curious to know if it's possible that the result of one of the reads is <code>{0, 1}</code>, and the result of the other read is <code>{1, 0}</code>.</p>
  
  <p>I wrote a simple program to check this and it turned out that this scenario is possible indeed. I am puzzled as to what could be the reason.</p>
  
  <p>I am not sure at all, but my guess is that this is caused due to two reasons:  transaction ordering in PCIe (the <a href="http://composter.com.ua/documents/PCI_Express_Base_Specification_Revision_3.0.pdf" rel="nofollow noreferrer">PCIe specification</a> in section 2.4.1 says that writes can pass reads, but not vice versa (I didn't understand rule B2b, though), and that DMA reads do not follow any deterministic byte ordering when accessing their data (for this I couldn't find any reliable resource).</p>
  
  <p>For example, this might be what happened under the hood:
  The first read starts from the beginning of the array, and the second read starts from the end of the array. The writes take place, and then the first read scans the second index, and the second read scans the first index.</p>