title: Does `isync` prevent Store-Load reordering on CPU PowerPC?
body: <p>As known, PowerPC has weak memory model, that permit any speculative reordering: Store-Store, Load-Store, Store-Load, Load-Load.</p>
  
  <p>There are at least 3 Fences: </p>
  
  <ul>
  <li><code>hwsync</code> or <code>sync</code> - full memory barrier, prevents any reordering</li>
  <li><code>lwsync</code> - memory barriers that prevents reordering: Load-Load, Store-Store, Load-Store</li>
  <li><code>isync</code> - <strong>instruction barrier</strong>: <a href="https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.alangref/idalangref_isync_ics_instrs.htm" rel="noreferrer">https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.alangref/idalangref_isync_ics_instrs.htm</a></li>
  </ul>
  
  <p>For example, can be reordered Store-<code>stwcx.</code> and Load-<code>lwz</code> in this code?: <a href="https://godbolt.org/g/84t5jM" rel="noreferrer">https://godbolt.org/g/84t5jM</a></p>
  
  <pre><code>    lwarx 9,0,10
      addi 9,9,2
      stwcx. 9,0,10
      bne- 0,.L2
      isync
      lwz 9,8(1)
  </code></pre>
  
  <p>As known, <code>isync</code> prevents reordering <code>lwarx</code>,<code>bne</code> &lt;--> <code>any following instructions</code>. </p>
  
  <p>But does <code>isync</code> prevent reordering <code>stwcx.</code>,<code>bne</code> &lt;--> <code>any following instructions</code>?</p>
  
  <p>I.e. can Store-<code>stwcx.</code> begins earlier than the following Load-<code>lwz</code>, and finishes performed later than Load-<code>lwz</code>?</p>
  
  <p>I.e. can Store-<code>stwcx.</code> preforms Store to the Store-Buffer earlier than the following Load-<code>lwz</code> begun, but the actual Store to the cache that visible for all CPU-cores occurs later than the Load-<code>lwz</code> finished?</p>
  
  <p>As we see from the following documents, articles and books: </p>
  
  <ul>
  <li><p><code>isync</code> is not memory fence, but it is only instruction fence. </p></li>
  <li><p><code>isync</code> does not force all external accesses to complete with respect to other processors and mechanisms that access memory.</p></li>
  <li><p><code>isync</code> does not wait for all other processors to detect storage accesses</p></li>
  <li><p><code>isync</code> is a very low-overhead and very weak (lower than <code>lwsync</code> and <code>hwsync</code>)</p></li>
  <li><p><code>isync</code> does not guarantee that previous and future stores will be perceived by other processors in the locally issued order - that requires one of the sync instructions.</p></li>
  <li><p><code>isync</code> is acquire barrier, but as we known, acquire can be applied only to Load-operations, not for Store (<code>stwcx.</code>)</p></li>
  <li><p><code>isync</code> does not affect data accesses and <strong>does not wait for all stores to be performed</strong>.</p></li>
  </ul>
  
  <p><strong>The main question</strong>, initially: a=0, b=0</p>
  
  <ul>
  <li>if CPU-Core-0 do: <code>stwcx. [a]=1</code> <code>bne-</code> <code>isync</code> <code>lwz [b]</code>. </li>
  <li>And CPU-Core-1 do: <code>hwsync</code> <code>stw [b]=1</code> <code>hwsync</code> <code>lwz [a]</code> <code>hwsync</code>. </li>
  </ul>
  
  <p>Then can Core-0 see <code>[b]==1</code> and Core-1 see <code>[a]==0</code>?</p>
  
  <hr>
  
  <p>Also:</p>
  
  <ol>
  <li><a href="https://www.ibm.com/developerworks/systems/articles/powerpc.html" rel="noreferrer">https://www.ibm.com/developerworks/systems/articles/powerpc.html</a></li>
  </ol>
  
  <blockquote>
    <p>The isync prevents speculative execution from accessing the data block
    before the flag has been set. And in conjunction with the preceding
    load, compare, and conditional branch instructions, the isync
    guarantees that the load on which the branch depends (the load of the
    flag) is performed prior to any loads that occur subsequent to the
    isync (loads from the shared block).
    <strong>isync is not a memory barrier instruction</strong>, but the
    <strong>load-compare-conditional branch-isync sequence can provide this
    ordering property</strong>.</p>
  </blockquote>
  
  <ol start="2">
  <li><a href="http://www.nxp.com/assets/documents/data/en/application-notes/AN2540.pdf" rel="noreferrer">http://www.nxp.com/assets/documents/data/en/application-notes/AN2540.pdf</a></li>
  </ol>
  
  <blockquote>
    <p><strong>Unlike isync</strong>, sync forces all external accesses to complete with
    respect to other processors and mechanisms that access memory.</p>
  </blockquote>
  
  <ol start="3">
  <li>Storage in the PowerPC Janice M. Stone, Robert P. Fitzgerald, 1995: <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.47.4033&amp;rep=rep1&amp;type=pdf" rel="noreferrer">http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.47.4033&amp;rep=rep1&amp;type=pdf</a></li>
  </ol>
  
  <blockquote>
    <p>Unlike sync , <strong>isync does not wait for all other processors to detect
    storage accesses</strong>. isync is a less conservative fence than sync because
    <strong>it does not delay until all processors detect previous loads and
    stores.</strong></p>
  </blockquote>
  
  <ol start="4">
  <li><a href="http://open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2745.html" rel="noreferrer">http://open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2745.html</a></li>
  </ol>
  
  <blockquote>
    <p>bc;<strong>isync: this is a very low-overhead and very weak form of memory
    fence.</strong> A specific set of preceding loads on which the bc (branch
    conditional) instruction depends are guaranteed to have completed
    before any subsequent instruction begins execution. However,
    store-buffer and cache-state effects can nevertheless make it appear
    that subsequent loads occur before the preceding loads upon which the
    twi instruction depends. That said, the PowerPC architecture does not
    permit stores to be executed speculatively, so any store following the
    twi;isync instruction is guaranteed to happen after any of the loads
    on which the bc depends.</p>
  </blockquote>
  
  <ol start="5">
  <li><a href="https://books.google.ru/books?id=TKOfDQAAQBAJ&amp;pg=PA264&amp;lpg=PA264&amp;dq=isync+store+load&amp;source=bl&amp;ots=-4FyWvxTwg&amp;sig=r1fitaG-Q3GHOxvSMTgLJMBVGUU&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIOjAC#v=onepage&amp;q=isync%20store%20load&amp;f=false" rel="noreferrer">https://books.google.ru/books?id=TKOfDQAAQBAJ&amp;pg=PA264&amp;lpg=PA264&amp;dq=isync+store+load&amp;source=bl&amp;ots=-4FyWvxTwg&amp;sig=r1fitaG-Q3GHOxvSMTgLJMBVGUU&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIOjAC#v=onepage&amp;q=isync%20store%20load&amp;f=false</a></li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/xJ8iY.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/xJ8iY.jpg" alt="enter image description here"></a></p>
  
  <ol start="6">
  <li><a href="https://books.google.ru/books?id=gZZgAQAAQBAJ&amp;pg=PA71&amp;lpg=PA71&amp;dq=isync+store+load&amp;source=bl&amp;ots=bo6nTLdzEZ&amp;sig=vCjoDmUWhn0buN_uMf8XgbDzCf4&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIcTAJ#v=onepage&amp;q=isync%20store%20load&amp;f=false" rel="noreferrer">https://books.google.ru/books?id=gZZgAQAAQBAJ&amp;pg=PA71&amp;lpg=PA71&amp;dq=isync+store+load&amp;source=bl&amp;ots=bo6nTLdzEZ&amp;sig=vCjoDmUWhn0buN_uMf8XgbDzCf4&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIcTAJ#v=onepage&amp;q=isync%20store%20load&amp;f=false</a></li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/cqvx1.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/cqvx1.jpg" alt="enter image description here"></a></p>
  
  <ol start="7">
  <li><a href="https://books.google.ru/books?id=G2fmCgAAQBAJ&amp;pg=PA321&amp;lpg=PA321&amp;dq=isync+store+load&amp;source=bl&amp;ots=YS4mE-4f_F&amp;sig=OVwaJYE-SNnor-KtKrjlkOd6AOs&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIYjAH#v=onepage&amp;q&amp;f=false" rel="noreferrer">https://books.google.ru/books?id=G2fmCgAAQBAJ&amp;pg=PA321&amp;lpg=PA321&amp;dq=isync+store+load&amp;source=bl&amp;ots=YS4mE-4f_F&amp;sig=OVwaJYE-SNnor-KtKrjlkOd6AOs&amp;hl=ru&amp;sa=X&amp;ved=0ahUKEwiKjYK97urTAhUJ_iwKHbfMA58Q6AEIYjAH#v=onepage&amp;q&amp;f=false</a></li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/kTaSb.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/kTaSb.jpg" alt="enter image description here"></a></p>
  
  <ol start="8">
  <li><a href="http://www.nxp.com/assets/documents/data/en/application-notes/AN3441.pdf" rel="noreferrer">http://www.nxp.com/assets/documents/data/en/application-notes/AN3441.pdf</a></li>
  </ol>
  
  <blockquote>
    <p>Note that <strong>isync does not affect data accesses and does not wait for
    all stores to be performed</strong>.</p>
  </blockquote>
  
  <ol start="9">
  <li>Page 77: <a href="https://www.setphaserstostun.org/power8/POWER8_UM_v1.3_16MAR2016_pub.pdf" rel="noreferrer">https://www.setphaserstostun.org/power8/POWER8_UM_v1.3_16MAR2016_pub.pdf</a></li>
  </ol>
  
  <blockquote>
    <p>3.5.7.2 Instruction Cache Block Invalidate (icbi)</p>
    
    <p>As a result of this and other implementation-specific design
    optimizations, instead of requiring the instruction sequence specified
    by the Power ISA to be executed on a per cache-line basis, software
    must only execute a single sequence of three instructions to make any
    previous code modifications become visible: <code>sync</code>, <code>icbi</code> (to any
    address), <strong><code>isync</code></strong>.</p>
  </blockquote>
  
  <hr>
  
  <p><strong>ANSWER:</strong></p>
  
  <p>So, <code>isync</code> doesn't guarantee Store-Load order, because  "<strong>isync is not a memory barrier instruction</strong>", then <code>isync</code> doesn't guarantee that any previous stores will be visible to other CPU-Cores (uses sequential-consistency) before next intruction will be finished. Instruction synchronization command <code>isync</code> guarantees only the order of starting instructions, but does not guarantee the order of completion of instructions, i.e. does not guarantee the order of their visible effect to other CPU-Cores. Those, <code>isync</code> allows to reorder visible effect of Store-Load in this code <code>stwcx. [a]=1; bne-; isync; lwz [b]</code>.</p>