Too much locality for stores to forward
&gt; In fact, perf showed that the query server as a whole was spending 4% of its CPU time on one instruction in that loop:<p>Looks like 40% to me actually. I think there&#x27;s a typo here. EDIT: The tweet is more clear. 4% of the total CPU time for the whole program. The 40% in that line is only for that particular function (I presume the function is taking ~10% total CPU time, and 40% of that is on that singular instruction)<p>------<p>&gt; The first thing to note is that instruction-level profiling tends to put the blame on the instruction following the one that triggered a sampling interrupt.<p>Hmm, in my experience with AMD-based hardware profiling, AMD CPUs may place the blame as far back as ~200 instructions (pathological worst-case) behind the instruction that was interrupted.<p>This is most obvious on AMD branch mispredictions. A branch mispredict can, by definition, only happen on branch instructions (a cmp&#x2F;jmp pair, since cmp&#x2F;jmp turn into a single double-instruction micro-op on both AMD and Intel). But you&#x27;ll find mispredictions &quot;blamed&quot; on all sorts of non-branching instructions on AMD-hardware. Its a known limitation of AMD&#x27;s hardware performance counters.<p>In the first case, the modvqu is almost certainly where the 40% of the time is being spent... later on there is...<p><pre><code>     9.91 |       lea        (%r12,,4),%rax
     0.64 |       prefetcht0 (%rdx,%rax,8)
    17.04 |       cmp        %rcx,0x28(%rsp)
</code></pre>
I can&#x27;t imagine that the prefetch0 instruction is taking up 17% of the time of the function. I expect that the 17% here must be something else. In particular, Agner Fog&#x27;s manuals state that PREFETCHINTA&#x2F;0&#x2F;1&#x2F;2 has 1-latency and 2-instructions&#x2F;clock throughput: a very fast instruction under Agner Fog&#x27;s test case on both Intel Skylake and AMD Zen.<p>AMD has &quot;instruction base sampling&quot;, a difficult to understand but more accurate profiling methodology. Intel has PEBS, precision event-based sampling, which helps narrow down the specific instruction to &quot;blame&quot;.<p>This blogpost doesn&#x27;t say which CPU its using, but unfortunately, these little details do matter. Ultimately, I&#x27;d investigate more before concluding the prefetcht0 instruction is actually the problem.<p>If Store Forwarding might be an issue, then LD_BLOCKS.STORE_FORWARD event on Intel Skylake would be a performance counter to check.<p>--------------<p>&gt; and accelerate random accesses to the hash table with software prefetching.<p>Does prefetching actually help in this case? Have you tested the code without prefetching?<p>Without prefetching, a MODERN processor will NOT &quot;stall&quot; on the load instruction. A modern processor will execute out-of-order, branch predict the next iteration (its an inner loop, so its probably always taken), and then effectively achieve the same throughput anyway.<p>I wouldn&#x27;t necessarily be &quot;surprised&quot; if prefetching helps in this case... but its not necessarily a given.