I&#x27;m arguing that modern superscalar out-of-order hardware supports lots of parallelism, but that modern compilers (and software practices) leave a lot of performance on the table.  The key to overcoming latency is concurrency, and current software rarely manages to reach its potential.  This makes it look like hardware isn&#x27;t getting faster, but really it&#x27;s just not getting faster at running software targeted at earlier in-order CPU&#x27;s.<p>Consider performing a series of simplified hash table lookups based a list of strings.  You hash the first string, use the prefix of the hash as an array offset to look up another pointer, use the pointer to access a variable length array, and then check whether you have a matching element.  Then you hash the second string, and repeat.  Third, fourth, etc.<p>To achieve high performance, you need to overlap the lookups, so that the latencies are shared rather than sequential.  But I don&#x27;t know of any compilers (although I&#x27;m only familiar with GCC, Intel, and Clang) smart enough to do so.  The hardware tries to get out ahead, but will almost always experience a branch prediction error before it can start working on the next iteration.  And if it does manage, it will usually get at most a single iteration ahead.<p>By contrast, if you were to do 8 hashes in parallel (often just as fast with AVX2), and then issue 8 loads, then chase those 8 pointers in parallel, you&#x27;ve increased your throughput almost 8x.  Little&#x27;s Law is the low-hanging fruit, and you don&#x27;t need to figure out how to do cycle accurate synchronization between cores.<p>I realize I probably sound like a quack.  Here&#x27;s a more formal paper describing some similar approaches with a Cuckoo Hash: <a href="http:&#x2F;&#x2F;www.cs.columbia.edu&#x2F;~orestis&#x2F;sigmod15.pdf" rel="nofollow">http:&#x2F;&#x2F;www.cs.columbia.edu&#x2F;~orestis&#x2F;sigmod15.pdf</a>