Optimizations can have unexpectedly large effects when combined with caches
I really love this point.<p>For most applications the optimal size of a (time-based) cache is around the point that keeping an item around in the cache is equal to the cost of accessing it again. This is a point that Jim Gray made in the classic &quot;The 5 Minute Rule for Trading Memory for Disc Accesses&quot; (<a href="http:&#x2F;&#x2F;www.hpl.hp.com&#x2F;techreports&#x2F;tandem&#x2F;TR-86.1.pdf" rel="nofollow">http:&#x2F;&#x2F;www.hpl.hp.com&#x2F;techreports&#x2F;tandem&#x2F;TR-86.1.pdf</a>) back in 1986.<p>A side effect of that is that systems that run faster, and therefore access each cached it more often, get more benefit out of a cache by pushing that set point out further. Whether that happens in any given system depends a lot on the access patterns, and especially temporal locality within those access patterns, but the effect is real in a lot of different kinds of systems.<p>&gt; It&#x27;s common wisdom that systems with caches can fail badly when the caches are cold. When every request misses the cache, the system is overloaded, and never recovers. What I observed was the mirror image.<p>The bottom line is that both locality and memoization are huge contributor to the performance of systems, both positive and negative. When folks (including me) point out the risks of caches, we&#x27;re not implying that caches are always bad. Instead, it&#x27;s pointing out a risk that sometimes they are too <i>good</i>, leading to a system that can&#x27;t function when the normal level of locality and memoization are not available.