New make --shuffle mode
Great feature. And shuffle isn&#x27;t just useful for finding build issues, it can actually improve performance in some really annoying edge cases!<p>The basic idea is this: Let&#x27;s say you have a target being built and one of the dependencies takes a long time to compile; say, a large auto-generated file. If the time to compile this file is very dominant and outlandish, then you <i>really</i> want to do it as early as possible (while satisfying all dependent constraints), no matter what order the rules are in. This is so that it can overlap with other work as much as possible. If you instead pick it very late in the build process, it can significantly extend the total build time.<p>A concrete example is one I had at a past job. We used CMake on a reasonably-sized and actively developed C codebase. Make took about 5 minutes to do a clean build. Switching to Ninja reduced that to 2:30s, on a clean build. That was great. But then, I used Shake -- a build system that ships with a Ninja-compatible build tool, just called &quot;shake&quot; -- to build the same project. 1m30s total![1]<p>It took me a while to realize this is because shake by default randomizes build plans, both to find errors, <i>and</i> to increase performance in cases like this. By just shuffling the build order, you will get a &quot;smooth&quot; distribution of build times, whereas always deterministically sticking with 1 plan can result in a build plan with very very poor &quot;tail latencies&quot;. This was the exact example I mentioned. The project built about 10 executables, and 1 of them was a very large auto-generated tool, and compiling it took 80% of the total wall clock time of all other jobs combined. So shake would often build this tool <i>first</i> or at least very early on, just by chance -- while Ninja and Make would often build it <i>last</i>, every single time. A very big impact on tail latency.<p>So the TL;DR is -- randomize your build plans, knock out the missing dependencies, stabilize and ensure determinism where possible, and be happy!<p>[1] Shake was only better on the &quot;rebuild entirely from scratch&quot; case versus Ninja. When doing incremental rebuilds with small edits, I basically couldn&#x27;t find any (meaningful) difference in their performance.