I/O is no longer the bottleneck
As the algorithm used in the example is straight-forward I figured that using UNIX command line tools might be an even simpler way to implement it. Here is what I came up with:<p><pre><code>  time cat kjvbible_x100.txt | tr &quot;[:upper:] &quot; &quot;[:lower:]\n&quot; | sort --buffer-size=50M | uniq -c | sort -hr &gt; &#x2F;dev&#x2F;null
</code></pre>
On my machine this turned out to be ~5 times slower than the provided Python implementation. Nearly all of the time is spent in the first invocation of `sort`. Further increasing the buffer size doesn&#x27;t make a significant difference. I also played around with the number of threads `sort` uses, but didn&#x27;t see any improvement there either.<p>I&#x27;m quite puzzled why `sort is so much slower, especially as it does sorting in parallel utilizing multiple CPU cores, while the Python implementation is single-threaded.<p>Does somebody have an explanation for that?