1.1B Taxi Rides Using OmniSciDB and a MacBook Pro
Whenever I look at large aggregation benchmarks like this, I try to estimate cycles/value or better cycles/byte.<p>Take this query:<p><pre><code>    SELECT cab_type,
           count(*)
    FROM trips
    GROUP BY cab_type;
</code></pre>
This is just counting occurrences of distinct values from a bag of total values sized @ 1.1B.<p>He's got 8 cores @ 2.7GHz, which presumably can clock up for short bursts at least a bit even when they're all running all out. Let's say 3B cycles/core/second. So in .134 seconds (the best measured time) he's burning ~3.2B cycles to aggregate 1.1B values, or about 3 cycles/value.<p>While that's ridiculously efficient for a traditional row-oriented database, for a columnar scheme as I'm sure OmniSciDB is using, it's less efficient than I might have expected.<p>Presumably the # of distinct cab types is relatively small, and you could dictionary-encode all possible values in a byte at worst. I'd expect opportunities both for computationally friendly compact encoding ("yellow" is presumably a dominant outlier and could make RLE quite profitable) and SIMD data parallel approaches that should let you roll through 4,8,16 values in a cycle or two.<p>Even adding LZ4 should only cost you about a cycle a byte.<p>That's not to denigrate OmniSciDB: They're already several orders of magnitude better than traditional database solutions, and plumbing all the way down from high-level SQL to bit twiddling SIMD is no small feat. More that there's substantial headroom to make systems like this even faster, at least until you hit the memory bandwidth wall.