Spark as a Compiler: Joining a Billion Rows per Second on a Laptop
To put this into context I would recommend reading &#x27;MonetDB&#x2F;X100: Hyper-Pipelining Query Execution&#x27; [0]. Vectorized execution has been sort of an open secret in database industry for quite some time now.<p>For me, is particularly interesting reading the Spark achievements. I was part of the similar Hive effort (the Stinger initiative [1]) and I contributed some parts of the Hive vectorized execution [2]. I see the same solution that applied to Hive now applies to Spark:<p>- move to a columnar, highly compressed storage format (Parquet, for Hive it was ORC)<p>- implement a vectorized execution engine<p>- code generation instead of plan interpretation. This is particularly interesting for me because for Hive this was discussed then and actually <i>not</i> adopted (ORC and vectorized execution had, justifiably, bigger priority).<p>Looking at the numbers presented in OP, it looks very nice. Aggregates, Filters, Sort, Scan (&#x27;decoding&#x27;) show big improvement (I would expected these, is exactly what vectorized execution is best at). I like that Hash-Join also shows significant improvement, is obvious their implementation is better than the HIVE-4850 I did, of which I&#x27;m not too proud. The SM&#x2F;SMB join is not affected, no surprise there.<p>I would like to see a separation of how much of the improvement comes from vectorization vs. how much from code generation. I get the feeling that the way they did it these cannot be separated. I think there is no vectorized plan&#x2F;operators to compare against the code generation, they implemented both simultaneously. I&#x27;m speculating, but I guess the new whole-stage code generation it generates vectorized code, so there is no vectorized execution w&#x2F;o code generation.<p>All in all, congrats to the DataBricks team. This will have a big impact.<p>[0] <a href="http:&#x2F;&#x2F;oai.cwi.nl&#x2F;oai&#x2F;asset&#x2F;16497&#x2F;16497B.pdf" rel="nofollow">http:&#x2F;&#x2F;oai.cwi.nl&#x2F;oai&#x2F;asset&#x2F;16497&#x2F;16497B.pdf</a> 
 [1] <a href="http:&#x2F;&#x2F;hortonworks.com&#x2F;blog&#x2F;100x-faster-hive&#x2F;" rel="nofollow">http:&#x2F;&#x2F;hortonworks.com&#x2F;blog&#x2F;100x-faster-hive&#x2F;</a>
 [2] <a href="https:&#x2F;&#x2F;issues.apache.org&#x2F;jira&#x2F;browse&#x2F;HIVE-4160" rel="nofollow">https:&#x2F;&#x2F;issues.apache.org&#x2F;jira&#x2F;browse&#x2F;HIVE-4160</a>