Do we need a third Apache project for columnar data representation?
This point is interesting:<p><i>So indeed, it does not matter whether the data is stored on disk or in memory --- column-stores are a win for these types of workloads. However, the reason is totally different.</i><p>He says that for tables on disk, column layout is a win due to less data transferred from disk.  But for tables in memory, the win is due to the fact that you can vectorize operations on adjacent vaules.<p>I also think the convergence of these two approaches is interesting: one is from a database POV; the other is from a programming language POV.<p>Column Databases came out of RDBMS research.  They want to speed up SQL-like queries.<p>Apache Arrow &#x2F; Pandas came at it from the point of view of R, which descended from S [1].  It&#x27;s basically an algebra for dealing with scientific data in an interactive programming language.  In this view, rows are observations and columns are variables.<p>It does seem to make sense for there to be two separate projects now, but I wonder if eventually they will converge.  Probably one big difference is what write operations are supported, if any.  R and I think Pandas do let you mutate a value in the middle of a column.<p>-----<p>On another note, I have been looking at how to implement data frames.  I believe the idea came from S, which dates back to 1976, but I  only know of a few open source implementations:<p>- R itself -- the code is known to be pretty old and slow, although rich with functionality.<p>- Hadley Wickham&#x27;s dplyr (the tbl type is an enhanced than data.frame).  C++.<p>- The data.table package in R.  In C.<p>- Pandas.  In a mix of Python, Cython, and C.<p>- Apache Arrow.  In C++.<p>- Julia DataFrames.jl.<p>If anyone knows of others, I&#x27;m interested.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;S_(programming_language)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;S_(programming_language)</a>