Comparing SQLite, DuckDB and Arrow with UN trade data
I&#x27;m a little confused with Arrow being included in the comparison.<p>What does Arrow have to do with Parquet? We are talking about the file format Parquet, right? Does Arrow use Parquet as its default data storage format?<p>But isn&#x27;t Arrow a format too? As I understand it, Arrow is a format optimized for transferring in-memory data from one distributed system to another (ser-des), while also facilitating and optimizing certain set operations. From RAM in one system to RAM in another.<p>Moreover, since Arrow is a format, why is it being compared to databases like SQLite and DuckDB? If we&#x27;re talking about formats, why not compare Arrow queries against Parquet data to DuckDB queries against Parquet data? <a href="https:&#x2F;&#x2F;duckdb.org&#x2F;docs&#x2F;data&#x2F;parquet" rel="nofollow">https:&#x2F;&#x2F;duckdb.org&#x2F;docs&#x2F;data&#x2F;parquet</a><p>Why not at least benchmark the query execution alone instead of startup and loading of data? For Arrow, isn&#x27;t it assumed that there is an engine like Spark or Snowflake already up and running that&#x27;s serving you data in the Arrow format? Ideally, with Arrow you should never be dealing with data starting in a resting format like Parquet. The data should already be in RAM to reap the benefits of Arrow. Its value proposition is it&#x27;ll get &quot;live&quot; data from point A to B as efficiently as possible, in an open, non-proprietary, ubiquitous (eventually) format.<p>Exactly what of SQLite, DuckDB and Arrow is being compared here?<p>I would assume the benefits of Arrow in R (or DataFrames in general) would be getting data from a data engine into your DataFrame runtime as efficiently as possible. (just as interesting might be where and how push-downs are handled)<p>Perhaps I&#x27;m missing the trees for the forest?<p>No disrespect to the author... Seems like they&#x27;re on a quest for knowledge, and while the article is confusing to me, it certainly got me thinking.<p>Disclaimer: I don&#x27;t read R too good, and I&#x27;m still struggling with what exactly Arrow is. (Comparisons like this actually leave me even more confused about what Arrow is)