My company is working [#] on a domain-specific language with similar premises. When running a script, we load data from  CSV, and use a clever caching structure to avoid any work done on a previous run.<p><a href="https:&#x2F;&#x2F;www.lokad.com&#x2F;envision-more-technical-overview" rel="nofollow">https:&#x2F;&#x2F;www.lokad.com&#x2F;envision-more-technical-overview</a><p>We&#x27;ve actually encountered two kinds of work with such data sets: exploratory work (when a data scientist is running brand new queries on a fixed data set) and production work (when a pre-written script is run on new data a daily or weekly basis, maybe with some user-provided parameters). These are separate optimization problems, the first is usually solved by caching the results of intermediate processing while the second is mostly about optimizing the script execution.<p>In the latter case, I think one of the main difficulties with SQL is that queries are independent and a typical data analysis session requires many queries. With typical database pipelines, there&#x27;s no obvious spot to perform global static analysis and optimize the query sequence as a whole (for instance, finding shared common expressions or indices). Stored procedures are a good start, but typical query optimizers just aren&#x27;t designed to achieve any kind of procedure-level analysis.<p>[#] and we&#x27;re looking to hire another compiler expert, if anyone&#x27;s interested.