Why Databricks Is Winning
I actually posted on a somewhat related topic here:<p>- <a href="https:&#x2F;&#x2F;managingml.substack.com&#x2F;p&#x2F;ml-model-training-is-an-etl" rel="nofollow">https:&#x2F;&#x2F;managingml.substack.com&#x2F;p&#x2F;ml-model-training-is-an-et...</a><p>As an ML manager one of the things that I dislike about the Databricks model is the reliance on notebooks and bespoke interactive cluster workflows.<p>ML workloads are stuck in the stone age because there’s no common pattern to plug them into existing ETL frameworks, but really that is what they are.<p>Model training is an ETL that just happens to need unique domain tools to visualize how it’s succeeding &#x2F; track observability metrics. But apart from that, you really need to automate model training and experimentation.<p>It is a total antipattern to <i>ever</i> use notebooks, or even non-notebook code developed interactively. Rather you should be submitting tasks to a task queue that maps your ML exploratory workload or training workload to a scheduled compute resource, runs it with observability baked in, and treats outputs like schema’d outputs of more traditional ETLs.<p>Instead of that, Databricks is much more of an early 2000s MATLAB model. Make it addictive and easy for unpragmatic researchers detached from real production use cases, then figure employers will have to adopt it since all the expensive-to-hire researchers can only be productive with it.<p>Long term I think it’s a very bad gamble. Just consider how much open source Python tools have eaten MATLAB’s lunch in the past 10 years.