COST in the land of databases
The author should spend some time delivering solutions to "real problems". It's easy to be snarky while writing a blogpost with toy datasets but it's difficult to actually deliver production solution in an organization. The real COST is often not the time consumed but costs that arise from maintaining heterogeneous architecture and remaining legally compliant.<p>A database or hadoop cluster is not just an execution engine, but rather a managable, reproducible execution engine. Sure you can code up a smarter algorithm on your personal i9 workstation. However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc.<p>Eventually we will have a containerized extensions to DB query execution flow that will allow us to have best of both worlds but for 99% practicing data scientists downloading a Gigabyte sized subset to their own workstation is not a viable option.<p>The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.