Command-line tools can be faster than a Hadoop cluster (2014)
I feel like every time something like this comes up people completely skip over the benefit of having as much of your data processing jobs in one ecosystem as possible.<p>Many of our jobs operate on low TBs and growing but even if the data for a job is super small I&#x27;ll write it in Hadoop (Spark these days) so that the build, deployment, and schedluing of the job is handled for free by our curent system.<p>Sure spending more time listing files on S3 at startup than running the job is a waste but far less than the man hours to build and maintain a custom data transformation.<p>The main benefit of these tools is not the scale or processing speed though. The main benefits are the fault tolerance, failure recovery, elasticity, and the massive ecosystem of aggregations, data types and external integrations provided by the community.