Command-line tools can be faster than your Hadoop cluster
To quote the memorable Ted Dziuba[0]:<p>"Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.<p>The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script."<p>[0] since his blog is gone: <a href="http://readwrite.com/2011/01/22/data-mining-and-taco-bell-prog" rel="nofollow">http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...</a>