Actually, piping data between standard Unix tools can be <i>extremely</i> efficient.  This example is working on gigabytes rather than terabytes of data, but memory utilization is basically limited to just buffers, so you could definitely scale that to terabytes without a problem.<p><a href="http:&#x2F;&#x2F;aadrake.com&#x2F;command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html" rel="nofollow">http:&#x2F;&#x2F;aadrake.com&#x2F;command-line-tools-can-be-235x-faster-tha...</a>