Scalability, but at what cost? (2015)
I like the analysis, basically it says &quot;hey you don&#x27;t have big data&quot; :-) but that requires a bit more explanation.<p>The <i>only</i> advantage of clustered systems like Spark, Hadoop, and others is aggregate bandwidth to disk and memory. We know that because Amdahl&#x27;s law tells us that parallelizing something invariably adds overhead. So from a systems perspective that overhead has to be &quot;paid for&quot; by some other improvement, and we&#x27;ll see that it is access to data.<p>If your task is to process a 2TB data set, on a single machine using a 6GBS SATA channel and 2TB of FLASH SSDs you can read in that dataset into memory in 3333 seconds (at 600MB&#x2F;sec which is very optimistic for our system), process it, and lets say you write out a 200GB reduced data set for another 333 seconds. so, conveniently, an hour of I&#x2F;O time.<p>Now you take that same dataset and distribute it evenly across a thousand nodes. Each one then has 2GB of the data on it. Each node  can read in their portion of the data set in 3 seconds, process it and write out their reduction in .3 seconds.<p>You have &quot;paid&quot; for the overhead of parallelization by trading an I&#x2F;O cost of an hour for an I&#x2F;O cost of about 4 seconds.<p><i>That</i> is when parallel data reduction architectures are better for a problem than a single threaded architecture. And that &quot;betterness&quot; is purely artificial in the sense that you would be better off with a single system that had 1,000 times the I&#x2F;O bandwidth (cough <i>mainframe</i> cough) than 1,000 systems with the more limited bandwidth. <i>However</i> a 1,000 machines with one SSD it still cheaper buy than one mainframe of similar capability. So <i>if</i>, and its a big if, your algorithm can be expressed as a data map &#x2F; reduce problem, and your data is large enough to push the cost of getting it into memory to have a look at significantly beyond cost of executing the program, <i>then</i> you can benefit positively by running it on a cluster rather than running it on a local machine.