What Happens to OO When Processors Are Free? (2015)
The real question is "what do we do with a lot of CPUs without shared memory?" Such hardware has been built many times - Thinking Machines, Ncube, the PS2's Cell - and has not been too useful for general purpose computing.One of the simplest cases for broad parallelism is search where most of the searches fail. The required intercommunication bandwidth is low - you start up all the machines and wait for somebody to report success.  Bitcoin miners and crypto key searchers are extreme examples.  Problems which can be hammered into that form parallelize well.Some problems that distribute spatially parallelize well on such hardware.  Weather prediction, wind tunnel simulation, nuclear explosion simulation - hydrodynamics problems like that can be handled in sections, with communications only for the edge elements.  That's what supercomputers are used for.Neural nets and deep learning, though - finally, there's a broad use case for such architectures. Google is using lots of GPU-type boards that way now.   This is a key concept. Massive parallelism on a single problem is really hard to apply to the things people used to do on computers.   Instead, it makes possible new things to do.An important question is how much memory each CPU should have.  Thinking Machines, NCube, and the Cell (256K each) clearly had too little. You need enough memory per CPU that the intercommunication doesn't dominate the problem.The other big problem is how to organize software for this. We have two main models of parallel software - shared memory, and clusters on a network.  Those are the two extremes for parallelism.  Systems where the CPUs are more tightly coupled than in an cluster but less tightly coupled than in a shared memory multiprocessor fall outside common programming paradigms.  This is a very hard problem, with no really good general-purpose solutions.