The New Intel: How Nvidia Went from Powering Video Games to Revolutionizing AI
If Nvidia are looking to supplant Intel, I think they key, weirdly, is i&#x2F;o. In a sense, GPU acceleration of general purpose computation is just a specific case of the more general case of distributed computing (albeit the computers are extremely &#x27;close&#x27;). For embarrassingly parallel workloads, i&#x2F;o is not an issue; the data and computations can be batched, distributed and no further communication required (until you collect the results).<p>But for mixed, semi-sequential&#x2F;semi-parallel workloads, I&#x2F;O becomes the bottleneck. Having to retrieve intermediate results, combine and then redistribute can eat up your gains from parallelisation. It will be interesting to see if Nvidia start pushing for a new standard to replace PCIe, and maybe invest in low-latency, high throughput networking R&amp;D. Who knows? Maybe they&#x27;ll just build some kind of networking functionality in to their GPUs directly, so data can be transferred NVRAM -&gt; NVRAM without having to travel up and down the stack (until they need to interact with userspace; once on the way in and once on the way out).<p>Given there are RDMA adapters that can manage ~20ns latency already, it would be interesting to see what could be achieved before we start pushing up against the laws of physics. Even at the physical layer, my understanding is that most networking fibre optics manage about 60% the speed of light.<p>So, just to make a wild prediction: within the next year, Nvidia will acquire a HPC network adapter company (e.g. Mellanox).