High performance models in TensorFlow
How would you ever saturate 8 GPUs if your average large motherboard with 2 CPUs has only 40 PCIe lanes at its disposal?<p>Is there a motherboard out there that would allow the usage of 8 GPUs with each of them allocated the full 16 lanes?<p>The best I've found so far would be 4 GPUs on one board, either with 16/8/8/8 or in a rare case 16/16/16/16 (but that requires 2 CPUs).<p>Besides the physical space, which again seems to be limited to 4 double-wide GPUs on one motherboard.