That&#x27;s just a combination of GPU model and data parallelism, which have been supported for about 2 years in Torch among others. You can add in the outer product trick for fc layers as well.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;soumith&#x2F;imagenet-multiGPU.torch&#x2F;blob&#x2F;master&#x2F;models&#x2F;alexnetowt.lua" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;soumith&#x2F;imagenet-multiGPU.torch&#x2F;blob&#x2F;mast...</a><p>nccl is in Torch as well, but it doesn&#x27;t always win, and has some weird interactions regarding streams and other such things with its use of persistent kernels.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;torch&#x2F;cunn&#x2F;blob&#x2F;master&#x2F;DataParallelTable.lua" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;torch&#x2F;cunn&#x2F;blob&#x2F;master&#x2F;DataParallelTable....</a><p>However, this feels more of a benchmarking thing now. Networks tend to be over-parameterized and redundant in many ways; sending less data with much deeper &#x2F; smaller parameter networks has been where the action is (e.g., residual networks, GoogLeNet, etc.), or to non-synchronous forms of communication among workers. Trying to squeeze out every last drop of GPU&#x2F;GPU bandwidth is not as important as iterating on the architecture and learning algorithms themselves.