&quot;The current implementation uses a tree reduction strategy. e.g. if there are 4 GPUs in the system, 0:1, 2:3 will exchange gradients, then 0:2 (top of the tree) will exchange gradients, 0 will calculate updated model, 0-&gt;2, and then 0-&gt;1, 2-&gt;3.&quot;<p>Nope...  This is the stupid way...  They should be using ring reductions, handily provided by the NCCL library:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;nccl" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;nccl</a>