I don&#x27;t use NCCL, I wrote my own ring collectives after reinventing them in 2014 (I felt really clever for a whole 20 minutes before a Google search popped my balloon) to avoid working with a vendor that was trying to gouge us on servers: they beat NCCL by ~50%.<p>That said, most people couldn&#x27;t write them so I advise NCCL.  You&#x27;re the third person to tell me NCCL has ish(tm), fascinating.<p>And sure, you could do the outer product trick.  You could use non-deterministic ASGD.  And you can do a lot of weird(tm) tricks.  But why do these (IMO ad-hoc and potentially sketchy task-specific) things when there&#x27;s an efficient way to parallelize the original network in a deterministic manner for training that allows performance on par with a DGX-1 server?<p>Because for me, the action is in automagically discovering the optimal deterministic distribution of the computation so researchers don&#x27;t need to worry about it.  And IMO that&#x27;s where the frameworks fail currently.