Thanks Soumith! I tried rebuilding with nccl2 as suggested, but unfortunately without much good news. nccl2 indeed made the per-batch compute time more stable than the fluctuations of nccl1, but total speed gains were very small or none.
I’ve tracked the bottleneck down to the effect of the GIL on parallel_apply during DataParallel.forward(). While scatter and replicate phases were not so harshly effected by the number of GPUs up to 8(even with async off), per-thread forward time in parallel_apply increased almost linearly as the number of threads(GPUs) increased.
So I tried modifying the DataParallel to spawn background processes instead of threads, and saw that it brought down the per-GPU forward time to the single-thread mode. However the cost of IPC, e.g. pickling and unpickling Modules and input Variables on each batch, limited the final result to be only slightly(around 10%) faster than the threaded version. And more importantly, I saw that Variable.grad_fn is blanked out upon passing through ForkingPickler; having lost the computation graph, automatic backward was no longer possible.
As you suggested, I will try to generalize our case into a code that we can share into the benchmark repo. But if this is indeed the limitations of the GIL, the new torch.jit might just be the solution. I’ll track the nightlies and report back if there’s some news.