Solved by wayi in post #5 
              
              
                A few other thoughts: 
Do you have a chance to set async_op=True in dist.all_reduce, so that you can overlap some computation with allreduce? 
Additionally, probably you can try to tune some Gloo environment variables. I am not familiar with Gloo, but I see tuning some NCCL parameters can help commu…