@yzz, thanks for posting and compare dist.allreduce with NCCL allreduce benchmark.
May I have your complete code for the comparison? also what kind of GPU are you using? and what is the network type (GPUDirect or ethernet and etc)? I can try to repro and see what it is going on here.