Oh I get it!
It is caused by following line:
os.environ['CUDA_VISIBLE_DEVICES'] = str(proc_id%8)

I set this in each process to avoid allocating extra GPU memory on device 0(without this env variable set there will be 8 processes on GPU 0 and 7 of them each occupies around 300MB extra memory).
I remove the above line and add the following line:
torch.cuda.set_device(proc_id%8)

then everything runs smoothly!!!
On 8 titanxp, resnet50 with batchsize=32 runs 0.2s per iteration, far more better than the ‘DataParallel’ which is 0.3s per iteration, this improvement is more obvious for larger models. The cool thing is that NCCL2 automatically detects infiniband and directly extend to multi-node! When using 16 GPUs, it is 0.23s per iteration, considering that I’m using raw allreduce-based synchronized SGD without any other optimizations, this result is really good!
Thank you teng-li! Your NCCL2 backend works really well! Hope it will be merged to master soon.