Hi teng-li,
I have tried your new NCCL2 backend, but getting the following error(I have tried NCCL2.0.5 and NCCL2.1, both raising the same error.):
NCCL error in: pytorch-nccl2/torch/lib/THD/base/data_channels/DataChannelNccl.cpp:330, unhandled cuda error

By setting NCCL_DEBUG=INFO, I get the following output:
INFO NET : Using interface ib1:10.10.120.68<0>
INFO NET/IB : Using interface ib1 for sideband communication
INFO NET/IB: [0] mlx4_0:2/IB
INFO Using internal Network IB
INFO NET : Using interface ib1:10.10.120.68<0>
INFO NET/IB : Using interface ib1 for sideband communication
INFO NET/IB: [0] mlx4_0:2/IB
INFO Using internal Network IB
NCCL version 2.0.5 compiled with CUDA 8.0
INFO CUDA Dev 0, IB Ports : mlx4_0/2(SOC)
INFO CUDA Dev 0, IB Ports : mlx4_0/2(SOC)
INFO Using 256 threads
INFO [0] Ring 0 :    0   1
INFO 1 -> 0 via P2P/IPC
INFO 1 -> 0 via P2P/IPC
INFO 0 -> 1 via P2P/IPC
INFO 0 -> 1 via P2P/IPC

transport/p2p.cu:429 WARN failed to open CUDA IPC handle : invalid argument
INFO transport/p2p.cu:439 -> 1
INFO init.cu:373 -> 1
INFO init.cu:432 -> 1
INFO
transport/p2p.cu:429 WARN misc/group.cu:87 -> 1 [Async thread]
failed to open CUDA IPC handle : invalid argument
INFO transport/p2p.cu:439 -> 1
INFO init.cu:373 -> 1
INFO init.cu:432 -> 1
INFO misc/group.cu:87 -> 1 [Async thread]

Any ideas?