Besides,
If I use only 2 nodes, each with 1 GPU, it works well with following log. But if 3 or 4 nodes, above error accurs.
451 tj1-asr-train-v100-11:41449:41449 [6] NCCL INFO Bootstrap : Using [0]eth0:10.38.10.4<0>
452 tj1-asr-train-v100-11:41449:41449 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
453 tj1-asr-train-v100-11:41449:41449 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
454 tj1-asr-train-v100-11:41449:41449 [6] NCCL INFO NET/Socket : Using [0]eth0:10.38.10.4<0>
455 NCCL version 2.4.8+cuda10.0
456 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO Setting affinity for GPU 6 to 03ff,f0003fff
457 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO CUDA Dev 6[6], Socket NIC distance :  PHB
458 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO Channel 00 :    0   1
459 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO Ring 00 : 1 -> 0 [receive] via NET/Socket/0
460 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO NET/Socket: Using 1 threads and 1 sockets per thread
461 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO Ring 00 : 0 -> 1 [send] via NET/Socket/0
462 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled
463 tj1-asr-train-v100-11:41449:43711 [6] NCCL INFO comm 0x2b5210001cf0 rank 0 nranks 2 cudaDev 6 nvmlDev 6 - Init COMPLETE
464 tj1-asr-train-v100-11:41449:41449 [6] NCCL INFO Launch mode Parallel
465 [tj1-asr-train-v100-11:0/2] 2020-05-18 15:18:44,949 (trainer:201) INFO: 1/200epoch started
466 Setting NCCL_SOCKET_IFNAME
467 Finished NCCL_SOCKET_IFNAME
468 ^lo,docker
469 self.dist_backend: nccl
470 self.dist_init_method: file:///home/storage15/huangying/tools/espnet/egs2/voxforge/asr1/vox.init
471 self.dist_world_size: 2
472 self.dist_rank: 0
473 auto allocate gpu device: 6
474 devices ids is  6