Hi, i am using PyTorch DistributedDataParallel to train some models and surpisingly. It become slower when i move from one GPU node to 2 GPU nodes.
Here is the example i am running: https://github.com/huggingface/transformers/blob/v2.3.0/examples/run_lm_finetuning.py 5
Here is my infrastructure:
Single Node: p3.16xlarge
Training time: 36 mins
Two Nodes: p3.16xlarge
Training time: 1 h 45 mins
In both cases, i am using PyTorch distributed data parallel and GPU utilization is almost always be 100%.
I have enabled NCCL_DEBUG=INFO
I copied the nccl output from single node training and multiple node training in this link below.

  
      gist.github.com
  
  
    https://gist.github.com/YingleiZhang/a8df48eb534ba20ff8f26b5309094b55 17
NCCL debug info on multiple node:
ip-10-0-42-19:13243:13243 [2] NCCL INFO Bootstrap : Using [0]eth0:10.0.42.19<0>
ip-10-0-42-19:13282:13282 [7] NCCL INFO Bootstrap : Using [0]eth0:10.0.42.19<0>
ip-10-0-42-19:13243:13243 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ip-10-0-42-19:13282:13282 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-42-19:13243:13243 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
 
ip-10-0-42-19:13282:13282 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-10-0-42-19:13282:13282 [7] NCCL INFO NET/Socket : Using [0]eth0:10.0.42.19<0>
ip-10-0-42-19:13243:13243 [2] NCCL INFO NET/Socket : Using [0]eth0:10.0.42.19<0>
This file has been truncated. show original
NCCL debug info on single node:
ip-10-0-28-255:46114:46114 [0] NCCL INFO Bootstrap : Using [0]eth0:10.0.28.255<0>
ip-10-0-28-255:46114:46114 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-28-255:46114:46114 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ip-10-0-28-255:46114:46114 [0] NCCL INFO NET/Socket : Using [0]eth0:10.0.28.255<0>
NCCL version 2.5.6+cuda10.0
ip-10-0-28-255:46149:46149 [5] NCCL INFO Bootstrap : Using [0]eth0:10.0.28.255<0>
ip-10-0-28-255:46149:46149 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
 
ip-10-0-28-255:46149:46149 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
This file has been truncated. show original




  
  
    
    
  
  


It seems on single node mode, NCCL are creating a lot of CHANNELs, but not in the multiple node mode. Could this slow speed due to the networking issues between different GPU nodes?
Any help is appreciated.