We are tracking this issue here: https://github.com/NVIDIA/nccl/issues/318 127
It looks like the bottleneck is the network bandwidth between two GPU nodes. This example is fine tuning roberta-base model, which is about 500 MB. During the synchronization of two nodes cases, the amount of data would be 8G. NCCL folks mentioned that on single instances, this communication is supported by nvlink, and it is about 120GB/s, however on if this is tcp/ip sockets, even if the bandwidth is 100GB/s, the all reduce bandwidth would be 10GB/s. So like a order of magnitude smaller, therefore the communication time become the bottleneck.
In this example, the epoch is set to 1 for both single node and two nodes case.
The training script uses distributedSampler, so the data partition on each node should be mutually exclusive.
if you feel there is anything else other than the network bandwidth would cause any issues, let me know.