I am training VGG11 over 16 nodes with data parallellism and NCCL backend. However, I found that the training time for one iteration is too long. I breakdown the time spent at IO, forward, backward, and optimization. It turns out that the I/O, forward, and optimization phases have similar time durations when compared with 8 nodes. The major time is increased by the gradient synchronization during the backward phase.
I profile the code. It turns out the NCCL allreduce takes the majority of the time (see figure below, the timeline for 1 iteration over 16 nodes). I think the most of time is spent on the classifier layers. However, the Pytorch NCCL allreduce time of these layers is much longer than the expected original NCCL allreduce performance on the same amount data. In addition, I also measured PyTorch NCCL allreduce performance over the model parameters (see code below). It turns out the classifier layers take 280.682 ms (total size: 471MB). However, if I directly use NCCL allreduce bechmark to report the performance of the same amount of data the time is about 60ms. I wonder if anyone might know the reason.
I am using Pytorch 1.4.
[0] NCCL INFO NET/IB : Using [0]mlx5_1:1/IB [1]mlx5_3:1/IB [2]mlx5_0:1/IB [3]mlx5_2:1/IB ;

  for param in model.parameters():
    event_start[i].record()
    dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
    event_end[i].record()