tyb_10:

I have noticed that the time taken by BackwardPass increases from 0.7 secs to 1.3 secs.


Have you tried using the NCCL backend? It should be considerably faster than Gloo.
And have measured the time spent on the entire iteration? Most overhead (replicating model, scatter input, gather output) of DataParallel is incurred during the forward pass.