Hey @xiefeiwhu
optimizer.step() is not part of DDP forward-backward. Which optimizer are you using? and do you observe the same slowness in local training.
BTW, how did you measure the delay? You might need to use CUDA events 10 to get accurate timing measures, as there could be pending ops in the CUDA stream so that time.time() cannot faithfully represent the time cost.