Do the jobs always run on the exactly same set of machines?
If not, can there be any straggler in cluster? Or can different network layouts (same rack, different rack, etc.) play a role here?
For debugging, it will be helpful to identify which step (forward, backward, opt.step) takes longer (on all nodes) when the throughput drops to 1000 images/s. elapsed_time 3 should be able to tell that. All communication in DDP occurs in the backward pass. If all steps are the same but the backward pass takes long for all processes, then it might be caused by network issues. If some processes suffers from slower data loading, or slower forward, or slower optimizer, it looks more like a straggler problem.