long all reduce time could be due to 1) low network bandwidth 2) data loading or computation caused desync btw ranks, as @ptrblck  mentioned fastest runner will wait for the others. Try to use simulated data and benchmark the performance.
One good way to profile is to use torch profiler and dump the trace to Chrome for viewing the event timelines