Looks like uneven inputs for me.
Can you add no_sync context manager to disable allreduce in DDP?
https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync 5
This way you shouldn’t have any gradient synchronization, and you can check if this issue still occurs.