A repro would be helpful, and in addition DistributedDataParallel cannot handle multi-device runs with non-evenly divisible batch sizes · Issue #46175 · pytorch/pytorch · GitHub 22 might also provide valuable context.
Are you using single GPU per process or multiple GPUs per process? The latter mode is deprecated and no longer maintained by PyTorch and we suggest moving to single GPU per process which is the more performant and supported way going forward.