Do you have even number of batches across all GPUs? If not, the training could get stuck since DDP performs a collective sync across all GPUs in the backward pass. If you have uneven batches, you can try using the join 20 API to get around this.