I’m using PyTorchv1.1.0 and DistributedDataParallel to train some models.
The training process will get stuck at some constant steps.
I noticed some wired things:

The Memory usage keeps freeze.
GPU usage increases to 100%. But the temperature of GPU isn’t very high.
The CPU usage of 4 main progress is 100%

I think it is not a code bug. Because I use some different models and they both get stuck at some iteration step.