Hi everyone,
I have stumbled upon a problem when using DistributedDataParallel. Strangely, after a few epochs of successful training, loss goes up for a while. I noticed that both train/validation losses got down for the batches which are on GPU0, but go up for the other 3 GPUs. I believe I’m doing something wrong with DistributedDataParallel, but can’t find a bug. Did anyone see a similar problem, or can guess what the reason can be?
In the chart you can see training and validation losses for GPU0 and average of all 4.
Screenshot from 2020-05-18 05-16-581920×1080 323 KB