I am experiencing a strange loss behavior when I migrated my coded to an RTX2080Ti machine from a Titan XP machine.
I am getting a large loss fluctuation when the loss is becoming small. I have suspicions that its a precision problem but I didn’t get any under flow error during training.
I have read some post about factory bad GPUs but I have tried identical code on 4 separate piece of GPU, they all have similar problems.
The loss is like this:

圖片.png1446×378 142 KB

Four colors represents different GPUs on RTX2080Ti
However, the validation loss looks fine , can’t upload that due to new users 
I am using PyTorch 1.2, Cuda 10.0 from Anaconda3 on an Ubuntu 18.0 machine.
Any comments are welcomed cause this had bothered me for quite some time.