Its something like this.

圖片.png1446×378 38.1 KB

As seen its much smoother at small losses and converge much better.
I only have one Titan Xp GPU though.
I used CUDA 9.0  with cudnn 7501 and Pytorch 1.1.
RTX 2080Ti used cudnn 7602 acording to torch.backends.cudnn.version()
Do you think its a version problem? I wouldn’t be able to downgrade to CUDA 9.2 for RTX2080Ti I suppose.