I am experiencing exactly the same issue as you do, but when I am training a segmentation network,  a UNET.
I have a system with CUDA 10.1, Nvidia Driver 455, GTX 1080 Ti with PyTorch 1.6, where the training runs successfully.
On the other hand, I have another system with CUDA 10.1, Nvidia Driver 418.35 (the first driver shipped with CUDA 10.1) and GTX 2080 TI with PyTorch 1.6. The training does not converge, let alone the results on the validation set,
I copied the same exact code from my local machine(with good results) to the other machines with no good results.
It is a very strange behaviour and I do not understand why it happens. I understand and know that different convergence paths can be obtained and perfect reproducibility can be obtained only by providing the same seeds, but I still do not understand how such a phenomenon is possible.
@Art did you manage to solve this problem? If so, would you be so kind as to give me an explanation how you did solve it?