Hi, yeah we encountered the same issue a few days ago where one of our two RTX2080Ti was blowing up the activations and the gradients without any reason, resulting in inf and nan all over the place. The other card works just fine. Here is the thread we were running: https://discuss.pytorch.org/t/different-losses-on-2-different-machines/ 90.
Thank you