I am struggling with the same issue. My non-local net converges with my local machine Windows 10 CUDA 11.2 RTX3060 with Pytorch 1.9.  But do not converges in Linux machine, CUDA 11.2 RTX3090 with Pytorch 1.9 .  What happens here? Is there anybody who knows how to fix it?