Thing is that on Windows 10 CUDA 10.1 with PyTorch 1.6 the UNET with different backbones seems to always start to converge after a point, while on Linux it always fails miserably.
I was starting to think that something is wrong with my code.
Nice+Sad   to see similar phenomenon happening.
Thank you for the discussion and good luck in your endeavours.