I was asking because you mentioned that changing the BN eps parameter helped. PyTorch binaries of different versions ship with different cudnn (later PyTorch versions ship with newer cudnn of course). So it might be cudnn related as well. If the memory is not too big an issue, maybe you want to run with cudnn disabled and see if it helps.