I am facing a similar issue while training with large tensors. The behaviour is not deterministic though. Varying (aka reducing) the batch size and the seed, the issue disappears in most of the cases. At the beginning I though it was due to insufficient memory capacity but then realized the two things are not related. I am basically stuck and now wondering how to effectively debug the code. Any thought/suggestion? The cuda-memcheck tool hasn’t helped much.