It could be a race condition or just a coincidence, since you said the issue was non-deterministic.
Could you update your PyTorch installation to use CUDA10.2 and rerun the script?
If you are still seeing this issue (and have a local CUDA installation), you could run a single training iteration (forward and backward) with cuda-memcheck --tool racecheck to check the code for potential races.