Actually, I am not sure how I solved this. After I re-installed pytorch, cuda/cudnn, nvidia-driver, the problem had gone away.
Also, this could be a hardware problem, since the problem occured on a flawed workstation (multi-gpu training might make it crash).