I often met a problem that PyTorch reported Error(30) after I ran train.py or infer.py using PyTorch. I always resolved it through 3 ways:

reboot system, then the problem is dissapeared.
Reinstall NVIDIA driver and CUDA if rebooting system can’t resolve problem.
Reinstall system. Sometimes reboot system and reinstall driver still can’t resolve the problem, I must reinstall system. I don’t think it’s the problem of NVIDIA driver or CUDA, because both nvidia-smi and deviceQuery can work normally after I reboot system or reinstall driver. Although deviceQuery can work normally, it will report Error 30 as long as PyTorch reports error when I run train.py or infer.py. So I think PyTorch makes this problem, but I don’t know why and how to resolve it, it bothers me for a long time. Anyone can give me some advices? Thanks!

Enviroment:
NVIDIA 1080 or 1080TI
CUDA 8.0, 9.0 or 10.0
ubuntu 16.04 or 18.04
PyTorch 0.4.0 or 1.0.0
I encountered this problem in all above enviroment.