[resolved] Cuda Runtime Error(30)
<SCODE>THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=109 error=30 : unknown error
Traceback (most recent call last):
  File "trainer.py", line 13, in <module>
    if torch.cuda.is_available():
  File "/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.py", line 30, in is_available
    return torch._C._cuda_getDeviceCount() > 0
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:109

<ECODE>
ycszen
There must be something wrong with your driver. Maybe try rebooting?
apaszke
OK. I have found the problem. After I update the linux system, the driver become useless. So I will reinstall the driver. Thank you for your reply.
ycszen
I get this when I put my laptop to sleep while in the middle of training. When I put it to sleep, my script stops and I get this error: <SCODE>THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generated/../THCReduceAll.cuh line=334 error=4 : unspecified launch failure
Traceback (most recent call last):
  File "trytry.py", line 111, in <module>
    loss = network.loss(prediction, label_batch) + 10*torch.mean(cheat_amount)
  File "trytry.py", line 73, in loss
    union = 1e-5 + prediction.sum() + label.sum()
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py", line 437, in sum
    return Sum(dim)(self)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/_functions/reduce.py", line 16, in forward
    return input.new((fn(),))
RuntimeError: cuda runtime error (4) : unspecified launch failure at /b/wheel/pytorch-src/torch/lib/THC/generated/../THCReduceAll.cuh:334
<ECODE> And afterward I get this: <SCODE>THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/THCGeneral.c line=66 error=30 : unknown error
Traceback (most recent call last):
  File "trytry.py", line 77, in <module>
    network = Net()
  File "trytry.py", line 57, in __init__
    self.squeezenet = models.squeezenet1_1(pretrained=True).features.cuda() 
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 147, in cuda
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 124, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 147, in <lambda>
    return self._apply(lambda t: t.cuda(device_id))
  File "/usr/local/lib/python3.5/dist-packages/torch/_utils.py", line 65, in _cuda
    return new_type(self.size()).copy_(self, async)
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/__init__.py", line 272, in __new__
    _lazy_init()
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/__init__.py", line 85, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /b/wheel/pytorch-src/torch/lib/THC/THCGeneral.c:66
<ECODE> A reboot has fixed the problem. This is with CUDA 8.0 and an nvidia 1060.
chrisranderson
I also have the same issue after the laptop wakes up. I think this is a bug. Tensorflow seems to work fine in such situations.
kindlychung
Just had the same failure after wake/sleep for desktop. Pytorch 0.2, ubuntu 16.04
arogozhnikov
Same here! GTX1050ti Ubuntu16.04, reboot fixes it, but one short sleep then wake breaks it!
hvasbath
Same here. ubuntu 16.04 cuda9, pytorch 0.2
psavine42
Have anyone solved this problem? Thanks.
Ke_Bai
This also happens on my system Ubuntu 16.04 Nvidia GeForce 940MX PyTorch 0.3.1 running on Python 3.6 Cuda 8.0 CUDNN 7 Any clues? I don’t see why this thread is marked as resolved, if the solution is to restart the laptop every time.
danakianfar
Same problem for me too
Ste_Millington
Run into the same problem
Adam_Harrison
reboot fixes the problem.
Jimmy_Xiaoke_Shen
Same problem some strange stuff after wake up (desktop ubuntu 16.04 cuda 8 1080 gtx)
Marat
happytaoxiaoli
<SCODE>RuntimeError: cuda runtime error (30) : unknown error at 
/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCGeneral.cpp:70
<ECODE> Ubuntu 18.04, Pytorch 0.4.1, cuda 9.2
waleeka
farhat_Ullah
<SCODE>sudo rmmod nvidia_uvm
sudo rmmod nvidia
sudo modprobe nvidia
sudo modprobe nvidia_uvm
<ECODE>
miladiouss
Go to NVIDIA Nsight Options and set ‘Enable Crash Detection And Handling = True’. Did the trick for me.
Mohamed_Ghadban
This always works for me (Win10, Cuda 10.1, Python 3.7.2, PyTorch 1.0.1, NVIDIA GTX 1050 Ti): <SCODE>import torch
torch.cuda.current_device()
<ECODE> but this always fails for me: <SCODE>import torch
torch.cuda.is_available()
torch.cuda.current_device()  # fails here
<ECODE>
AndreiCostinescu