I’m getting a CUDA error: unspecified launch failure error when training for longer periods of time. When I was training for few hours for 20 epochs everything worked fine, but when I train for over 40 epochs my code terminates randomly with a CUDA error: unspecified launch failure.
I’m using pin_memory=False in my dataloader, if that matters, and when I train one epoch my GPU consumption is at 2gb/8gb. Are there any GPU debugging tricks i can use to find out where the problem is?
The fact that I not get the error at first, but only later makes me thinking its about GPU allocating resources
Traceback (most recent call last):
  File "c:/Users/C/Desktop/projects/M5/train.py", line 51, in <module>
    scheduler=scheduler, norm_factors=norm_vec, norm_idx=norm_idx)
  File "c:\Users\C\Desktop\projects\M5\seq2seq_forecasting\seq2seq_forecasting\core.py", line 413, in train
    outputs = model(x, target.clone())
  File "C:\Users\C\AppData\Local\Programs\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "c:\Users\C\Desktop\projects\M5\seq2seq_forecasting\seq2seq_forecasting\core.py", line 245, in forward
    outp, hiddens = self.decode_n(dec_inp,hiddens)
  File "c:\Users\C\Desktop\projects\M5\seq2seq_forecasting\seq2seq_forecasting\core.py", line 231, in decode_n
    outp = self.out_act(self.fc(o))
  File "C:\Users\C\AppData\Local\Programs\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\C\AppData\Local\Programs\Miniconda3\envs\torch\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\C\AppData\Local\Programs\Miniconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 1370, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: unspecified launch failure