I am new to pytorch and tried to Run a seq2seq model. My code runs well for few batches (around 4 to 5) before I get the following error:
…
RuntimeError: cuda runtime error (59) : device-side assert triggered at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:76
When I enable cudnn, I get the following error:
…
cudnn.rnn.forward(self, input, hx, weight, output, hy)
File “/u/kumarkun/.local/lib/python2.7/site-packages/torch/backends/cudnn/rnn.py”, line 241, in forward
w.zero_()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu:26
Strangely, the source code line-number that causes the error is different for different runs but
Has anyone encountered such an issue before? Is this a known issue and is there an easy way to fix this?
I would greatly appreciate any pointers.