I am training my model for several epochs without errors and suddenly got a cublas runtime error. The complete stack trace is:

Traceback (most recent call last):
File “train_revise.py”, line 405, in 
main()
File “train_revise.py”, line 363, in main
model, criterionCTC, optimizer, data, target)
File “train_revise.py”, line 144, in train_batch
cost.backward()
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/haojiedong/tools/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py”, line 90, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cublas runtime error : an internal operation failed at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCBlas.cu:249

If I resume training, I will get this error right away. So I can not start training now.
The solution poposed on this related post 2  does not work.