RuntimeError: cublas runtime error : an internal operation failed at /pytorch/aten/src/THC/THCBlas.cu:258
I implemented my model referring to the implementation of the examples, when my model running several batches, the error shown in the title occurs.
The whole trace is as followed:
Traceback (most recent call last):
  File "train.py", line 549, in <module>
    train()
  File "/media/***/***/***/***/***.py", line 120, in forward
    topic_representation = self.bertRepresentation(topic_ids,topic_type_ids,topic_mask)
  File "/media/***/***/***/***/***.py", line 113, in bertRepresentation
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/Downloads/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py", line 736, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/Downloads/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py", line 409, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/Downloads/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py", line 394, in forward
    attention_output = self.attention(hidden_states, attention_mask)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/Downloads/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py", line 352, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/Downloads/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py", line 303, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/***/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1354, in linear
    output = input.matmul(weight.t())
RuntimeError: cublas runtime error : an internal operation failed at /pytorch/aten/src/THC/THCBlas.cu:258
In the trace, self.bert is the function to call the BERT model, I am looking forward to all the suggestions from all of you. I have google this problem and there are no methods could solve this problem.
I also run my model using CPU, although it is very slow, no errors.
Python version is 3.6, the cuda version is 10.1.105, cudnn version is  7.51, and pytorch version is 1.0.1, platform is ubuntu 14.04. And GPU is Titan GTX 1080Ti with 11g memory.
Thanks all of you!<|||||>I am running into almost exactly the same issue. python3.6 cuda 10.0 ubuntu 18.04 and a 1080ti as well. Not sure what it is coming from. I see lots of google results from 2080ti's but those looked like architecture issues. <|||||>I tried running with CUDA_LAUNCH_BLOCKING=1 because others have stated this will get us a more accurate error. This led to RuntimeError: Creating MTGP constants failed. at /pytorch/aten/src/THC/THCTensorRandom.cu:33
. Doing some searching as to why this might be. Some people said indexing error somewhere. seems to be happening in the dropout area though so I dont see how that is possible. <|||||>https://github.com/pytorch/pytorch/issues/20489
https://github.com/pytorch/pytorch/pull/20886
These look related<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Python version is 3.6, the cuda version is 10.0 , cudnn version is 7.5, and pytorch version is 1.1.0, platform is ubuntu 16.04. And GPU is Titan GTX 2080Ti.
Solve this problem.