RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmStridedBatched (handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)
## Environment info

NVIDIA-SMI 495.46
Driver Version: 460.32.03
CUDA Version: 11.2   

- `transformers` version: 4.16.2
- Platform: Google Colab Pro +
- Python version: 3.7
- PyTorch version (GPU?): 1.10.0+cu111
- Using GPU in script?: Yes

Error:

```
Saving model checkpoint to ../models/checkpoints/checkpoint-287
Configuration saved in ../models/checkpoints/checkpoint-287/config.json
Model weights saved in ../models/checkpoints/checkpoint-287/pytorch_model.bin
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f7f41780dc73> in <module>()
    157 )
    158 
--> 159 trainer.train()
    160 
    161 trainer.save_model()

14 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)
    273             attention_probs = attention_probs * head_mask
    274 
--> 275         context_layer = torch.matmul(attention_probs, value_layer)
    276 
    277         context_layer = context_layer.permute(0, 2, 1, 3).contiguous()

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`
```

Models:
- RoBERTa

Library:
- Trainer: @sgugger

## To reproduce

Steps to reproduce the behavior:
 I'm calling the trainer with `trainer.train()` and training a model for 10 epochs. I'm %99 percent sure I don't have a memory issue since I'm using a high ram node. The model successfully gets trained for one epoch and checkpoints are saved (as seen in the log,) but as soon as the second epoch starts, I see the error.

## Expected behavior
Completing the train without error!

P.S. Just to add, the very script in the very environment with the same GPU works perfectly fine with `bert-large-*` models. So I wonder if this error could have anything to do with the RoBERTa model?
With no information on how to reproduce the error, there is little we can do to help.<|||||>> With no information on how to reproduce the error, there is little we can do to help.

Thanks for your response. Unfortunately, the data needed for reproducibility is proprietory and not public, so I can't share it for reproducibility. I thought this might be a common or known issue.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I meet similar problem on an A100 GPU with 11.7 cuda.