run_glue.py fails with RoBERTa but succeeds with other models
## Environment info
I'm following this instructions: https://github.com/huggingface/transformers/tree/master/examples, meaning I installed the library from source.
     
- `transformers` version: 4.1.0.dev0
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.7.0
- Using GPU in script?: Yes, Tesla K80
- Using distributed or parallel set-up in script?: running `CUDA_VISIBLE_DEVICES=0 python run_glue.py` 

 -->

## The problem

I'm running the official `run_glue.py` code, with the command and arguments given here: https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-pytorch-version

When I use BERT - it **succeeds**. 
For example, BERT: 
```CUDA_VISIBLE_DEVICES=0 python run_glue.py --task_name cola --output_dir results/normal/bert/cola/ --cache_dir cache/normal/bert --model_name_or_path bert-base-cased --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir```

And I receive score that makes sense:
```
[d]$ cat /transformers/examples/text-classification/results/normal/bert/cola/eval_results_cola.txt
eval_loss = 0.518086314201355
eval_matthews_correlation = 0.572739655014278
epoch = 3.0
```

When I use RoBERTA, it **fails** with a stacktrace: 
```CUDA_VISIBLE_DEVICES=0 python run_glue.py --task_name cola --output_dir results/normal/roberta/cola/ --cache_dir cache/normal/roberta --model_name_or_path roberta-base --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3.0 --do_predict --overwrite_output_dir```

The error message: 
```python
[INFO|trainer.py:674] 2020-12-04 20:23:30,937 >>   Total optimization steps = 804
  0%|                                                   | 0/804 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [33,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [33,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
Traceback (most recent call last):
  File "/transformers/examples/text-classification/run_yonatan.py", line 464, in <module>
    main()
  File "/transformers/examples/text-classification/run_yonatan.py", line 399, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/transformers/src/transformers/trainer.py", line 767, in train
    tr_loss += self.training_step(model, inputs)
  File "/transformers/src/transformers/trainer.py", line 1096, in training_step
    loss = self.compute_loss(model, inputs)
  File "/transformers/src/transformers/trainer.py", line 1120, in compute_loss
    outputs = model(**inputs)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 1029, in forward
    return_dict=return_dict,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 717, in forward
    return_dict=return_dict,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 450, in forward
    output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 368, in forward
    output_attentions=output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 302, in forward
    output_attentions,
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/src/transformers/models/roberta/modeling_roberta.py", line 184, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/transformers/trans_env/lib/python3.6/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
  0%|                                                   | 0/804 [00:00<?, ?it/s]
```

I've searched for related solutions but didn't find any relevant solution (https://github.com/huggingface/transformers/issues?q=CUBLAS_STATUS_ALLOC_FAILED).

What am I missing? 

Thanks
When **importing** transformers (instead of using the source) the problem does not occur.<|||||>Pinging the Trainer master @sgugger!<|||||>This looks like a problem on the CUDA initialization in your enviromnent. The command runs fine on my side. 

> When **importing** transformers (instead of using the source) the problem does not occur.

What do you mean exactly by this?<|||||>> This looks like a problem on the CUDA initialization in your enviromnent. The command runs fine on my side.
> 
> > When **importing** transformers (instead of using the source) the problem does not occur.
> 
> What do you mean exactly by this?

The note here: https://github.com/huggingface/transformers/tree/master/examples#important-note
suggests to install the library from source. When I do it (with git clone) it doesn't work - I receive the error described here.

On the other hand, when I use the 'transformers' from 'pip install transformers', it does work.

I'm not sure if this specific difference causes the error only in my environment or not. <|||||>This issue has been stale for 1 month.