CUDA Error when running run_language_modeling.py
I am trying to run this script[run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py)

I can load the pre-trained roberta model without a problem. However, when it starts training (loss.backward()), then there are issues like these:

> /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
> /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.

I assume it was caused by the `loss.backward()` line:

> 
> File "2_fine_tune_bert.py", line 386, in train
>     loss.backward()
>   File "/home/lily/zl379/anaconda2/envs/py36/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
>     torch.autograd.backward(self, gradient, retain_graph, create_graph)
>   File "/home/lily/zl379/anaconda2/envs/py36/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
>     allow_unreachable=True)  # allow_unreachable flag
> RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)

Is it caused by CUDA version or pytorch version? My pytorch is 1.4.0, CUDA: release 10.1, V10.1.243.
Thanks.
This seems related to the classes that you use in NLLLoss, That loss function expects a torch.LongTensor with values in the range [0, nb_classes-1] with no values left out in between.<|||||>Which version of `transformers` do you have installed? There was a recent change from using -1 to -100 for tokens that should be ignored during the calculation of loss. For example in this line: https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py#L218. Therefore, running the latest version of run_language_modeling.py with older versions of `transformers` will give an error similar to what you are seeing.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Problem solved, it was run with Python 3.6+. Return to Python 3.5, no more errors.<|||||>> Problem solved, it was run with Python 3.6+. Return to Python 3.5, no more errors.

That should not be the problem. The repo officially only supports 3.6+ any way. <|||||>I'm seeing exactly the same thing training Roberta using `run_mlm.py`. It's at the same step in the training cycle so I can reproduce but I've not tracked down what the issue is, either there's a problem with my input data in a single batch, or perhaps the training has diverged so forward() produces NaN's.

I'll keep digging.