`run_glue.py` fails with models `bert-base-cased`, `distil-bert-cased`, others
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `bert-base-cased`

Language I am using the model on (English, Chinese ...): `English`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download GLUE data
2. run `python run_glue.py --model_name_or_path bert-base-cased`

Error message:

```
06/07/2020 00:01:06 - INFO - transformers.trainer -   ***** Running training *****
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num examples = 3668
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num Epochs = 3
06/07/2020 00:01:06 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total optimization steps = 2751
Iteration:   1%|█▊                                                                                                                                                                                            | 9/917 [00:24<40:42,  2.69s/it]
Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:24<?, ?it/s]
Traceback (most recent call last):
  File "./run_glue.py", line 247, in <module>
    main()
  File "./run_glue.py", line 174, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1143, in forward
    inputs_embeds=inputs_embeds,
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to a
ccess index 29597 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
## Expected behavior

Model successfully trains. The script works well on my machine for many other models, including `bert-base-uncased` ~~and `distilbert-base-cased`~~.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
```
- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: [yes]
- Using distributed or parallel set-up in script?: [yes, parallel, but I received the same error when I ran the script just using the CPU]
```s
update: this also happens with `distil-bert-cased` for `RTE` and `WNLI` tasks:

```
ds)  # (bs, seq_length, dim)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_distilbert.py", line 91, in forward
    word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 29236 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```<|||||>update: a different error occurs with `roberta-base` on `STS-B`:

```
06/06/2020 23:49:26 - INFO - transformers.trainer -   ***** Running training *****
06/06/2020 23:49:26 - INFO - transformers.trainer -     Num examples = 5749
06/06/2020 23:49:26 - INFO - transformers.trainer -     Num Epochs = 3
06/06/2020 23:49:26 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
06/06/2020 23:49:26 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
06/06/2020 23:49:26 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/06/2020 23:49:26 - INFO - transformers.trainer -     Total optimization steps = 1080
Iteration:   0%|                                                                                                                                                                                                      | 0/360 [00:09<?, ?it/s]
Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:09<?, ?it/s]

wandb: Waiting for W&B process to finish, PID 7670
Traceback (most recent call last):
  File "./run_glue.py", line 246, in <module>
    main()
  File "./run_glue.py", line 173, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    wandb: Program failed with code 1. Press ctrl-c to abort syncing.
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bart.py", line 1103, in forward
    loss = F.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
```<|||||>Hello! I tried to reproduce, but couldn't get any results to crash.
Do you mind showing me the exact command you use to launch the script?<|||||>Thanks for the response @LysandreJik -- here's an example of one command:

```
CUDA_VISIBLE_DEVICES="" python run_glue.py --model_name_or_path bert-base-cased --tokenizer_name bert-base-cased   --task_name MRPC   --do_train --do_eval --save_steps -1   --data_dir=./glue_data/MRPC/   --max_seq_length 256   --per_device_eval_batch_size=16   --per_device_train_batch_size=16   --learning_rate 2e-5   --num_train_epochs 3   --output_dir=./glue_models/bert-base-cased/MRPC/
```

Here's the full output:
```
06/11/2020 13:19:37 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /u/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1
Made tokenizer: <transformers.tokenization_bert.BertTokenizer object at 0x7fef6cd09550>
06/11/2020 13:19:38 - INFO - transformers.modeling_utils -   loading weights file https://cdn.huggingface.co/bert-base-cased-pytorch_model.bin from cache at /u/.cache/torch/transformers/d8f11f061e407be64c4d5d7867ee61d1465263e24085cfa26abf183fdc830569.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
06/11/2020 13:19:41 - INFO - transformers.modeling_utils -   Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
06/11/2020 13:19:41 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173336656 acquired on ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.data.datasets.glue -   Loading features from cached file ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc [took 0.110 s]
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173336656 released on ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173337552 acquired on ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.data.datasets.glue -   Loading features from cached file ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc [took 0.013 s]
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173337552 released on ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.trainer -   Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Tracking run with wandb version 0.8.36
wandb: Wandb version 0.9.1 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Run data is saved locally in wandb/run-20200611_171941-90an9vn0
wandb: Syncing run solar-sun-89
wandb: ⭐️ View project at https://app.wandb.ai/jxmorris12/huggingface
wandb: 🚀 View run at https://app.wandb.ai/jxmorris12/huggingface/runs/90an9vn0
wandb: Run `wandb off` to turn off syncing.

06/11/2020 13:19:43 - INFO - transformers.trainer -   ***** Running training *****
06/11/2020 13:19:43 - INFO - transformers.trainer -     Num examples = 3668
06/11/2020 13:19:43 - INFO - transformers.trainer -     Num Epochs = 3
06/11/2020 13:19:43 - INFO - transformers.trainer -     Instantaneous batch size per device = 16
06/11/2020 13:19:43 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
06/11/2020 13:19:43 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/11/2020 13:19:43 - INFO - transformers.trainer -     Total optimization steps = 690
Iteration:   1%|█▍                                                                                                                                                                          | 2/230 [00:16<31:42,  8.35s/it]
Epoch:   0%|                                                                                                                                                                                          | 0/3 [00:16<?, ?it/s]
Traceback (most recent call last):
  File "run_glue.py", line 247, in <module>
    main()
  File "run_glue.py", line 174, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1143, in forward
    inputs_embeds=inputs_embeds,
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to acc
wandb: Waiting for W&B process to finish, PID 46776
ess index 29597 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: Process crashed early, not syncing files
```

But -- at least on my machine -- I think a host of combinations of model/task combinations (`--model_name_or_path` and `--data_dir`) fail, as I mentioned above.<|||||>I had the same problem, but on my own data set. Have you solved your problem?<|||||>Nope @SizhaoXu.<|||||>@LysandreJik -- have you had a chance to check this out again? Thanks.<|||||>> Nope @SizhaoXu.

For question: RuntimeError: index out of range: Tried to access index 29597 out of table with 28995 rows.
The reason for this question is that the maximum sequence length of the model is 512. <|||||>> > Nope @SizhaoXu.
> 
> For question: RuntimeError: index out of range: Tried to access index 29597 out of table with 28995 rows.
> The reason for this question is that the maximum sequence length of the model is 512.

This solves my problem. I hope that will help you<|||||>@SizhaoXu how did you fix it then? by truncating the inputs?<|||||>> @SizhaoXu how did you fix it then? by truncating the inputs?

yes! you can try it. The maximum sequence length I set is 512 and I keep the first 200 words, the last 200 words and the middle 112 words<|||||>Thanks @SizhaoXu but my problems go beyond that specific one.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> update: a different error occurs with `roberta-base` on `STS-B`:
> 
> ```
> 06/06/2020 23:49:26 - INFO - transformers.trainer -   ***** Running training *****
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Num examples = 5749
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Num Epochs = 3
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Total optimization steps = 1080
> Iteration:   0%|                                                                                                                                                                                                      | 0/360 [00:09<?, ?it/s]
> Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:09<?, ?it/s]
> 
> wandb: Waiting for W&B process to finish, PID 7670
> Traceback (most recent call last):
>   File "./run_glue.py", line 246, in <module>
>     main()
>   File "./run_glue.py", line 173, in main
>     model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
>     tr_loss += self._training_step(model, inputs, optimizer)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
>     outputs = model(**inputs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
>     outputs = self.parallel_apply(replicas, inputs, kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
>     wandb: Program failed with code 1. Press ctrl-c to abort syncing.
> return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
>     output.reraise()
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
>     raise self.exc_type(msg)
> RuntimeError: Caught RuntimeError in replica 0 on device 0.
> Original Traceback (most recent call last):
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
>     output = module(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bart.py", line 1103, in forward
>     loss = F.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1838, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
> ```

Hello ! I encountered the same problem when using the STSB dataset for fine-tuning BERT. How did you solve this problem?