IndexError: index out of range in self
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Ubuntu Linux
- Python version:3.88
- PyTorch version (GPU?):1.8.1
- Tensorflow version (GPU?):N/A
- Using GPU in script?:No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@LysandreJik
@patrickvonplaten
@patil-suraj 
## Information

Model I am using (Bert, XLNet ...):
BERT

The problem arises when using:
my own modified scripts

The tasks I am working on is:
word sense disambiguation

## To reproduce

Steps to reproduce the behavior:


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```

class my_BERT(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.bert = BertModel(config)
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)

        self.ranking_linear = torch.nn.Linear(config.hidden_size, 1)

        self.init_weights()

...
model_dir = "... bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6"


model = my_Bert.from_pretrained(model_dir)
tokenizer = BertTokenizer.from_pretrained(model_dir)
...

model.bert(
                  input_ids=torch.tensor(bert_input.input_ids, dtype=torch.long).unsqueeze(0).to(DEVICE),
                  attention_mask=torch.tensor(bert_input.input_mask, dtype=torch.long).unsqueeze(0).to(DEVICE),
                  token_type_ids=torch.tensor(bert_input.segment_ids, dtype=torch.long).unsqueeze(0).to(DEVICE)
              )[1]
          )
```

```
Traceback (most recent call last):
  File "... my_bert.py", line 605, in <module>
    print(my_match.matching())
  File "... my_bert.py", line 595, in matching
    best_sense = get_sense(modified_sentence)
  File "..my_bert.py", line 565, in get_sense
    model.bert(
  File "... my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "... my_env/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 964, in forward
    embedding_output = self.embeddings(
  File "... my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "... my_env/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 201, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "... my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "... my_env/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 156, in forward
    return F.embedding(
  File "... my_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1916, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

## Expected behavior

When I run my code with transformers==2.9.0, there is no problem, but when I upgrade to the newer version (4.6.1), this error appears.

Here is the [url ](https://entuedu-my.sharepoint.com/personal/boonpeng001_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fboonpeng001%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2FBERT%2DWSD%2Fmodel%2Fbert%5Fbase%2Daugmented%2Dbatch%5Fsize%3D128%2Dlr%3D2e%2D5%2Dmax%5Fgloss%3D6&originalPath=aHR0cHM6Ly9lbnR1ZWR1LW15LnNoYXJlcG9pbnQuY29tLzpmOi9nL3BlcnNvbmFsL2Jvb25wZW5nMDAxX2VfbnR1X2VkdV9zZy9FaVd6YmxPeXlPQkR0dU8za2xVYlhvQUIzVEhGemtlLTJNTFdndUlYckRvcFdnP3J0aW1lPTZaYVhQYzVLMlVn)where I got the word sense disambiguation model:

Here is its GitHub [repo](https://github.com/BPYap/BERT-WSD):

This problem seems to be similar to this other [issue](https://github.com/huggingface/transformers/issues/5611), but I don't know what value I need to specify for the 'vocab_size' (the library said the vocab_size was 30523, but specifying this didn't change anything).

Also, I tried using a really small input sequence, but this error still came.

Thank you to anyone who is able to help!
Hello! Could you print `model.config.vocab_size` and `len(tokenizer)`? These two should be the same, as otherwise your tokenizer will generate IDs that the model cannot understand (which seems to be the case here)<|||||>@LysandreJik

```model.config.vocab_size``` returns 30523, and ```len(tokenizer)``` returns 30524

I tried adding ```vocab_size=30524``` as a parameter to ```model = my_Bert.from_pretrained(model_dir)```, but then I got the following error:
```
RuntimeError: Error(s) in loading state_dict for BertWSD:
        size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([30523, 768]) from checkpoint, the shape in current model is torch.Size([30524, 768]).
```

<|||||>Before trying to resize your model, we should understand why your tokenizer is suddenly bigger than it was in version v2.9.0. Resizing the model embedding is possible, but it implies initializing randomly the embeddings linked to the last token that your tokenizer can generate - which isn't ideal.

Would it be possible for you to upload your model and tokenizer on the hub, so that I may take a look? You can see a guide of how to do it here: https://huggingface.co/docs/hub/adding-a-model<|||||>Thank you for your reponse @LysandreJik 

I am using a model and tokenizer from the following github [repo ](https://github.com/BPYap/BERT-WSD/tree/master/model) and the model and tokenizer themselves are in this microsoft sharepoint [folder](https://entuedu-my.sharepoint.com/:f:/g/personal/boonpeng001_e_ntu_edu_sg/EiWzblOyyOBDtuO3klUbXoAB3THFzke-2MLWguIXrDopWg?e=08umXD). If the link to the sharepoint folder doesn't work, please look at "bert_base-augmented-batch_size=128-lr=2e-5-max_gloss=6" under the github repo's models folder. Please let me know if this is sufficient, or whether I should still upload this model to the hugging face hub.<|||||>@LysandreJik

It appears that in the ```config.json``` (I am pretty sure this is for the model), the ```vocab_size``` is  30523, which matches the results from above. The ```tokenizer_config.json``` doesn't include a ```vocab_size``` key in the json object. When I go to the ```vocab.txt```, it shows that there are 30522 tokens (line 30523 is blank). However, this doesn't match the results from above, where len(tokenizer) returned 30524. Do you think this may be impacting why the model works for transformers==2.9.0, but doesn't work for transformers==4.6.1?
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I have added custom tokens using tokenizer.add_tokens and ran train(), got the same error. I found out that, need to resize model like this model.resize_token_embeddings(len(tokenizer)) and it worked.
So, every call of add_token should be followed by resize model.

Check if it's the case with yours as well.