OPT vocab size of model and tokenizer does not match
### System Info

- `transformers` version: 4.19.2
- Platform: Linux-5.4.0-72-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: no


### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('facebook/opt-350m')
tok = AutoTokenizer.from_pretrained('facebook/opt-350m', use_fast=False)
    
print(model.config.vocab_size)        # 50272
print(tok.vocab_size)                      # 50265
```


### Expected behavior

Hello,
I'm not sure whether this is a bug or if I am missing something.
In the reproduction script above, the model has a bigger vocabulary than the tokenizer. In my project, the LM produces the token `50272`, which the tokenizer doesn't know and thus the decode() function fails. 
(I use my own text generation script, so is it by any chance that the model is not supposed to output the last 7 tokens that the tokenizer doesn't know?)

Best, David
cc @ArthurZucker <|||||>Hey, thanks for noticing this! I am gonna add @younesbelkada  to the loop. 
It seems that the original tokenizer [vocabulary](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/assets/gpt2-vocab.json) has 50264 words, with some "madeupwords". Let us have a look! <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Duplicate of https://github.com/huggingface/transformers/issues/17431#issuecomment-1224231170<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.