Camenbert length Tokenizer not equal config vocab_size
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi there,
when I load the pretrained Camenbert model and tokenizer via 

`model = CamembertForMaskedLM.from_pretrained('camembert-base')
tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`

the length of the tokenizer is 32004 but the vocab_size of the model is 32005.
`print(len(tokenizer))`
'print(model.config.vocab_size'

This throws me an error
> Index out of range

when I try to adapt the lm_finetuning example because of 
`model.resize_token_embeddings(len(tokenizer))`

It runs when I comment out this line. So my question is, is this the intended behaviour resp. what's the reason for the unevenness between the tokenizer and the model vocab_size?
Indeed, upon deeper investigation, it appears that the original fairseq model has a bunch of duplicate tokens in the dictionary:
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁de', '.']
```

I'm cleaning and updating for this in #2065<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
