Bart tokenizer and bart model for conditional generation have different vocab size 
## Environment info

- `transformers` version: 4.6.1
- Platform: Google colab
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1
- Using GPU in script?: No


### Who can help
@patrickvonplaten, @patil-suraj

Models:

- bart


Library:

- text generation: @patrickvonplaten
- tokenizers: @LysandreJik



## Information

Model I am using (BART):

The problem arises when using the pretrained model of bart : 'bart-large-xsum'. 
I tried to load the tokenizer and the model from `bart-large-xsum`.
I then tried to add mask to the inputs (as mentionned in the original paper section 5.1 (https://arxiv.org/pdf/1910.13461.pdf)).
But the tokenizer and the bart model don't have the same vocab size.
Does it mean the `bart-large-xsum` model doesn't take masks as inputs ? Do I need to add it to the vocabulary myself ? 



## To reproduce

Steps to reproduce the behavior:

```python
tokenizer =  BartTokenizer.from_pretrained('facebook/bart-large-xsum')
bart = BartForConditionalGeneration.from_pretrained('facebook/bart-large-xsum')
print(tokenizer.vocab_size)
print(bart.config.to_dict()['vocab_size'])
```
![Screenshot 2021-05-25 at 11 59 44](https://user-images.githubusercontent.com/44928643/119479244-bebca700-bd50-11eb-9a9f-e438abd66592.png)




Hi there,

This is because the [orginal](https://github.com/pytorch/fairseq/tree/master/examples/bart#pre-trained-models) `bart-large-xsum` model uses 50264 for token_embedding, so you would probably need to extend the token embedding layer.