Longformer config - vocabulary size
## Environment info
- `transformers` version: 3.1.0

### Who can help
 Longformer/Reformer: @patrickvonplaten
 
## Information

Why does the LongformerConfig's vocab_size defaults to 30522 while the Longformer has embeddings first dimension 50265? It reuses the same config as RoBERTa which also has embeddings shape (50265,768) due to the BPE tokenization (and also defaults to 30522). 

```python
LongformerConfig {
  "attention_probs_dropout_prob": 0.1,
  "attention_window": 512,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "longformer",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "sep_token_id": 2,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
```
Hey @blawok, 

which model are you referring to exactly? This longformer model: https://s3.amazonaws.com/models.huggingface.co/bert/allenai/longformer-base-4096/config.json has vocab_size set to 50265 <|||||>Thanks for the answer @patrickvonplaten :) 

Using the code from the docs (https://huggingface.co/transformers/model_doc/longformer.html#longformerconfig):

```python
from transformers import LongformerConfig, LongformerModel
# Initializing a Longformer configuration
configuration = LongformerConfig()
# Initializing a model from the configuration
model = LongformerModel(configuration)
# Accessing the model configuration
configuration = model.config
print(configuration)
```

I am getting this output:

```
LongformerConfig {
  "attention_probs_dropout_prob": 0.1,
  "attention_window": [
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "longformer",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "sep_token_id": 2,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
```

It resulted in error while trying to train:
```python
config = LongformerConfig()
model = TFLongformerModel.from_pretrained('allenai/longformer-base-4096', config=config) 
```
LongformerTokenizer correctly used the 50265 vocab, but the model expected it to be 30522. However, I am not getting this error when I am not specifying the configuration.

Example to reproduce:

```python
from transformers import LongformerTokenizer, TFLongformerModel, LongformerConfig

tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
input = tokenizer('Hello world')

config = LongformerConfig()
model = TFLongformerModel.from_pretrained('allenai/longformer-base-4096',
                                          config=config) 
```

Error I am getting with the code above:

```python
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-4-80be6efb2343> in <module>()
      6 config = LongformerConfig()
      7 model = TFLongformerModel.from_pretrained('allenai/longformer-base-4096',
----> 8                                           config=config) 
      9 print(model(input))

2 frames

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group_by_name(f, layers, skip_mismatch)
    784                                symbolic_weights[i])) +
    785                            ', but the saved weight has shape ' +
--> 786                            str(weight_values[i].shape) + '.')
    787 
    788         else:

ValueError: Layer #0 (named "longformer"), weight <tf.Variable 'tf_longformer_model/longformer/embeddings/word_embeddings/weight:0' shape=(30522, 768) dtype=float32, numpy=
array([[-0.01930627, -0.01715518, -0.00557071, ..., -0.01202598,
         0.01007012, -0.00184635],
       [ 0.00633493,  0.00123013, -0.0134872 , ..., -0.01304915,
        -0.00157391,  0.00082429],
       [-0.01581489,  0.01005882, -0.01242067, ...,  0.00555116,
         0.02116241,  0.03123646],
       ...,
       [-0.01625618,  0.01438301,  0.03368756, ..., -0.02742909,
         0.00300512,  0.00728624],
       [-0.0078434 , -0.01735217,  0.00178284, ..., -0.01191203,
        -0.01451435,  0.03031485],
       [-0.00814894,  0.01228636,  0.00573935, ...,  0.01143655,
        -0.00131886, -0.03910364]], dtype=float32)> has shape (30522, 768), but the saved weight has shape (50265, 768).
```<|||||>Hi @blawok, you're initializing a configuration using the default parameters, which may not be the same as the checkpoint's parameters you're initializing (it's not the case here).

You should initialize the configuration from the checkpoint here too:

```py
config = LongformerConfig.from_pretrained('allenai/longformer-base-4096')
model = TFLongformerModel.from_pretrained('allenai/longformer-base-4096', config=config)
```<|||||>Great, thank you for the explanation @LysandreJik :) 

I am closing this issue.