Longformer pretrained weights are not really pretrained?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:3.0.2
- Platform:Ubuntu 18.04
- Python version:3.7
- PyTorch version (GPU?):1.6 Yes
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

 @patrickvonplaten


Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [V] my own modified scripts

The tasks I am working on is:
* [V] my own task or dataset: (give details below)

## To reproduce

As a preprocessing step in my pipeline I train a pre-trained model on a subset of Wikipedia dataset.
When using roBERTa the results of my fine-tuning steps are as follows (LM task):
![image](https://user-images.githubusercontent.com/31047807/90328740-71ccc880-dfa7-11ea-9df5-0163ae48533b.png)

Unfortunately, when simply plugging in longformer (and pointing the pretrained path to `allenai/longformer-base-4096`
![image](https://user-images.githubusercontent.com/31047807/90328734-5e216200-dfa7-11ea-888b-53e759fd2e29.png)
Which seems like the weights are simply random-initialized.
I both cases (roberta and longformer) I get the message stating the weights have been initialized.
What went wrong?

Hey @dvirginz,

Can you post a code snippet here so that we can reproduce your results or at least see what might have been falsely configured?<|||||>@patrickvonplaten Hi!
I'm sorry to have troubled you, the problem was with how I initialized and loaded the weights for the model.
It is now working, Thanks!