pytorch_model.bin file is different after uploading to HuggingFace Models
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `distilroberta-base`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:

1. Save my model with `.save_pretrained()`
2. Download the models `pytorch_model.bin`
3. Check the `diff` between `pytorch_model.bin` _before_ uploading and _after_ downloading, it is not the same.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

I first noticed that a model I have trained produced different outputs when I loaded it from a local directory compared to uploading it to https://huggingface.co/models and downloading it. Unfortunately, the error is a little hard to reproduce as you need access to my saved model. I have uploaded it [here](https://drive.google.com/file/d/1C8okSoS4tJHtZllQ8qIJbmXRLb8ITL6N/view?usp=sharing). With that model downloaded, I compare its outputs before/after uploading:

_Before uploading_ (e.g. loading the model from a local directory)

```python
import torch
from scipy.spatial.distance import cosine

from transformers import AutoModel, AutoTokenizer

# Load the model
tokenizer = AutoTokenizer.from_pretrained("declutr-small")
model = AutoModel.from_pretrained("declutr-small")

# Prepare some text to embed
text = [
    "A smiling costumed woman is holding an umbrella.",
    "A happy woman in a fairy costume holds an umbrella.",
]
inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")

# Embed the text
with torch.no_grad():
    sequence_output, _ = model(**inputs, output_hidden_states=False)

# Mean pool the token-level embeddings to get sentence-level embeddings
embeddings = torch.sum(
    sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)

# Compute a semantic similarity via the cosine distance
semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
print(semantic_sim)  # => ~0.83
```

_After uploading_ (e.g. loading the model from https://huggingface.co/models)

```python
# Load the model
tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small", force_download=True)
model = AutoModel.from_pretrained("johngiorgi/declutr-small", force_download=True)

# Prepare some text to embed
text = [
    "A smiling costumed woman is holding an umbrella.",
    "A happy woman in a fairy costume holds an umbrella.",
]
inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")

# Embed the text
with torch.no_grad():
    sequence_output, _ = model(**inputs, output_hidden_states=False)

# Mean pool the token-level embeddings to get sentence-level embeddings
embeddings = torch.sum(
    sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)

# Compute a semantic similarity via the cosine distance
semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
print(semantic_sim)  # => ~0.99, NOT the same as the local model!
```

The embeddings must be different, as their semantic similarity is. After some more digging, I realized that the `pytorch_model.bin` of the local model and the uploaded then downloaded model are not the same, which I checked with `diff`. 

I tried everything I could think of, deleting my `transformers` cache folder, deleting the model from https://huggingface.co/models and re-uploading. I also tried uploading it from both macOS/Linux. The error persists.

Does anyone have any clue how this could happen? It's such a frustrating error b/c it essentially passes silently until you look at your models outputs.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I expect that the outputs of my model to be identical when I load it from a local directory, and when I upload it and then download it from https://huggingface.co/models.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

Very strange, I just tried the code and it returns `0.8289950489997864`. Maybe you try to re-download the model with:

```python
# Load the model
tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small", force_download=True)
model = AutoModel.from_pretrained("johngiorgi/declutr-small", force_download=True)
```

Hopefully this helps :)<|||||>Hi @stefan-it,

I did try with `force_download=True` a bunch of times but it never worked. I updated the example to include that.

Hmm, mind sending the exact code you used to get the `0.8289950489997864`? I had a collegue run the code on their own system and like me they got the wrong answer of: `0.9928748607635498`. I tried on two machines (linux and mac), both produce `0.9928748607635498`.

Just to be sure, I deleted my conda environment, made a new one and reinstalled `transformers`. Then I deleted the default cache dir (for me this was at `~/.cache/torch/transformers/`). Finally, I tried the code again this time with `force_download=True`. No beans, exactly the same issue and the `semantic_sim` is ~0.99:


```bash
In [1]: import torch
   ...: from scipy.spatial.distance import cosine
   ...:
   ...: from transformers import AutoModel, AutoTokenizer

In [2]: tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small", force_download=True, cache_dir="./declutr-small")
   ...: model = AutoModel.from_pretrained("johngiorgi/declutr-small", force_download=True, cache_dir="./declutr-small")
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 547/547 [00:00<00:00, 277kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 798k/798k [00:00<00:00, 4.71MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 3.83MB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 157kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54.0/54.0 [00:00<00:00, 35.4kB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 547/547 [00:00<00:00, 403kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 331M/331M [00:14<00:00, 23.2MB/s]

In [3]: # Prepare some text to embed
   ...: text = [
   ...:     "A smiling costumed woman is holding an umbrella.",
   ...:     "A happy woman in a fairy costume holds an umbrella.",
   ...: ]
   ...: inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")
   ...:
   ...: # Embed the text
   ...: with torch.no_grad():
   ...:     sequence_output, _ = model(**inputs, output_hidden_states=False)
   ...:
   ...: # Mean pool the token-level embeddings to get sentence-level embeddings
   ...: embeddings = torch.sum(
   ...:     sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
   ...: ) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)
   ...:
   ...: # Compute a semantic similarity via the cosine distance
   ...: semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
   ...: print(semantic_sim)  # => ~0.99, NOT the same as the local model!
0.992874801158905
```<|||||>@JohnGiorgi I just ran:

```python
import torch
from scipy.spatial.distance import cosine

from transformers import AutoTokenizer, AutoModel

# Load the model
tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small", force_download=True)
model = AutoModel.from_pretrained("johngiorgi/declutr-small", force_download=True)

# Prepare some text to embed
text = [
    "A smiling costumed woman is holding an umbrella.",
    "A happy woman in a fairy costume holds an umbrella.",
]
inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")

# Embed the text
with torch.no_grad():
    sequence_output, _ = model(**inputs, output_hidden_states=False)

# Mean pool the token-level embeddings to get sentence-level embeddings
embeddings = torch.sum(
    sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)

# Compute a semantic similarity via the cosine distance
semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
print(semantic_sim)
```

I re-downloaded the model and it still returns 0.8289950489997864 😅<|||||>That is maddening, I literally copy-pasted that code and I get `0.992874801158905` 😢 Thanks anyways for confirming it works somewhere at least!

<img width="1440" alt="image" src="https://user-images.githubusercontent.com/8917831/87212687-d1232300-c2ed-11ea-9039-3b05077be964.png">
