Pretrained bare model have weights that are not beein used
## Environment info
- `transformers` version: 4.12.5
- Platform: Linux-5.11.0-40-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Models:
- DistilBERT, BERT @LysandreJik

Documentation: @sgugger

## Information

The problem arises when using:
* [x] the official example scripts: (give details below)

## To reproduce

Hi I want to use the raw hidden-states from a pretrained bert-like model as input to another model.
I don't want a too big models so I tried the bare DistilBERT like this from the [docs](https://huggingface.co/transformers/master/model_doc/distilbert.html#transformers.DistilBertModel):
```python
from transformers import DistilBertTokenizer, DistilBertModel
import torch

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
model = DistilBertModel.from_pretrained('distilbert-base-uncased')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
```
But I get the warning about weights that are not used:
```
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertModel: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias', 'vocab_projector.bias', 'vocab_projector.weight', 'vocab_layer_norm.weight']
- This IS expected if you are initializing DistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```
Should I see this warning? I didn't expect to.

I also tried with the bare BERT model, also by copying from the [docs](https://huggingface.co/transformers/master/model_doc/bert.html#transformers.BertModel). But this too shows a warning about weights not being used:
```
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```
Should I really be seeing this? I assume the weights are matched with the correct model. If this is expected behavior maybe that can be noted in the docs?
In both case you are using a checkpoint pretrained on masked language modeling. The warning tells you some weights of the checkpoint are not used, which is the case: you are using the bare model without the masked LM classification head, so you are not using those weights.<|||||>Alright so the default checkpoints contain the masked LM head? Is there a checkpoint without it? 

Thanks <|||||>I'm not aware of any pretrained model without any head, so I think the answer is no.<|||||>Okay, thanks. I was just not expecting to see any warnings when following the examples.