Loading BertModel from BertForMaskedLM without randomly initializing weights
I've trained a `BertForMaskedLM` model for a few days, and I've `save_pretrained()` it.

I want to compare encoder vectors from a `BertModel`, with the embeddings from
this trained `BertForMaskedLM`.

However, simply using `BertModel.from_pretrained("path/to/new_BertForMaskedLM_model")`
warns me that "You should probably TRAIN this model....". 

It seems that `bert.pooler.dense.weight` and `bert.pooler.dense.bias` have been added and randomly initialized after the LM head was removed. I assume this means the output of this current model will be useless.

What do I need to do to compare `BertForMaskedLM` encoder embeddings (after removing the LM head) with `BertModel` encoder embeddings? 
Hey! The model returns several outputs:

```
    last_hidden_state: torch.FloatTensor = None
    pooler_output: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
    past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None
    attentions: Optional[Tuple[torch.FloatTensor]] = None
    cross_attentions: Optional[Tuple[torch.FloatTensor]] = None
```

The `pooler_output` is going to be random as the pooler weights will be randomly generated, but the rest will not be! In order to compare the two, you could compare the `hidden_states` values, which contain all intermediary values from the word embeddings up until the last transformer layer's output.<|||||>I suppose more specifically I'm after the following output:

```
model = BertModel('path/to/new_BertForMaskedLM_model')
output = model(**encoded_input)[1]
```

I've got other baselines with embeddings extracted like this. 

Do you mind explaining the difference between `hidden_states` output and `model(**encoded_input)[1]` please?<|||||>`[1]` will take the second value of the output. It may differ across models, while specifying `output.hidden_states` will always return the hidden states.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.