[Bug?] Warning about weights not being initialized, even though not part of the model
Just opening this issue to keep track of it.

As reported on the [forum](https://discuss.huggingface.co/t/some-weights-of-bertmodel-were-not-initialized-from-the-model-checkpoint/3805/4), sometimes a warning gets printed about certain weights not being initialized while they are not part of the model. These always seem to be pooler weights.

## Example 1
`BertForMaskedLM` only has a language modeling head and no pooler, yet the following warning might get printed:
```
Some weights of BertModel were not initialized from the model checkpoint at ./output_model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Even though `BertForMaskedLM` uses a `BertModel` with `add_pooling_layer=False` as can be seen [here](https://github.com/huggingface/transformers/blob/d5b82bb70c2e8c4b184a6f2a7d1c91d7fd156956/src/transformers/models/bert/modeling_bert.py#L1292).

## Example 2
I've added the TrOCR models in #13874, and when converting these models, I'm using a `ViTModel` with `add_pooling_layer=False` and a `TrOCRForCausalLM`. However, when you do the following:

```
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
```
This prints the following warning:

```
Some weights of VisionEncoderDecoderModel were not initialized from the model checkpoint at microsoft/trocr-base-handwritten and are newly initialized: ['encoder.pooler.dense.weight', 'encoder.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

This warning shouldn't be printed, as the encoder shouldn't have a pooler.

Curious to know what's the cause of this!
I cannot reproduce this. Does it only occur for custom models? I get the expected behavior.

```
> from transformers import BertForMaskedLM
> model = BertForMaskedLM.from_pretrained("bert-base-cased")
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

For the second part (and this is a guess), are you perhaps indeed using `add_pooling_layer` for conversion (where you add the enc/dec manually), but not so when you are loading `VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")`? From what I remember, `add_pooling_layer` is True by default. So if you are just loading default configs, then the pooler is still enabled by default. So maybe this

https://github.com/NielsRogge/transformers/blob/f3d9e9483d8d6b915260880312cdd58518e68cf4/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py#L173-L174

has to be something like

```python
if encoder is None:
    encoder = AutoModel.from_config(config.encoder, add_pooling_layer=False)
```<|||||>Ok, so clearly not a bug! Thanks for investigating. 

Indeed, whether or not to add a pooling layer is not part of the config, hence it will add it as it is set to True by default. We could indeed add `add_pooling_layer=False`, as I don't think any encoder-decoder model needs the pooler of the encoder (but only the final hidden states). However, not every vision encoder will have an `add_pooling_layer` argument in its init (the ones that are currently added, namely ViT, BEiT and DEiT do, but I'm working on a new one that doesn't have it).

<|||||>It might be worth it to have all base models inits have a `**kwargs` to absorb any unrelated arguments. _But_ I am also not a big fan of that because it can lead to non-obvious errors or misunderstandings, especially for the user. Such as "Well, I passed add_pooling_layer=True, and I do not get an error, but still I do not get a pooling layer."

I am not sure what the best design choice is here.<|||||>> It might be worth it to have all base models inits have a **kwargs to absorb any unrelated arguments.

No that's definitely something we don't want. Otherwise it leads to all the models and the `AutoModel` API accepting any kwarg, so any typo will be forgotten silently:
```
AutoModelForSequenceClassiciation.from_pretrained("bert-base-cased", num_label=20)
```
will silently leave the labels at 2 for instance, producing bugs that are hard to debug.<|||||>Yes, that was exactly the "against" argument that I wrote after the _But_. :D 

Making it part of the configs seems sensible, as it pertains to the structure of the model. But perhaps there are arguments against that as well?<|||||>No the config is shared for all task-specific models and you can have the same base model with the same config used with different kinds of heads that require (or not) the pooling layer. That's why this parameter was not set in the config.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.