CausalLMHead Models do not register head parameters.
## Environment info
- `transformers` version: 4.12.2
- Platform: Linux-5.4.0-87-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik

## Information

Model I am using (Bert, XLNet ...): BertLMHeadModel

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Instantiate a BertLMHeadModel. The model contains, among others, a linear layer under the path `cls.predictions.decoder `. Specifically, it contains 2 parameters: `cls.predictions.decoder.weight` and `cls.predictions.decoder.bias`.
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("bert-base-uncased", is_decoder=True)
print(model.cls.predictions.decoder)
```
2. Get a list of model parameters.
```python
parameter_keys = list(dict(model.named_parameters()).keys())
```
3. The decoder parameters are missing.
```python
print("cls.predictions.decoder.weight" in parameter_keys)
print("cls.predictions.decoder.bias" in parameter_keys)
```
4. Note that the parameters appear under the cls module:
```python
print(list(dict(model.cls.named_parameters()).keys()))
```
5. Note that the `cls.decoder` module appears to be registerred.
```python
print(list(dict(model.named_modules()).keys()))
``` 

## Expected behavior
Decoder parameters should be incldued in the model parameters.

This is because input- and output embeddings are tied (i.e. shared).

But this causes a lot of confusion, I know. Would be better indeed if these get included in the named parameters of a model. cc @LysandreJik <|||||>Thanks for the explanation. Makes sense. I am nut sure including it in the named parameters would help since the optimizer will complain of duplicate parameters in that case. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.