named_parameters not showing embedding matrix of RobertaLMHead (more a question than a bug)
## Environment info
- `transformers` version: 4.2.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no 

### Who can help
@LysandreJik @patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): RobertaForCausalLM

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load standard RobertaForCausalLM
2. Record the difference between .named_parameters() and .state_dict() of the model
```
from transformers import RobertaForCausalLM, RobertaConfig
import torch

config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaForCausalLM.from_pretrained('roberta-base', config=config)

named_params = [n for n, _ in model.named_parameters()]
print("Difference: ", [n for n in list(model.state_dict().keys()) if n not in named_params])
```

This outputs:
```
Difference:  ['roberta.embeddings.position_ids', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
```

## Expected behavior

I would expect the parameters: `'lm_head.decoder.weight', 'lm_head.decoder.bias'` to show up in `.named_parameters()`, why do they not? That `'roberta.embeddings.position_ids'` does not show up in `.named_parameters()` is expected as they are not learned parameters, but just of help with getting the position embedding, but this is always the same I believe.

I would like to tie my input embedding matrix weights to my output embedding matrix. But now I am not really sure how to go about this. I thought of just doing:

`model.lm_head.decoder.weight = model.roberta.embeddings.word_embeddings.weight` but because of this thing with the named_parameters, I am not sure if this will work as expected. Also, the output embedding has a bias `lm_head.decoder.bias`, while the the input embeddings don't. My goal is to have the initial hidden state space be the same space as the last hidden state space.

Thanks in advance, 

Claartje

I found out that the input embedding and output embedding weights are tied by default, which is why the output embedding weights end up in state_dict, but not in the named_parameters (they are overcomplete).<|||||>I've found this issue too when trying to log parameters/gradients to Weights & Biases. It doesn't log `roberta.lm_head.decoder`.

I can't quite work out the logic of [this code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/roberta/modeling_roberta.py#L1143-L1145) in `RobertaLMHead`:
```py
        self.decoder = nn.Linear(config.hidden_size, config.vocab_size)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        self.decoder.bias = self.bias
```

If I understand what @ClaartjeBarkhof is saying, this is the reason for `decoder` not showing up in `named_parameters()`. Is that right? I can't quite make the connection between tying weights and then the thing no longer being considered a named parameter. And follow up question, what does that code actually do? How does it differ from just having the first line?
