[roberta] lm_head.decoder save/load needs fixing
```
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("distilroberta-base")

print('decoder.weight' in dict(model.lm_head.named_parameters()).keys())
print('dense.weight' in dict(model.lm_head.named_parameters()).keys())

print('lm_head.decoder.weight' in dict(model.named_parameters()).keys())
print('lm_head.dense.weight' in dict(model.named_parameters()).keys())
```

gives:
```
True
True
False
True
```

So if we query `lm_head` we can see `lm_head.decoder.weight`, however it's not visible to the whole model via `parameters()` (named or not).

The problem comes from `tie_weights`:
```
       output_embeddings = self.get_output_embeddings()
        if output_embeddings is not None and self.config.tie_word_embeddings:
            self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
```

which essentially does:
```
lm_head.decoder.weight = embeddings.word_embeddings.weight
```

which disconnects it from the module list.

This is inconsistent behavior.

But the main issue is that `lm_head.decoder.weight` is saved in the `save_pretrained` and then is expected to be there on `torch.load` but since it's tied frameworks like deepspeed won't save it.

So if `config.tie_word_embeddings` is `True`, it shouldn't save that key and not expect to load it.

Please correct me if I have missed something obvious.

@LysandreJik, @sgugger 

Indeed, the key should be added to the keys to ignore on save, and expected missing keys.

Thanks for your investigation!<|||||>What if someone sets `config.tie_word_embeddings=False`. Should the save/expected keys be dynamically adjusted in `__init__`? 

or do it the other way around - have `lm_head.decoder.weight` on the list and instead remove it in `__init__` if  `config.tie_word_embeddings=False` as the former is the most likely behavior.<|||||>Yes, I think the second proposition makes a lot of sense! Great catch.

We'll need to check, but I wouldn't be surprised if other models have their lm head on the list - and nothing set in the `__init__` to prevent the save/expected keys from interacting with that layer. If so, it's not a high priority issue as no one has brought it up - but it is an issue nonetheless that potentially affects several models.