Weights of lm_head and input embedding are not tied in google/mt5-base
## Environment info

- `transformers` version: 4.16.2
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>

### Who can help

@patrickvonplaten 

## Information

I am tring to prune the embedding matrix of [google/mt5-base](https://huggingface.co/google/mt5-base), which can be tracked in [this thread](https://discuss.huggingface.co/t/pruning-a-model-embedding-matrix-for-memory-efficiency/5502/7?u=bookworm).  
During the process, I find that the weights of lm_head and the input embedding are not tied.

So I wonder:

1. Is this the expected behaviour for mT5?
2. How can I prune the embedding matrix correctly if its weights are shared with lm_head?

## To reproduce

```python
import torch
from transformers import MT5ForConditionalGeneration

model = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")
lm_weight = model.lm_head.weight  # shape: torch.Size([250112, 768])
em_weight = model.get_input_embeddings().state_dict()["weight"]  # shape: torch.Size([250112, 768])

# Expect: less than 1e-3
# Actual: 113.6006
print(torch.max(lm_weight - em_weight))
```

Hey @z-bookworm,

Yes this should be correct. As MT5 is built upon T5v1.1: https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511 the input and output embeddings should not be tied as you can read upon in the link<|||||>@patrickvonplaten Thanks for the reference