T5ForConditionalGeneration checkpoint size mismatch
### System Info

## Error Description
I trained a `T5ForConditionalGeneration` model and saved the checkpoint using PyTorch Lightning's Trainer to a `.ckpt` file. But when I try to load back the state_dict using `model.from_state_dict()`, I get this error:
```python
RuntimeError: Error(s) in loading state_dict for T5ForConditionalGeneration:
        Unexpected key(s) in state_dict: "decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight". 
        size mismatch for shared.weight: copying a param with shape torch.Size([32103, 512]) from checkpoint, the shape in current model is torch.Size([32128, 512]).
        size mismatch for encoder.embed_tokens.weight: copying a param with shape torch.Size([32103, 512]) from checkpoint, the shape in current model is torch.Size([32128, 512]).
        size mismatch for decoder.embed_tokens.weight: copying a param with shape torch.Size([32103, 512]) from checkpoint, the shape in current model is torch.Size([32128, 512]).
        size mismatch for lm_head.weight: copying a param with shape torch.Size([32103, 512]) from checkpoint, the shape in current model is torch.Size([32128, 512]).
```

I have not changed the model definition in any way. The keys also match. So, I'm really not sure how the sizes could mismatch magically when loading?

## Loading the model
This is how I'm loading the model:
```python
tokenizer = T5Tokenizer.from_pretrained(args["model_checkpoint"], bos_token="[bos]", eos_token="[eos]", sep_token="[sep]")
    model = T5ForConditionalGeneration.from_pretrained(args["model_checkpoint"], ignore_mismatched_sizes=True)
    model.load_state_dict({k[6:]: v for k, v in ckpt["state_dict"].items()})
```

I even tried to pass `ignore_mismatched_sizes=True` to the `from_pretrained` call, and that didn't help either.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

As described above.

### Expected behavior

No error
Hey @msamogh 👋 

To explain why there is a mismatch, we would need to know exactly how the model was trained :) However, the most important part -- you may be able to load the checkpoint with these two strategies:
1. Load the model architecture from the same configuration as your trained model
2. After initializing the model architecture (and before loading the checkpoint), [resize the embeddings](https://huggingface.co/docs/transformers/v4.22.2/en/main_classes/model#transformers.PreTrainedModel.resize_token_embeddings)

Both strategies should change the shape of your architecture to match your checkpoint<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @gante, does interpolate_pos_embedding type functions do what you mentioned in point 2?

Here is a snippet:

```
def interpolate_pos_embed_multimae(model, checkpoint_model):
    pattern = "input_adapters\.(.*)\.pos_emb"
    matched_keys = [k for k in checkpoint_model if bool(re.match(pattern, k))]

    for key in matched_keys:
        domain = re.match(pattern, key).group(1)  # group(0) is entire matched regex
        if getattr(model.input_adapters, domain, None) is not None:
            pos_embed_checkpoint = checkpoint_model[key]
            _, _, orig_H, orig_W = pos_embed_checkpoint.shape
            _, _, new_H, new_W = getattr(model.input_adapters, domain).pos_emb.shape
            if (orig_H != new_H) or (orig_W != new_W):
                print(f"Key {key}: Position interpolate from {orig_H}x{orig_W} to {new_H}x{new_W}")
                pos_embed_checkpoint = torch.nn.functional.interpolate(
                    pos_embed_checkpoint, size=(new_H, new_W), mode='bicubic', align_corners=False)
                checkpoint_model[key] = pos_embed_checkpoint

```

<|||||>Hey @forkbabu 👋 I do not know the answer to your question. However, from your code snippet, it seems like you are working with a vision model -- my recommendation would be to open a new issue and tag one of our vision experts 