ignore_mismatched_sizes do not work propoerly
- `transformers` version: 4.11.3
- Platform: Linux-4.19.117.bsk.5-amd64-x86_64-with-debian-10.6
- Python version: 3.7.3
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

I'm trying to load a pretrained pytorch Bert model with a different type_vocab_size with the following code:

```python
from transformers import AutoConfig, AutoModel

name = 'bert-base-uncased'
config = AutoConfig.from_pretrained(name)
config.type_vocab_size = 5
model = AutoModel.from_pretrained(name, config = config, ignore_mismatched_sizes = True)
```
and got Runtime Error:
```
Traceback (most recent call last):
  File "a.py", line 7, in <module>
    model = AutoModel.from_pretrained(name, config = config, ignore_mismatched_sizes = True)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/auto_factory.py", line 419, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1429, in from_pretrained
    _fast_init=_fast_init,
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1576, in _load_state_dict_into_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for BertModel:
        size mismatch for bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([5, 768]).
```

It seems ignore_mismatched_sizes is not worked properly. When debugging the codes, I found in class function `load_state_dict_info_model`, the `model_key` is not generated correctly. (e.g. it should be `embeddings.word_embeddings.weight` but `bert.bert.embeddings.word_embeddings.weight` instead). 

https://github.com/huggingface/transformers/blob/3fefa292c1c419f0c4c3e2697cdd94cafaeb4b66/src/transformers/modeling_utils.py#L1516
https://github.com/huggingface/transformers/blob/3fefa292c1c419f0c4c3e2697cdd94cafaeb4b66/src/transformers/modeling_utils.py#L1518

I tried to swap above two lines and the code works fine. Is it a bug, and it's the proper way to fix?

As seen on the PR linked above, this may fix the issue for a model with no task-specific head, but it will fail for others, so the fix is a  bit more convoluted, probably.

**Edit:** I dived into the code a bit more (that function is so confusing!) and it is indeed the right order.<|||||>Thank you for your reply, now I just generate the original model and replace `model.embeddings.token_type_embeddings` by a new `nn.Embedding`, and it seems worked in my case. I'm looking forward to having this bug fixed.<|||||>This is not worked for Blip2ForConditionalGeneration
I changed the size of the vocab in the configuration file from 50272 to 21128 and set ignore_mismatched_sizes=True, but raise a error as follow:
![image](https://user-images.githubusercontent.com/51687594/222946923-5f9cbe82-2551-4e2c-91c5-6f8f4400f852.png)@sgugger
I have no idea. please help!<|||||>It seems that there are two model parameter files, which are loaded in batches, whereas model_state_dict loads them all, so KeyError occurs. This is a bug. I hope we have time to revise it @sgugger<|||||>cc @younesbelkada <|||||>Will definitely look into it 
@lixinliu1995 can you provide a reproducible script? Thanks!<|||||>I meet the same problem.