bug for from_pretrained method with  ignore_mismatched_sizes=True
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-1040-azure-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @sgugg

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When setting `ignore_mismatched_sizes=True` in `from_pretrained` method, it would give errors.
```python
from transformers import AutoModelForCausalLM
model_type = "facebook/opt-6.7b"
model = AutoModelForCausalLM.from_pretrained(model_type,max_position_embeddings=4096,
                                            ignore_mismatched_sizes=True)
```

The tracebacks are:
<img width="987" alt="image" src="https://github.com/huggingface/transformers/assets/38466901/ce67a66b-ed9a-42fe-871d-eb1ffe348da0">

However, this wouldn't happen in smaller model. Simply changing `model_type` to `facebook/opt-2.7b` is fine. And this bug is not specific for `OptModel`, `EleutherAI/pythia-6.9b` would also trigger error.

The trackback for `EleutherAI/pythia-6.9b` is:
<img width="882" alt="image" src="https://github.com/huggingface/transformers/assets/38466901/7b3b3bea-0a03-4e5c-aef1-b386bff11be3">


### Expected behavior

loading weights successfully without error.
Hey 🤗 ! Thanks for reporting, this indeed a bug! I tracked this down to #24505  (this [commit](https://github.com/huggingface/transformers/commit/8e5d1619b3e57367701d74647e87b95f8dba5409)). Will probably let @sgugger handle, I don't have bandwidth to solve it right now! <|||||>No, this has no link to #24505, the code sample also fails on v4.30.2 and looking at the fix, it never worked before. The PR linked above should fix it.<|||||>I tested with `pip install -q transformers==4.30.2` and it worked fine ( no error but maybe no resize?), same for previous versions. When checking out commit after comit, this was the failing one but I think I missed something! 