ConvNextV2 weight not initialized
### System Info

kaggle NoteBook:

- `transformers` version: 4.27.4
- Platform: Linux-5.15.90+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 1.13.0+cpu (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.6.4 (cpu)
- Jax version: 0.3.25
- JaxLib version: 0.3.25
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger @alaradirik @amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoImageProcessor, ConvNextV2ForImageClassification
import torch
from datasets import load_dataset

dataset = load_dataset("huggingface/cats-image")
image = dataset["test"]["image"][0]

image_processor = AutoImageProcessor.from_pretrained("facebook/convnextv2-atto-1k-224")
model = ConvNextV2ForImageClassification.from_pretrained("facebook/convnextv2-atto-1k-224")

inputs = image_processor(image, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# model predicts one of the 1000 ImageNet classes
predicted_label = logits.argmax(-1).item()
print(model.config.id2label[predicted_label])
```

output:

```
Downloading builder script: 100%
2.56k/2.56k [00:00<00:00, 120kB/s]
Downloading and preparing dataset cats_image/image to /root/.cache/huggingface/datasets/huggingface___cats_image/image/1.9.0/68fbc793fb10cd165e490867f5d61fa366086ea40c73e549a020103dcb4f597e...
Downloading data files: 100%
1/1 [00:00<00:00, 2.30it/s]
Downloading data: 100%
173k/173k [00:00<00:00, 637kB/s]
Extracting data files: 100%
1/1 [00:00<00:00, 59.25it/s]
Dataset cats_image downloaded and prepared to /root/.cache/huggingface/datasets/huggingface___cats_image/image/1.9.0/68fbc793fb10cd165e490867f5d61fa366086ea40c73e549a020103dcb4f597e. Subsequent calls will reuse this data.
100%
1/1 [00:00<00:00, 56.19it/s]
Downloading (…)rocessor_config.json: 100%
352/352 [00:00<00:00, 12.5kB/s]
Downloading (…)lve/main/config.json: 100%
69.7k/69.7k [00:00<00:00, 2.71MB/s]
Downloading pytorch_model.bin: 100%
14.9M/14.9M [00:00<00:00, 70.7MB/s]
Some weights of the model checkpoint at facebook/convnextv2-atto-1k-224 were not used when initializing ConvNextV2ForImageClassification: ['convnextv2.encoder.stages.2.layers.3.grn.weight', 'convnextv2.encoder.stages.1.layers.0.grn.weight', 'convnextv2.encoder.stages.2.layers.4.grn.weight', 'convnextv2.encoder.stages.0.layers.1.grn.bias', 'convnextv2.encoder.stages.2.layers.5.grn.bias', 'convnextv2.encoder.stages.0.layers.1.grn.weight', 'convnextv2.encoder.stages.3.layers.1.grn.weight', 'convnextv2.encoder.stages.1.layers.1.grn.weight', 'convnextv2.encoder.stages.0.layers.0.grn.weight', 'convnextv2.encoder.stages.2.layers.0.grn.bias', 'convnextv2.encoder.stages.2.layers.2.grn.bias', 'convnextv2.encoder.stages.1.layers.0.grn.bias', 'convnextv2.encoder.stages.3.layers.0.grn.weight', 'convnextv2.encoder.stages.3.layers.1.grn.bias', 'convnextv2.encoder.stages.1.layers.1.grn.bias', 'convnextv2.encoder.stages.2.layers.0.grn.weight', 'convnextv2.encoder.stages.2.layers.1.grn.weight', 'convnextv2.encoder.stages.2.layers.4.grn.bias', 'convnextv2.encoder.stages.2.layers.1.grn.bias', 'convnextv2.encoder.stages.2.layers.3.grn.bias', 'convnextv2.encoder.stages.2.layers.5.grn.weight', 'convnextv2.encoder.stages.2.layers.2.grn.weight', 'convnextv2.encoder.stages.0.layers.0.grn.bias', 'convnextv2.encoder.stages.3.layers.0.grn.bias']
- This IS expected if you are initializing ConvNextV2ForImageClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ConvNextV2ForImageClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ConvNextV2ForImageClassification were not initialized from the model checkpoint at facebook/convnextv2-atto-1k-224 and are newly initialized: ['convnextv2.encoder.stages.2.layers.5.grn.beta', 'convnextv2.encoder.stages.1.layers.0.grn.beta', 'convnextv2.encoder.stages.1.layers.1.grn.gamma', 'convnextv2.encoder.stages.0.layers.0.grn.beta', 'convnextv2.encoder.stages.2.layers.0.grn.gamma', 'convnextv2.encoder.stages.0.layers.1.grn.gamma', 'convnextv2.encoder.stages.3.layers.1.grn.beta', 'convnextv2.encoder.stages.2.layers.4.grn.gamma', 'convnextv2.encoder.stages.1.layers.1.grn.beta', 'convnextv2.encoder.stages.3.layers.0.grn.beta', 'convnextv2.encoder.stages.2.layers.0.grn.beta', 'convnextv2.encoder.stages.3.layers.1.grn.gamma', 'convnextv2.encoder.stages.2.layers.5.grn.gamma', 'convnextv2.encoder.stages.2.layers.3.grn.gamma', 'convnextv2.encoder.stages.2.layers.2.grn.beta', 'convnextv2.encoder.stages.2.layers.4.grn.beta', 'convnextv2.encoder.stages.2.layers.1.grn.gamma', 'convnextv2.encoder.stages.0.layers.1.grn.beta', 'convnextv2.encoder.stages.2.layers.2.grn.gamma', 'convnextv2.encoder.stages.3.layers.0.grn.gamma', 'convnextv2.encoder.stages.2.layers.1.grn.beta', 'convnextv2.encoder.stages.0.layers.0.grn.gamma', 'convnextv2.encoder.stages.2.layers.3.grn.beta', 'convnextv2.encoder.stages.1.layers.0.grn.gamma']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
racket, racquet
```

### Expected behavior

No warning should be there about initialization of weights
Hi @IMvision12, thanks for opening the issue! This doesn't effect the output logits significantly but we pinpointed the issue and will fix it shortly.