Passing in num_labels to ConvNextForImageClassification.from_pretrained raises size mismatch error
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@NielsRogge, @sgugger

## Information

Model I am using (Bert, XLNet ...):
`ConvNext`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Initating a `ConvNextForImageClassification.from_preainted` fails when passing in `num_labels`.  The same steps work sucesfully for `ViTForImageClassification`
2. When passing in a custom number of labels a size mismatch error is raised. 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python 
# define labels for import image classification problem 
id2label = {0:"doggie", 1: "cat"}
label2id = {v:k for k,v in id2label.items()}

# This works fine
from transformers import ViTForImageClassification, ViTFeatureExtractor
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224-in21k',
                                                  num_labels=len(id2label),
                                                  id2label=id2label,
                                                  label2id=label2id)
# this failes 
model = ConvNextForImageClassification.from_pretrained("facebook/convnext-tiny-224",
                                                        num_labels=len(id2label),
                                                        id2label=id2label,
                                                        label2id=label2id)

```
This raises a `RunTimeError`:

```python
---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-22-9991914f5786> in <module>()
      2                                                         num_labels=len(id2label),
      3                                                         id2label=id2label,
----> 4                                                         label2id=label2id)
      5 

1 frames

/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1535                     pretrained_model_name_or_path,
   1536                     ignore_mismatched_sizes=ignore_mismatched_sizes,
-> 1537                     _fast_init=_fast_init,
   1538                 )
   1539 

/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in _load_state_dict_into_model(cls, model, state_dict, pretrained_model_name_or_path, ignore_mismatched_sizes, _fast_init)
   1688         if len(error_msgs) > 0:
   1689             error_msg = "\n\t".join(error_msgs)
-> 1690             raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
   1691 
   1692         if len(unexpected_keys) > 0:

RuntimeError: Error(s) in loading state_dict for ConvNextForImageClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([1000, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([2]).
```

[Colab notebook](https://colab.research.google.com/drive/1jIX49dOIPKOO7z8XH86DnbZhJn89KDRA?usp=sharing) reproducing the above error. 

## Expected behavior

I expected `**Model**ForImageClassification` to behave fairly consistently across model types i.e. I can swap out `VIT` for `ConvNext` and the behaviour will be similar. I haven't dug into the source of this carefully but if it's not just me misunderstanding the docs I'm happy to try and help track down this source of this. 

As a side note I'm very happy to see the growth of vision models in transformers! 
This is because the checkpoint you are using has 1,000 labels. To ignore the pretrained weights of the classifier, you have to pass along `ignore_mismatched_sizes=True`.

You don't need this for other models if the pretrained model does not have a classifier head.<|||||>> This is because the checkpoint you are using has 1,000 labels. To ignore the pretrained weights of the classifier, you have to pass along `ignore_mismatched_sizes=True`.
> 
> You don't need this for other models if the pretrained model does not have a classifier head.

Thanks for clarifying @sgugger 🤗 Will close this now since it's not a bug (sorry should have checked this one on the forums) 