DEIT -Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTMode
### System Info

```shell
Nvidia 3080
Windows 11
```


### Who can help?

@NielsRogge
The warning I get is this and a big list of layers
You are using a model of type vit to instantiate a model of type deit. This is not supported for all configurations of models and can yield errors.
Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTModel: 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I did the following
from transformers import DeiTFeatureExtractor, DeiTModel
feature_extractor = DeiTFeatureExtractor.from_pretrained("facebook/deit-base-patch16-224")
model = DeiTModel.from_pretrained("facebook/deit-base-patch16-224")
inputs_ref = feature_extractor(images=im_ref, return_tensors="pt")
with torch.no_grad():
        outputs_ref = model(**inputs_ref)
last_hidden_states_ref = outputs_ref.last_hidden_state


The warning I get is this and a big list of layers
You are using a model of type vit to instantiate a model of type deit. This is not supported for all configurations of models and can yield errors.
Some weights of the model checkpoint at facebook/deit-base-patch16-224 were not used when initializing DeiTModel: 

The images are simple scenes only.

### Expected behavior

```shell
Is this warning something I should take seriously. I need only weights from a pre trained model to use.
The warning is 'You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference'

I am new to Hugging face community and I appreciate any help. I tried to follow the guidelines, please revert to me in case of more information
```

Hi,

That's because the checkpoint you are loading ([facebook/deit-base-patch16-224](https://huggingface.co/facebook/deit-base-patch16-224)) needs to be loaded in a `ViTModel`/`ViTForImageClassification` rather than a `DeiTModel`. As explained in the [docs](https://huggingface.co/docs/transformers/model_doc/deit) of DeiT, the authors also trained more efficient ViT models:

> The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [ViTModel](https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/vit#transformers.ViTModel) or [ViTForImageClassification](https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/vit#transformers.ViTForImageClassification). Techniques like data augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes): facebook/deit-tiny-patch16-224, facebook/deit-small-patch16-224, facebook/deit-base-patch16-224 and facebook/deit-base-patch16-384. Note that one should use [DeiTFeatureExtractor](https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/deit#transformers.DeiTFeatureExtractor) in order to prepare images for the model.<|||||>@NielsRogge 

Thank you very much for responding and I have got my mistake but still I have a confusion. Because Initially I started with 'facebook/deit-base-distilled-patch16-224' from the tutorial mentioned
https://huggingface.co/docs/transformers/v4.19.2/en/model_doc/deit#transformers.DeiTFeatureExtractor

Still I get this issue.  I understand I am not doing any classification so I get some warning but still I am not able to comprehend this. Any direction will be appreciated. Thanks in advance


Some weights of the model checkpoint at facebook/deit-base-distilled-patch16-224 were not used when initializing DeiTModel: ['distillation_classifier.bias', 'cls_classifier.weight', 'distillation_classifier.weight', 'cls_classifier.bias']
- This IS expected if you are initializing DeiTModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DeiTModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DeiTModel were not initialized from the model checkpoint at facebook/deit-base-distilled-patch16-224 and are newly initialized: ['deit.pooler.dense.weight', 'deit.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi,

When initializing a `DeiTModel`, it won't include the heads on top. For that, you'll need to instantiate a `DeiTForImageClassification` or `DeiTForImageClassificationWithTeacher` model.