Deit
# 🚀 Feature request

The Deit model from torchhub is scriptable, however the pertained version in HF repo, "facebook/deit-base-distilled-patch16-224" is the distilled one which is not scriptable. 

This can be tried out using the code below 
```
import torch
model = torch.hub.load('facebookresearch/deit:main', 'deit_base_patch16_224', pretrained=True)
ts_model = torch.jit.script(model)
```
vs 

```
feature_extractor = DeiTFeatureExtractor.from_pretrained('facebook/deit-base-distilled-patch16-224')
model = DeiTForImageClassificationWithTeacher.from_pretrained('facebook/deit-base-distilled-patch16-224')
ts = torch.jit.script(model)
```
## Motivation
Torchscripted models can boost the inference speed and it would benefit the community having vision models that are scriptable.



Are you interested in working on this?

Basically, to make DeiT scriptable, you need to set `test_torchscript` of the model test as seen [here](https://github.com/huggingface/transformers/blob/0d1f67e651220bffef1441fa7589620e426ba958/tests/test_modeling_deit.py#L160) to `True`, then run the tests (running `pytest tests/test_modeling_deit.py` from the root of the repo), and then fix the modeling file (`modeling_deit.py`) to make the failing tests pass.

We could then do the same for `modeling_vit.py`. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.