Integrate Timm models as vision encoders in Vision encoder decoder models 
### Feature request

Hello, 
I would like to use timm models as vision encoders in vision encoder-decoder model who can I do so?

### Motivation

Timm models are very powerful and allow a lot of image processing which would allow us to make better document AI models

### Your contribution

I would be down to help out with PR
@NielsRogge <|||||>Hi,

I don't think that makes sense as for the Vision encoder-decoder framework, one needs a Transformer-based encoder or at least an encoder that outputs a sequence of hidden states, which can be used for cross-attention with the Transformer-based language decoder. Given that most backbones in timm are convolution-based, which output 3D feature maps, one would first need to flatten/project them into a sequence of hidden states.

The VisionEncoderDecoderModel [framework](https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder) currently supports ViT, DeiT, BEiT and Swin Transformer, which is already a fair amount of models.<|||||>I was more looking to use Swin with non-square image sizes and the Timm implementation allows us to do that and I would like to implement SwinV2 with bert<|||||>>  I was more looking to use Swin with non-square image sizes

Our implementation also allows that, did you try it?<|||||>I havent tried it can you share an example?
<|||||>```
from transformers import AutoModelForImageClassification
import torch

model = AutoModelForImageClassification.from_pretrained("microsoft/swinv2-tiny-patch4-window8-256")

pixel_values = torch.randn(1, 3, 244, 522)

outputs = model(pixel_values)
```