Support ViT model in EncoderDecoder
The idea is to use ViT as the encoder and then a LM as the decoder for Image captioning generation, *e.g.*? @patil-suraj and I were also thinking of using `EncoderDecoder` for Speech2Text. 

At the moment, I see two solutions: -> adapt `EncoderDecoder` to be usable for all modalities not just text2text **or** create new classes, *e.g.* a `SpeechEncoderDecoder` and a `VisionEncoderDecoder` since I'm not sure `EncoderDecoder` will be able to handle all the new use-cases. *E.g.* we might end up ending way too many if-else statements that would make the code unreadable... @patil-suraj @abhi1thakur what do you think? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>Huhu,
any update at this ? :) 
a VisionEncoderDecoderModel would be a great also for models which follow in future
for example this one: [TrOCR](https://arxiv.org/abs/2109.10282)<|||||>We'll add a new `VisionEncoderDecoder` class for this actually<|||||>@patrickvonplaten nice !
Is there any way or need to contribute ? :)<|||||>Closing this as [VisionEncoderDecoder](https://huggingface.co/docs/transformers/main/en/model_doc/vision-encoder-decoder#vision-encoder-decoder-models) now exists.