Add OFA to transformers
# 🌟 New model addition
We recently proposed OFA, a unified model for multimodal pretraining, which achieves multiple SoTAs on downstream tasks, including image captioning, text-to-image generation, referring expression comprehension, etc. We would like to implement OFA on transformers if it is possible.

## Model description
OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework. For more information, please refer to our paper: [Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](http://arxiv.org/abs/2202.03052).

<!-- Important information -->
Our code base is Fairseq. We wonder if there is any simpler solution for the transfer from Fairseq to transformers. To email us, please contact zheluo.wp@alibaba-inc.com or junyang.ljy@alibaba-inc.com.

## Open source status

* [x] the model implementation is available: ([OFA official repo](https://github.com/OFA-Sys/OFA))
* [x] the model weights are available: ([OFA checkpoints](https://github.com/OFA-Sys/OFA/blob/main/checkpoints.md))
* [x] who are the authors: (Peng Wang @[logicwong](https://github.com/logicwong), An Yang @[yangapku](https://github.com/yangapku), Rui Men @[jxst539246](https://github.com/jxst539246), Junyang Lin @[JustinLin610](https://github.com/JustinLin610), Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yangf)

cc @gchhablani you might find this interesting :-) <|||||>@patil-suraj Yes! Definitely would love to take this up :)<|||||>Awesome! Happy to help you with this! Let me know if you have questions :) <|||||>> Not sure if this is was a coordinated collaborative effort, but the OFA-Sys team seems to have made a preliminary release of a transformers implementation on a [separate OFA branch](https://github.com/OFA-Sys/OFA/tree/feature/add_transformers/transformers/src/transformers/models/ofa).

Any news on this ? If the implementation exists a new PR could be opened without too much work.

Also, I have question on the OFA-large huggingface [repo](https://huggingface.co/OFA-Sys/ofa-large/tree/main) the model is 2.45 GB but on the checkpoint in the OFA [repo](https://github.com/OFA-Sys/OFA/blob/feature/add_transformers/checkpoints.md#pretraining) it downloads a 5.8 GB file. Why such a difference between the two ?