We’re planning to use ViT encoder, mBART decoder and train them end-to-end for image captioning in different languages.

Model
Pre-trained ViT 2, mBART 3 (will be merged soon) can be leveraged for our task.

Datasets
Currently we’re thinking of using WIT 4. COCO dataset can also be used but it has caption only in English language. Dataset can be changed later if we come across more suitable dataset for this task.

Available training scripts
Since this is a Seq2Seq model, the run_summarization_flax.py 7 script can be used for training this model with some modifications.

(Optional) Desired project outcome
Our end use case would be to run this model on a video clip or movie and make it like an accessibility tool for visually challenged people. This can be showcased with a streamlit or gradio app.
We’re also planning to benchmark its zero-shot performance on VizWiz 4

(Optional) Challenges
-Training encoder-decoder model end-to-end in FLAX/JAX will require some effort
-Data Processing incase new datasets are available only in English