Description
Our team is working on building an image captioning model which can generate a movie/t.v show plot from it’s poster.
The goal of this project is to create an image captioning model using a transformer encoder model like Vision Transformer (ViT) and a transformer decoder language model like GPT-2

Model(s)
Any vision based encoder and language model decoder would be a good candidate to train the VisionEncoderDecoderModel for image captioning. We are trying the following models first:

Encoder: google/vit-base-patch16-224-in21k 1

Decoder: gpt2 1



Datasets
We are using publicly available IMDb datasets to train the model.
Some examples:

IMDb movies extensive dataset | Kaggle 1
48K IMDB Movies With Posters | Kaggle 1


Challenges
The main challenge is to create a good dataset of poster and movie plots. Also it will be interesting to see if the model gives good predictions for non-english movies/tv shows.

Desired project outcomes
We will create a Streamlit or Gradio app on  Spaces 3 that can predict a movie/t.v show plot from it’s poster.