Hi there,
Regrading the dataset:
Yes, the dataset is not available in datasets lib. You will need to download the tsv and the prepare the dataset by downloading the image. The tsv file for wit contains the image URLs and other metadata.
This script might help 8. It’s for downloading conceptual captions data, but you could re-purpose it to download WIT.
Regarding model:
There is no off-the-shelf model for this in transformers (yet!). What we need here is a Seq2Seq model. Where that image encoder is a pre-trained vision model like ViT or CLIP's vision model and the decoder is any pre-trained text model (BERT/ROBERTa).
To do this we will need to modify the BERT/ROBERTa model and add a cross-attention layer in it.
The encoder will the encode images, the decoder will take the decoder_input_ids and the encoder_hidden_states.
This should be similar to encoder-decoder model in pytorch transformers/modeling_encoder_decoder.py at master · huggingface/transformers · GitHub 60.
@gchhablani  is also working on a similar project and might have the modeling code ready 
Let me know if you have any more question.