What You See is What You Read? Improving Text-Image Alignment Evaluation
Automatically determining whether a text and a corresponding image are
semantically aligned is a significant challenge for vision-language models,
with applications in generative text-to-image and image-to-text tasks. In this
work, we study methods for automatic text-image alignment evaluation. We first
introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets
from both text-to-image and image-to-text generation tasks, with human
judgements for whether a given text-image pair is semantically aligned. We then
describe two automatic methods to determine alignment: the first involving a
pipeline based on question generation and visual question answering models, and
the second employing an end-to-end classification approach by finetuning
multimodal pretrained models. Both methods surpass prior approaches in various
text-image alignment tasks, with significant improvements in challenging cases
that involve complex composition or unnatural images. Finally, we demonstrate
how our approaches can localize specific misalignments between an image and a
given text, and how they can be used to automatically re-rank candidates in
text-to-image generation.
64657e3a34acce85aea4a44a
Michal Yarom
64657e3a34acce85aea4a44b
Yonatan Bitton
admin_assigned
2023-05-19T14:17:42.319Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
64657e3a34acce85aea4a44c
Soravit Changpinyo
64657e3a34acce85aea4a44d
Roee Aharoni
admin_assigned
2023-05-19T14:18:41.711Z
6363bf2b123a5d5cd4a8fe7c
https://cdn-avatars.huggingface.co/v1/production/uploads/1667756718733-6363bf2b123a5d5cd4a8fe7c.jpeg
Roee Aharoni
user
roeeaharoni
64657e3a34acce85aea4a44e
Jonathan Herzig
64657e3a34acce85aea4a44f
Oran Lang
64657e3a34acce85aea4a450
Eran Ofek
64657e3a34acce85aea4a451
Idan Szpektor
What You See is What You Read? Improving Text-Image Alignment Evaluation
Automatically determining whether a text and a corresponding image are
semantically aligned is a significant challenge for vision-language models,
with applications in generative text-to-image and image-to-text tasks. In this
work, we study methods for automatic text-image alignment evaluation. We first
introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets
from both text-to-image and image-to-text generation tasks, with human
judgements for whether a given text-image pair is semantically aligned. We then
describe two automatic methods to determine alignment: the first involving a
pipeline based on question generation and visual question answering models, and
the second employing an end-to-end classification approach by finetuning
multimodal pretrained models. Both methods surpass prior approaches in various
text-image alignment tasks, with significant improvements in challenging cases
that involve complex composition or unnatural images. Finally, we demonstrate
how our approaches can localize specific misalignments between an image and a
given text, and how they can be used to automatically re-rank candidates in
text-to-image generation.
SeeTRUE evaluates text-image alignment using question generation and visual question answering or end-to-end classification with fine-tuned multimodal models, surpassing prior methods in complex scenarios.
SeeTRUE question generation visual question answering end-to-end classification multimodal pretrained models text-image alignment localization of misalignments re-ranking candidates