title: What You See is What You Read? Improving Text-Image Alignment Evaluation
summary: Automatically determining whether a text and a corresponding image are
  semantically aligned is a significant challenge for vision-language models,
  with applications in generative text-to-image and image-to-text tasks. In this
  work, we study methods for automatic text-image alignment evaluation. We first
  introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets
  from both text-to-image and image-to-text generation tasks, with human
  judgements for whether a given text-image pair is semantically aligned. We then
  describe two automatic methods to determine alignment: the first involving a
  pipeline based on question generation and visual question answering models, and
  the second employing an end-to-end classification approach by finetuning
  multimodal pretrained models. Both methods surpass prior approaches in various
  text-image alignment tasks, with significant improvements in challenging cases
  that involve complex composition or unnatural images. Finally, we demonstrate
  how our approaches can localize specific misalignments between an image and a
  given text, and how they can be used to automatically re-rank candidates in
  text-to-image generation.
paper_authors._id: 64657e3a34acce85aea4a44a
paper_authors.name: Michal Yarom
paper_authors._id: 64657e3a34acce85aea4a44b
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-19T14:17:42.319Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 64657e3a34acce85aea4a44c
paper_authors.name: Soravit Changpinyo
paper_authors._id: 64657e3a34acce85aea4a44d
paper_authors.name: Roee Aharoni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-19T14:18:41.711Z
paper_authors._id: 6363bf2b123a5d5cd4a8fe7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667756718733-6363bf2b123a5d5cd4a8fe7c.jpeg
paper_authors.fullname: Roee Aharoni
paper_authors.type: user
paper_authors.user: roeeaharoni
paper_authors._id: 64657e3a34acce85aea4a44e
paper_authors.name: Jonathan Herzig
paper_authors._id: 64657e3a34acce85aea4a44f
paper_authors.name: Oran Lang
paper_authors._id: 64657e3a34acce85aea4a450
paper_authors.name: Eran Ofek
paper_authors._id: 64657e3a34acce85aea4a451
paper_authors.name: Idan Szpektor
paper_title: What You See is What You Read? Improving Text-Image Alignment Evaluation
paper_summary: Automatically determining whether a text and a corresponding image are
  semantically aligned is a significant challenge for vision-language models,
  with applications in generative text-to-image and image-to-text tasks. In this
  work, we study methods for automatic text-image alignment evaluation. We first
  introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets
  from both text-to-image and image-to-text generation tasks, with human
  judgements for whether a given text-image pair is semantically aligned. We then
  describe two automatic methods to determine alignment: the first involving a
  pipeline based on question generation and visual question answering models, and
  the second employing an end-to-end classification approach by finetuning
  multimodal pretrained models. Both methods surpass prior approaches in various
  text-image alignment tasks, with significant improvements in challenging cases
  that involve complex composition or unnatural images. Finally, we demonstrate
  how our approaches can localize specific misalignments between an image and a
  given text, and how they can be used to automatically re-rank candidates in
  text-to-image generation.
paper_ai_summary: SeeTRUE evaluates text-image alignment using question generation and visual question answering or end-to-end classification with fine-tuned multimodal models, surpassing prior methods in complex scenarios.
paper_ai_keywords: SeeTRUE question generation visual question answering end-to-end classification multimodal pretrained models text-image alignment localization of misalignments re-ranking candidates