title: Learning the Visualness of Text Using Large Vision-Language Models
summary: Visual text evokes an image in a person's mind, while non-visual text fails
  to do so. A method to automatically detect visualness in text will unlock the
  ability to augment text with relevant images, as neural text-to-image
  generation and retrieval models operate on the implicit assumption that the
  input text is visual in nature. We curate a dataset of 3,620 English sentences
  and their visualness scores provided by multiple human annotators.
  Additionally, we use documents that contain text and visual assets to create a
  distantly supervised corpus of document text and associated images. We also
  propose a fine-tuning strategy that adapts large vision-language models like
  CLIP that assume a one-to-one correspondence between text and image to the task
  of scoring text visualness from text input alone. Our strategy involves
  modifying the model's contrastive learning objective to map text identified as
  non-visual to a common NULL image while matching visual text to their
  corresponding images in the document. We evaluate the proposed approach on its
  ability to (i) classify visual and non-visual text accurately, and (ii) attend
  over words that are identified as visual in psycholinguistic studies. Empirical
  evaluation indicates that our approach performs better than several heuristics
  and baseline models for the proposed task. Furthermore, to highlight the
  importance of modeling the visualness of text, we conduct qualitative analyses
  of text-to-image generation systems like DALL-E.
paper_authors._id: 6466fa2aac657f60661ddee6
paper_authors.name: Gaurav Verma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T14:33:46.938Z
paper_authors._id: 6359599c6805fab09b0912dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6359599c6805fab09b0912dc/645rCR99A6CyIXSFy-w6_.jpeg
paper_authors.fullname: Gaurav Verma
paper_authors.type: user
paper_authors.user: gauravverma
paper_authors._id: 6466fa2aac657f60661ddee7
paper_authors.name: Ryan A. Rossi
paper_authors._id: 6466fa2aac657f60661ddee8
paper_authors.name: Christopher Tensmeyer
paper_authors._id: 6466fa2aac657f60661ddee9
paper_authors.name: Jiuxiang Gu
paper_authors._id: 6466fa2aac657f60661ddeea
paper_authors.name: Ani Nenkova
paper_title: Learning the Visualness of Text Using Large Vision-Language Models
paper_summary: Visual text evokes an image in a person's mind, while non-visual text fails
  to do so. A method to automatically detect visualness in text will unlock the
  ability to augment text with relevant images, as neural text-to-image
  generation and retrieval models operate on the implicit assumption that the
  input text is visual in nature. We curate a dataset of 3,620 English sentences
  and their visualness scores provided by multiple human annotators.
  Additionally, we use documents that contain text and visual assets to create a
  distantly supervised corpus of document text and associated images. We also
  propose a fine-tuning strategy that adapts large vision-language models like
  CLIP that assume a one-to-one correspondence between text and image to the task
  of scoring text visualness from text input alone. Our strategy involves
  modifying the model's contrastive learning objective to map text identified as
  non-visual to a common NULL image while matching visual text to their
  corresponding images in the document. We evaluate the proposed approach on its
  ability to (i) classify visual and non-visual text accurately, and (ii) attend
  over words that are identified as visual in psycholinguistic studies. Empirical
  evaluation indicates that our approach performs better than several heuristics
  and baseline models for the proposed task. Furthermore, to highlight the
  importance of modeling the visualness of text, we conduct qualitative analyses
  of text-to-image generation systems like DALL-E.
paper_ai_summary: A method to detect visualness in text improves neural text-to-image generation by fine-tuning large vision-language models to classify visual and non-visual text accurately.
paper_ai_keywords: visualness neural text-to-image generation retrieval models CLIP contrastive learning objective NULL image psycholinguistic studies DALL-E