title: VisualGPTScore: Visio-Linguistic Reasoning with Multimodal Generative
    Pre-Training Scores
summary: Vision-language models (VLMs) discriminatively pre-trained with contrastive
  image-text matching losses such as P(match|text, image)
  have been criticized for lacking compositional understanding. This means they
  might output similar scores even if the original caption is rearranged into a
  different semantic statement. To address this, we propose to use the {bf
  V}isual {bf G}enerative {bf P}re-{bf T}raining Score ({bf
  VisualGPTScore}) of P(text|image), a multimodal
  generative score that captures the likelihood of a text caption conditioned
  on an image using an image-conditioned language model. Contrary to the belief
  that VLMs are mere bag-of-words models, our off-the-shelf VisualGPTScore
  demonstrates top-tier performance on recently proposed image-text retrieval
  benchmarks like ARO and Crepe that assess compositional reasoning. Furthermore,
  we factorize VisualGPTScore into a product of the marginal P(text)
  and the Pointwise Mutual Information (PMI). This helps to (a)
  diagnose datasets with strong language bias, and (b) debias results on other
  benchmarks like Winoground using an information-theoretic framework.
  VisualGPTScore provides valuable insights and serves as a strong baseline for
  future evaluation of visio-linguistic compositionality.
paper_authors._id: 647ea243770c299e56fcebf7
paper_authors.name: Zhiqiu Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:27:29.827Z
paper_authors._id: 64c170190bfb901b04399295
paper_authors.avatarUrl: /avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
paper_authors.fullname: Zhiqiu Lin
paper_authors.type: user
paper_authors.user: zhiqiulin
paper_authors._id: 647ea243770c299e56fcebf8
paper_authors.name: Xinyue Chen
paper_authors._id: 647ea243770c299e56fcebf9
paper_authors.name: Deepak Pathak
paper_authors._id: 647ea243770c299e56fcebfa
paper_authors.name: Pengchuan Zhang
paper_authors._id: 647ea243770c299e56fcebfb
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-06T09:00:23.979Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_title: VisualGPTScore: Visio-Linguistic Reasoning with Multimodal Generative
    Pre-Training Scores
paper_summary: Vision-language models (VLMs) discriminatively pre-trained with contrastive
  image-text matching losses such as P(match|text, image)
  have been criticized for lacking compositional understanding. This means they
  might output similar scores even if the original caption is rearranged into a
  different semantic statement. To address this, we propose to use the {bf
  V}isual {bf G}enerative {bf P}re-{bf T}raining Score ({bf
  VisualGPTScore}) of P(text|image), a multimodal
  generative score that captures the likelihood of a text caption conditioned
  on an image using an image-conditioned language model. Contrary to the belief
  that VLMs are mere bag-of-words models, our off-the-shelf VisualGPTScore
  demonstrates top-tier performance on recently proposed image-text retrieval
  benchmarks like ARO and Crepe that assess compositional reasoning. Furthermore,
  we factorize VisualGPTScore into a product of the marginal P(text)
  and the Pointwise Mutual Information (PMI). This helps to (a)
  diagnose datasets with strong language bias, and (b) debias results on other
  benchmarks like Winoground using an information-theoretic framework.
  VisualGPTScore provides valuable insights and serves as a strong baseline for
  future evaluation of visio-linguistic compositionality.
paper_ai_summary: VisualGPTScore, a multimodal generative score using an image-conditioned language model, enhances compositional understanding in vision-language models and outperforms discriminatively pre-trained models on image-text retrieval benchmarks.
paper_ai_keywords: contrastive image-text matching vision-language models VisualGPTScore multimodal generative image-conditioned language model compositional reasoning ARO Crepe Winoground Pointwise Mutual Information (PMI)