title: NL-Eye: Abductive NLI for Images
summary: Will a Visual Language Model (VLM)-based bot warn us about slipping if it
  detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
  their ability to infer outcomes and causes remains underexplored. To address
  this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
  abductive reasoning skills. NL-Eye adapts the abductive Natural Language
  Inference (NLI) task to the visual domain, requiring models to evaluate the
  plausibility of hypothesis images based on a premise image and explain their
  decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
  images) spanning diverse reasoning categories: physical, functional, logical,
  emotional, cultural, and social. The data curation process involved two steps -
  writing textual descriptions and generating images using text-to-image models,
  both requiring substantial human involvement to ensure high-quality and
  challenging scenes. Our experiments show that VLMs struggle significantly on
  NL-Eye, often performing at random baseline levels, while humans excel in both
  plausibility prediction and explanation quality. This demonstrates a deficiency
  in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
  crucial step toward developing VLMs capable of robust multimodal reasoning for
  real-world applications, including accident-prevention bots and generated video
  verification.
paper_authors._id: 67024ddf8572eb2d80f01e3a
paper_authors.name: Mor Ventura
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:26:50.210Z
paper_authors._id: 623f4bd2e801e8c1e59d948e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648315730065-623f4bd2e801e8c1e59d948e.jpeg
paper_authors.fullname: Mor Ventura
paper_authors.type: user
paper_authors.user: MorVentura
paper_authors._id: 67024ddf8572eb2d80f01e3b
paper_authors.name: Michael Toker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:26:48.325Z
paper_authors._id: 61c865e3d3702a3bbf50bc04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c865e3d3702a3bbf50bc04/eGgcyeUnOHz0hyap5vhvr.jpeg
paper_authors.fullname: Michael Toker
paper_authors.type: user
paper_authors.user: tokeron
paper_authors._id: 67024ddf8572eb2d80f01e3c
paper_authors.name: Nitay Calderon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T08:44:00.870Z
paper_authors._id: 62d6a0c18faee0ac953c51fa
paper_authors.avatarUrl: /avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
paper_authors.fullname: Nitay Calderon
paper_authors.type: user
paper_authors.user: nitay
paper_authors._id: 67024ddf8572eb2d80f01e3d
paper_authors.name: Zorik Gekhman
paper_authors._id: 67024ddf8572eb2d80f01e3e
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T08:42:35.195Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 67024ddf8572eb2d80f01e3f
paper_authors.name: Roi Reichart
paper_title: NL-Eye: Abductive NLI for Images
paper_summary: Will a Visual Language Model (VLM)-based bot warn us about slipping if it
  detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
  their ability to infer outcomes and causes remains underexplored. To address
  this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
  abductive reasoning skills. NL-Eye adapts the abductive Natural Language
  Inference (NLI) task to the visual domain, requiring models to evaluate the
  plausibility of hypothesis images based on a premise image and explain their
  decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
  images) spanning diverse reasoning categories: physical, functional, logical,
  emotional, cultural, and social. The data curation process involved two steps -
  writing textual descriptions and generating images using text-to-image models,
  both requiring substantial human involvement to ensure high-quality and
  challenging scenes. Our experiments show that VLMs struggle significantly on
  NL-Eye, often performing at random baseline levels, while humans excel in both
  plausibility prediction and explanation quality. This demonstrates a deficiency
  in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
  crucial step toward developing VLMs capable of robust multimodal reasoning for
  real-world applications, including accident-prevention bots and generated video
  verification.
paper_ai_summary: NL-Eye is a benchmark that assesses VLMs' visual abductive reasoning skills by evaluating their ability to predict and explain the plausibility of hypothesis images based on premise images.
paper_ai_keywords: Visual Language Model VLM NL-Eye abductive reasoning abductive Natural Language Inference NLI text-to-image models multimodal reasoning accident-prevention bots generated video verification