title: Multi-Object Hallucination in Vision-Language Models
summary: Large vision language models (LVLMs) often suffer from object hallucination,
  producing objects not present in the given images. While current benchmarks for
  object hallucination primarily concentrate on the presence of a single object
  class rather than individual entities, this work systematically investigates
  multi-object hallucination, examining how models misperceive (e.g., invent
  nonexistent objects or become distracted) when tasked with focusing on multiple
  objects simultaneously. We introduce Recognition-based Object Probing
  Evaluation (ROPE), an automated evaluation protocol that considers the
  distribution of object classes within a single image during testing and uses
  visual referring prompts to eliminate ambiguity. With comprehensive empirical
  studies and analysis of potential factors leading to multi-object
  hallucination, we found that (1) LVLMs suffer more hallucinations when focusing
  on multiple objects compared to a single object. (2) The tested object class
  distribution affects hallucination behaviors, indicating that LVLMs may follow
  shortcuts and spurious correlations.(3) Hallucinatory behaviors are influenced
  by data-specific factors, salience and frequency, and model intrinsic
  behaviors. We hope to enable LVLMs to recognize and reason about multiple
  objects that often occur in realistic visual scenes, provide insights, and
  quantify our progress towards mitigating the issues.
paper_authors._id: 668d5680f4ab25acfedd9bea
paper_authors.name: Xuweiyi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T11:57:26.951Z
paper_authors._id: 634632aaac1cb29fb2ac9f14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634632aaac1cb29fb2ac9f14/nGZ2TzKOOcKMAR_NFYKkR.jpeg
paper_authors.fullname: Xuweiyi Chen
paper_authors.type: user
paper_authors.user: Xuweiyi
paper_authors._id: 668d5680f4ab25acfedd9beb
paper_authors.name: Ziqiao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:15.401Z
paper_authors._id: 630cfc45b66f088d547b2768
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630cfc45b66f088d547b2768/ge2BZLDYeTlpnlIlraHNw.png
paper_authors.fullname: Martin Ziqiao Ma
paper_authors.type: user
paper_authors.user: marstin
paper_authors._id: 668d5680f4ab25acfedd9bec
paper_authors.name: Xuejun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T11:57:47.199Z
paper_authors._id: 655701548d9e07913d7eb335
paper_authors.avatarUrl: /avatars/27dba831406a663bcce00c16c5f63962.svg
paper_authors.fullname: Xuejun Zhang
paper_authors.type: user
paper_authors.user: yifan-Eva
paper_authors._id: 668d5680f4ab25acfedd9bed
paper_authors.name: Sihan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T11:58:02.773Z
paper_authors._id: 63f233820a16587ea967adc2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f233820a16587ea967adc2/0RDgru4GW4Hjqh8F5PxXL.jpeg
paper_authors.fullname: Sihan XU
paper_authors.type: user
paper_authors.user: sihanxu
paper_authors._id: 668d5680f4ab25acfedd9bee
paper_authors.name: Shengyi Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T11:58:08.541Z
paper_authors._id: 6445dc7e219dc6142a6a6861
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
paper_authors.fullname: Shengyi Qian
paper_authors.type: user
paper_authors.user: shengyi-qian
paper_authors._id: 668d5680f4ab25acfedd9bef
paper_authors.name: Jianing Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:06.952Z
paper_authors._id: 646afdd535c7a57f936d3ff5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LsDQPftQrnp_Gvanzim4J.jpeg
paper_authors.fullname: Jed Yang
paper_authors.type: user
paper_authors.user: jedyang97
paper_authors._id: 668d5680f4ab25acfedd9bf0
paper_authors.name: David F. Fouhey
paper_authors._id: 668d5680f4ab25acfedd9bf1
paper_authors.name: Joyce Chai
paper_title: Multi-Object Hallucination in Vision-Language Models
paper_summary: Large vision language models (LVLMs) often suffer from object hallucination,
  producing objects not present in the given images. While current benchmarks for
  object hallucination primarily concentrate on the presence of a single object
  class rather than individual entities, this work systematically investigates
  multi-object hallucination, examining how models misperceive (e.g., invent
  nonexistent objects or become distracted) when tasked with focusing on multiple
  objects simultaneously. We introduce Recognition-based Object Probing
  Evaluation (ROPE), an automated evaluation protocol that considers the
  distribution of object classes within a single image during testing and uses
  visual referring prompts to eliminate ambiguity. With comprehensive empirical
  studies and analysis of potential factors leading to multi-object
  hallucination, we found that (1) LVLMs suffer more hallucinations when focusing
  on multiple objects compared to a single object. (2) The tested object class
  distribution affects hallucination behaviors, indicating that LVLMs may follow
  shortcuts and spurious correlations.(3) Hallucinatory behaviors are influenced
  by data-specific factors, salience and frequency, and model intrinsic
  behaviors. We hope to enable LVLMs to recognize and reason about multiple
  objects that often occur in realistic visual scenes, provide insights, and
  quantify our progress towards mitigating the issues.
paper_ai_summary: ROPE, an automated evaluation protocol, examines multi-object hallucination in LVLMs, identifying factors affecting hallucinations and aiming to improve performance on realistic scenes.
paper_ai_keywords: Large vision language models LVLMs object hallucination Recognition-based Object Probing Evaluation ROPE visual referring prompts multi-object hallucination shortcuts spurious correlations