title: CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via
    Occluded Object Counting
summary: Recognizing and reasoning about occluded (partially or fully hidden) objects
  is vital to understanding visual scenes, as occlusions frequently occur in
  real-world environments and act as obstacles for spatial comprehension. To test
  models' ability to reason about multiple occluded objects, we introduce a novel
  task, Counting Amodally for Patterns Through Unseen REgions (CAPTURe), which
  requires a model to count objects arranged in a pattern by inferring how the
  pattern continues behind an occluder (an object which blocks parts of the
  scene). CAPTURe requires both recognizing visual patterns and reasoning, making
  it a useful testbed for evaluating vision-language models (VLMs) on whether
  they understand occluded patterns and possess spatial understanding skills. By
  requiring models to reason about occluded objects, CAPTURe also tests VLMs'
  ability to form world models that would allow them to fill in missing
  information. CAPTURe consists of two parts: (1) CAPTURe-real, with manually
  filtered images of real objects in patterns and (2) CAPTURe-synthetic, a
  controlled diagnostic with generated patterned images. We evaluate four strong
  VLMs (GPT-4o, Intern-VL2, Molmo, and Qwen2-VL) on CAPTURe, finding that models
  struggle to count on both occluded and unoccluded patterns. Crucially, we find
  that models perform worse with occlusion, suggesting that VLMs are also
  deficient in inferring unseen spatial relationships: even the strongest VLMs
  like GPT-4o fail to count with occlusion. In contrast, we find that humans
  achieve very little error on CAPTURe. We also find that providing auxiliary
  information of occluded object locations increases performance, underscoring
  that the model error comes both from an inability to handle occlusion as well
  as difficulty counting in images.
paper_authors._id: 6808461cddf841838b544205
paper_authors.name: Atin Pothiraj
paper_authors._id: 6808461cddf841838b544206
paper_authors.name: Elias Stengel-Eskin
paper_authors._id: 6808461cddf841838b544207
paper_authors.name: Jaemin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-24T09:11:08.244Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 6808461cddf841838b544208
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T14:18:46.714Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_title: CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via
    Occluded Object Counting
paper_summary: Recognizing and reasoning about occluded (partially or fully hidden) objects
  is vital to understanding visual scenes, as occlusions frequently occur in
  real-world environments and act as obstacles for spatial comprehension. To test
  models' ability to reason about multiple occluded objects, we introduce a novel
  task, Counting Amodally for Patterns Through Unseen REgions (CAPTURe), which
  requires a model to count objects arranged in a pattern by inferring how the
  pattern continues behind an occluder (an object which blocks parts of the
  scene). CAPTURe requires both recognizing visual patterns and reasoning, making
  it a useful testbed for evaluating vision-language models (VLMs) on whether
  they understand occluded patterns and possess spatial understanding skills. By
  requiring models to reason about occluded objects, CAPTURe also tests VLMs'
  ability to form world models that would allow them to fill in missing
  information. CAPTURe consists of two parts: (1) CAPTURe-real, with manually
  filtered images of real objects in patterns and (2) CAPTURe-synthetic, a
  controlled diagnostic with generated patterned images. We evaluate four strong
  VLMs (GPT-4o, Intern-VL2, Molmo, and Qwen2-VL) on CAPTURe, finding that models
  struggle to count on both occluded and unoccluded patterns. Crucially, we find
  that models perform worse with occlusion, suggesting that VLMs are also
  deficient in inferring unseen spatial relationships: even the strongest VLMs
  like GPT-4o fail to count with occlusion. In contrast, we find that humans
  achieve very little error on CAPTURe. We also find that providing auxiliary
  information of occluded object locations increases performance, underscoring
  that the model error comes both from an inability to handle occlusion as well
  as difficulty counting in images.
paper_ai_summary: A novel task, CAPTURe, is introduced to evaluate vision-language models' ability to count objects in patterns with occlusions, revealing that models struggle significantly compared to humans.
paper_ai_keywords: vision-language models counting occlusions world models CAPTURe-real CAPTURe-synthetic spatial understanding auxiliary information