title: Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
summary: Rebus puzzles, visual riddles that encode language through imagery, spatial
  arrangement, and symbolic substitution, pose a unique challenge to current
  vision-language models (VLMs). Unlike traditional image captioning or question
  answering tasks, rebus solving requires multi-modal abstraction, symbolic
  reasoning, and a grasp of cultural, phonetic and linguistic puns. In this
  paper, we investigate the capacity of contemporary VLMs to interpret and solve
  rebus puzzles by constructing a hand-generated and annotated benchmark of
  diverse English-language rebus puzzles, ranging from simple pictographic
  substitutions to spatially-dependent cues ("head" over "heels"). We analyze how
  different VLMs perform, and our findings reveal that while VLMs exhibit some
  surprising capabilities in decoding simple visual clues, they struggle
  significantly with tasks requiring abstract reasoning, lateral thinking, and
  understanding visual metaphors.
paper_authors._id: 6839d69eb385aa5d7a37bc2b
paper_authors.name: Heekyung Lee
paper_authors._id: 6839d69eb385aa5d7a37bc2c
paper_authors.name: Jiaxin Ge
paper_authors._id: 6839d69eb385aa5d7a37bc2d
paper_authors.name: Tsung-Han Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:45:00.174Z
paper_authors._id: 644a767044b75fd95805d232
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
paper_authors.fullname: Patrick (Tsung-Han) Wu
paper_authors.type: user
paper_authors.user: tsunghanwu
paper_authors._id: 6839d69eb385aa5d7a37bc2e
paper_authors.name: Minwoo Kang
paper_authors._id: 6839d69eb385aa5d7a37bc2f
paper_authors.name: Trevor Darrell
paper_authors._id: 6839d69eb385aa5d7a37bc30
paper_authors.name: David M. Chan
paper_title: Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint
paper_summary: Rebus puzzles, visual riddles that encode language through imagery, spatial
  arrangement, and symbolic substitution, pose a unique challenge to current
  vision-language models (VLMs). Unlike traditional image captioning or question
  answering tasks, rebus solving requires multi-modal abstraction, symbolic
  reasoning, and a grasp of cultural, phonetic and linguistic puns. In this
  paper, we investigate the capacity of contemporary VLMs to interpret and solve
  rebus puzzles by constructing a hand-generated and annotated benchmark of
  diverse English-language rebus puzzles, ranging from simple pictographic
  substitutions to spatially-dependent cues ("head" over "heels"). We analyze how
  different VLMs perform, and our findings reveal that while VLMs exhibit some
  surprising capabilities in decoding simple visual clues, they struggle
  significantly with tasks requiring abstract reasoning, lateral thinking, and
  understanding visual metaphors.
paper_ai_summary: Vision-language models struggle with rebus puzzles, which require abstract reasoning and understanding of visual metaphors, despite performing well on simple visual cues.
paper_ai_keywords: vision-language models VLMs multi-modal abstraction symbolic reasoning cultural puns phonetic puns linguistic puns benchmark pictographic substitutions spatially-dependent cues visual metaphors abstract reasoning lateral thinking