title: Visual Riddles: a Commonsense and World Knowledge Challenge for Large
    Vision and Language Models
summary: Imagine observing someone scratching their arm; to understand why, additional
  context would be necessary. However, spotting a mosquito nearby would
  immediately offer a likely explanation for the person's discomfort, thereby
  alleviating the need for further information. This example illustrates how
  subtle visual cues can challenge our cognitive skills and demonstrates the
  complexity of interpreting visual scenarios. To study these skills, we present
  Visual Riddles, a benchmark aimed to test vision and language models on visual
  riddles requiring commonsense and world knowledge. The benchmark comprises 400
  visual riddles, each featuring a unique image created by a variety of
  text-to-image models, question, ground-truth answer, textual hint, and
  attribution. Human evaluation reveals that existing models lag significantly
  behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
  leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
  to make assessment scalable. These findings underscore the potential of Visual
  Riddles as a valuable resource for enhancing vision and language models'
  capabilities in interpreting complex visual scenarios.
paper_authors._id: 66a854cddb77470d3b3ccbed
paper_authors.name: Nitzan Bitton-Guetta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-30T04:01:03.702Z
paper_authors._id: 64680ec8efbd7ae309749b8a
paper_authors.avatarUrl: /avatars/d38ff11ce1678c186e6452f0259992fc.svg
paper_authors.fullname: Yonatan Bitton
paper_authors.type: user
paper_authors.user: Yonatan-Bitton
paper_authors._id: 66a854cddb77470d3b3ccbee
paper_authors.name: Aviv Slobodkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:00:09.922Z
paper_authors._id: 631da07f6d6a5870f3d2c375
paper_authors.avatarUrl: /avatars/242e344dca08057bdf1eef09f69b41b2.svg
paper_authors.fullname: Aviv Slobodkin
paper_authors.type: user
paper_authors.user: lovodkin93
paper_authors._id: 66a854cddb77470d3b3ccbef
paper_authors.name: Aviya Maimon
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-30T02:49:53.023Z
paper_authors._id: 640b2a6d91d9f65c58a71880
paper_authors.avatarUrl: /avatars/e5b9bdddb14fc4d3f031cee2eaacd698.svg
paper_authors.fullname: Aviya Maimon
paper_authors.type: user
paper_authors.user: Aviya
paper_authors._id: 66a854cddb77470d3b3ccbf0
paper_authors.name: Eliya Habba
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:00:15.336Z
paper_authors._id: 62cd15106f6f759f2666d03c
paper_authors.avatarUrl: /avatars/695a6872f55119f08c8eef31e215a498.svg
paper_authors.fullname: Eliya Habba
paper_authors.type: user
paper_authors.user: eliyahabba
paper_authors._id: 66a854cddb77470d3b3ccbf1
paper_authors.name: Royi Rassin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:00:22.822Z
paper_authors._id: 62a7581cf049be35252a2e7c
paper_authors.avatarUrl: /avatars/91de4eb48f51bfd6e028c08ccfa98f8c.svg
paper_authors.fullname: Royi Rassin
paper_authors.type: user
paper_authors.user: Royir
paper_authors._id: 66a854cddb77470d3b3ccbf2
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:00:28.881Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 66a854cddb77470d3b3ccbf3
paper_authors.name: Idan Szpektor
paper_authors._id: 66a854cddb77470d3b3ccbf4
paper_authors.name: Amir Globerson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:00:41.852Z
paper_authors._id: 62cd956bc589e4a9e23ea347
paper_authors.avatarUrl: /avatars/dbdccd9258c9d18173d312a8bf14dd6e.svg
paper_authors.fullname: Amir Globerson
paper_authors.type: user
paper_authors.user: amirgloberson
paper_authors._id: 66a854cddb77470d3b3ccbf5
paper_authors.name: Yuval Elovici
paper_title: Visual Riddles: a Commonsense and World Knowledge Challenge for Large
    Vision and Language Models
paper_summary: Imagine observing someone scratching their arm; to understand why, additional
  context would be necessary. However, spotting a mosquito nearby would
  immediately offer a likely explanation for the person's discomfort, thereby
  alleviating the need for further information. This example illustrates how
  subtle visual cues can challenge our cognitive skills and demonstrates the
  complexity of interpreting visual scenarios. To study these skills, we present
  Visual Riddles, a benchmark aimed to test vision and language models on visual
  riddles requiring commonsense and world knowledge. The benchmark comprises 400
  visual riddles, each featuring a unique image created by a variety of
  text-to-image models, question, ground-truth answer, textual hint, and
  attribution. Human evaluation reveals that existing models lag significantly
  behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
  leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
  to make assessment scalable. These findings underscore the potential of Visual
  Riddles as a valuable resource for enhancing vision and language models'
  capabilities in interpreting complex visual scenarios.