Visual Riddles: a Commonsense and World Knowledge Challenge for Large
  Vision and Language Models
Imagine observing someone scratching their arm; to understand why, additional
context would be necessary. However, spotting a mosquito nearby would
immediately offer a likely explanation for the person's discomfort, thereby
alleviating the need for further information. This example illustrates how
subtle visual cues can challenge our cognitive skills and demonstrates the
complexity of interpreting visual scenarios. To study these skills, we present
Visual Riddles, a benchmark aimed to test vision and language models on visual
riddles requiring commonsense and world knowledge. The benchmark comprises 400
visual riddles, each featuring a unique image created by a variety of
text-to-image models, question, ground-truth answer, textual hint, and
attribution. Human evaluation reveals that existing models lag significantly
behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
to make assessment scalable. These findings underscore the potential of Visual
Riddles as a valuable resource for enhancing vision and language models'
capabilities in interpreting complex visual scenarios.
66a854cddb77470d3b3ccbed
Nitzan Bitton-Guetta
extracted_confirmed
2024-07-30T04:01:03.702Z
64680ec8efbd7ae309749b8a
/avatars/d38ff11ce1678c186e6452f0259992fc.svg
Yonatan Bitton
user
Yonatan-Bitton
66a854cddb77470d3b3ccbee
Aviv Slobodkin
admin_assigned
2024-07-30T09:00:09.922Z
631da07f6d6a5870f3d2c375
/avatars/242e344dca08057bdf1eef09f69b41b2.svg
Aviv Slobodkin
user
lovodkin93
66a854cddb77470d3b3ccbef
Aviya Maimon
extracted_pending
2024-07-30T02:49:53.023Z
640b2a6d91d9f65c58a71880
/avatars/e5b9bdddb14fc4d3f031cee2eaacd698.svg
Aviya Maimon
user
Aviya
66a854cddb77470d3b3ccbf0
Eliya Habba
admin_assigned
2024-07-30T09:00:15.336Z
62cd15106f6f759f2666d03c
/avatars/695a6872f55119f08c8eef31e215a498.svg
Eliya Habba
user
eliyahabba
66a854cddb77470d3b3ccbf1
Royi Rassin
admin_assigned
2024-07-30T09:00:22.822Z
62a7581cf049be35252a2e7c
/avatars/91de4eb48f51bfd6e028c08ccfa98f8c.svg
Royi Rassin
user
Royir
66a854cddb77470d3b3ccbf2
Yonatan Bitton
admin_assigned
2024-07-30T09:00:28.881Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
66a854cddb77470d3b3ccbf3
Idan Szpektor
66a854cddb77470d3b3ccbf4
Amir Globerson
admin_assigned
2024-07-30T09:00:41.852Z
62cd956bc589e4a9e23ea347
/avatars/dbdccd9258c9d18173d312a8bf14dd6e.svg
Amir Globerson
user
amirgloberson
66a854cddb77470d3b3ccbf5
Yuval Elovici
Visual Riddles: a Commonsense and World Knowledge Challenge for Large
  Vision and Language Models
Imagine observing someone scratching their arm; to understand why, additional
context would be necessary. However, spotting a mosquito nearby would
immediately offer a likely explanation for the person's discomfort, thereby
alleviating the need for further information. This example illustrates how
subtle visual cues can challenge our cognitive skills and demonstrates the
complexity of interpreting visual scenarios. To study these skills, we present
Visual Riddles, a benchmark aimed to test vision and language models on visual
riddles requiring commonsense and world knowledge. The benchmark comprises 400
visual riddles, each featuring a unique image created by a variety of
text-to-image models, question, ground-truth answer, textual hint, and
attribution. Human evaluation reveals that existing models lag significantly
behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
to make assessment scalable. These findings underscore the potential of Visual
Riddles as a valuable resource for enhancing vision and language models'
capabilities in interpreting complex visual scenarios.
Visual Riddles is a benchmark that tests vision and language models on visual riddles requiring commonsense and world knowledge, highlighting the gap between human and model performance.
vision and language models visual riddles commonsense world knowledge human evaluation text-to-image models