AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand
  Audio-Visual Information?
Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini
1.5 Pro, and Reka Core, have expanded their capabilities to include vision and
audio modalities. While these models demonstrate impressive performance across
a wide range of audio-visual applications, our proposed DeafTest reveals that
MLLMs often struggle with simple tasks humans find trivial: 1) determining
which of two sounds is louder, and 2) determining which of two sounds has a
higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a
comprehensive audio-visual benchmark designed to assess whether those MLLMs can
truly understand the audio-visual information. This benchmark encompasses 4,555
carefully crafted problems, each incorporating text, visual, and audio
components. To successfully infer answers, models must effectively leverage
clues from both visual and audio inputs. To ensure precise and objective
evaluation of MLLM responses, we have structured the questions as
multiple-choice, eliminating the need for human evaluation or LLM-assisted
assessment. We benchmark a series of closed-source and open-source models and
summarize the observations. By revealing the limitations of current models, we
aim to provide useful insight for future dataset collection and model
development.
674fcf2daef6d44142cd7d7e
Kaixiong Gong
admin_assigned
2024-12-04T12:00:06.067Z
642e427f6748dd4f8eeb2f38
/avatars/07158ff6aa1803c846403594c5d55a34.svg
Kaixiong Gong
user
kxgong
674fcf2daef6d44142cd7d7f
Kaituo Feng
admin_assigned
2024-12-04T12:00:00.501Z
67079840a9bcb7459b8d2a46
/avatars/32466863c5554f20cb2775b138832ac3.svg
Kaituo Feng
user
KaituoFeng
674fcf2daef6d44142cd7d80
Bohao Li
admin_assigned
2024-12-04T11:59:54.784Z
6310b7e70a43f97f6c56191e
/avatars/4a24c76e34d12c3d6230a4a081115f72.svg
Bohao Li
user
BreakLee
674fcf2daef6d44142cd7d81
Yibing Wang
674fcf2daef6d44142cd7d82
Mofan Cheng
admin_assigned
2024-12-04T11:59:13.651Z
64d5fe73bb9b5ddddc6887e1
/avatars/6f5d678f114d47e72c436ce86d0cd51c.svg
MoFanCheng
user
MoFanCheng
674fcf2daef6d44142cd7d83
Shijia Yang
admin_assigned
2024-12-04T11:59:07.269Z
64374254a701a7e744c033a1
/avatars/a103d27fe32c520cb51191920a5b548a.svg
Shijia Yang
user
shijiay
674fcf2daef6d44142cd7d84
Jiaming Han
admin_assigned
2024-12-04T11:59:01.877Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
674fcf2daef6d44142cd7d85
Benyou Wang
admin_assigned
2024-12-04T11:58:54.968Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
674fcf2daef6d44142cd7d86
Yutong Bai
admin_assigned
2024-12-04T11:58:45.063Z
6332253749a95639154cc894
/avatars/f8e53fff5a324591026114c431cd407e.svg
Yutong Bai
user
Emma02
674fcf2daef6d44142cd7d87
Zhuoran Yang
admin_assigned
2024-12-04T11:58:39.115Z
646d769cda8e99940b71928e
/avatars/acee495a23362aa39b3d3e75c9afd967.svg
Zhuoran Yang
user
zhuoranyang
674fcf2daef6d44142cd7d88
Xiangyu Yue
admin_assigned
2024-12-04T11:58:16.819Z
666a8f24e2990b0cb16b7bf9
/avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
Xiangyu Yue
user
xyyue
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand
  Audio-Visual Information?
Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini
1.5 Pro, and Reka Core, have expanded their capabilities to include vision and
audio modalities. While these models demonstrate impressive performance across
a wide range of audio-visual applications, our proposed DeafTest reveals that
MLLMs often struggle with simple tasks humans find trivial: 1) determining
which of two sounds is louder, and 2) determining which of two sounds has a
higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a
comprehensive audio-visual benchmark designed to assess whether those MLLMs can
truly understand the audio-visual information. This benchmark encompasses 4,555
carefully crafted problems, each incorporating text, visual, and audio
components. To successfully infer answers, models must effectively leverage
clues from both visual and audio inputs. To ensure precise and objective
evaluation of MLLM responses, we have structured the questions as
multiple-choice, eliminating the need for human evaluation or LLM-assisted
assessment. We benchmark a series of closed-source and open-source models and
summarize the observations. By revealing the limitations of current models, we
aim to provide useful insight for future dataset collection and model
development.
AV-Odyssey Bench evaluates the audio-visual understanding of large language models through a comprehensive benchmark of 4,555 multiple-choice questions.
multimodal large language models MLLMs GPT-4o Gemini 1.5 Pro Reka Core audio-visual benchmark audio-visual understanding multiple-choice questions