title: AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand
    Audio-Visual Information?
summary: Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini
  1.5 Pro, and Reka Core, have expanded their capabilities to include vision and
  audio modalities. While these models demonstrate impressive performance across
  a wide range of audio-visual applications, our proposed DeafTest reveals that
  MLLMs often struggle with simple tasks humans find trivial: 1) determining
  which of two sounds is louder, and 2) determining which of two sounds has a
  higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a
  comprehensive audio-visual benchmark designed to assess whether those MLLMs can
  truly understand the audio-visual information. This benchmark encompasses 4,555
  carefully crafted problems, each incorporating text, visual, and audio
  components. To successfully infer answers, models must effectively leverage
  clues from both visual and audio inputs. To ensure precise and objective
  evaluation of MLLM responses, we have structured the questions as
  multiple-choice, eliminating the need for human evaluation or LLM-assisted
  assessment. We benchmark a series of closed-source and open-source models and
  summarize the observations. By revealing the limitations of current models, we
  aim to provide useful insight for future dataset collection and model
  development.
paper_authors._id: 674fcf2daef6d44142cd7d7e
paper_authors.name: Kaixiong Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:00:06.067Z
paper_authors._id: 642e427f6748dd4f8eeb2f38
paper_authors.avatarUrl: /avatars/07158ff6aa1803c846403594c5d55a34.svg
paper_authors.fullname: Kaixiong Gong
paper_authors.type: user
paper_authors.user: kxgong
paper_authors._id: 674fcf2daef6d44142cd7d7f
paper_authors.name: Kaituo Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:00:00.501Z
paper_authors._id: 67079840a9bcb7459b8d2a46
paper_authors.avatarUrl: /avatars/32466863c5554f20cb2775b138832ac3.svg
paper_authors.fullname: Kaituo Feng
paper_authors.type: user
paper_authors.user: KaituoFeng
paper_authors._id: 674fcf2daef6d44142cd7d80
paper_authors.name: Bohao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:59:54.784Z
paper_authors._id: 6310b7e70a43f97f6c56191e
paper_authors.avatarUrl: /avatars/4a24c76e34d12c3d6230a4a081115f72.svg
paper_authors.fullname: Bohao Li
paper_authors.type: user
paper_authors.user: BreakLee
paper_authors._id: 674fcf2daef6d44142cd7d81
paper_authors.name: Yibing Wang
paper_authors._id: 674fcf2daef6d44142cd7d82
paper_authors.name: Mofan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:59:13.651Z
paper_authors._id: 64d5fe73bb9b5ddddc6887e1
paper_authors.avatarUrl: /avatars/6f5d678f114d47e72c436ce86d0cd51c.svg
paper_authors.fullname: MoFanCheng
paper_authors.type: user
paper_authors.user: MoFanCheng
paper_authors._id: 674fcf2daef6d44142cd7d83
paper_authors.name: Shijia Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:59:07.269Z
paper_authors._id: 64374254a701a7e744c033a1
paper_authors.avatarUrl: /avatars/a103d27fe32c520cb51191920a5b548a.svg
paper_authors.fullname: Shijia Yang
paper_authors.type: user
paper_authors.user: shijiay
paper_authors._id: 674fcf2daef6d44142cd7d84
paper_authors.name: Jiaming Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:59:01.877Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 674fcf2daef6d44142cd7d85
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:58:54.968Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 674fcf2daef6d44142cd7d86
paper_authors.name: Yutong Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:58:45.063Z
paper_authors._id: 6332253749a95639154cc894
paper_authors.avatarUrl: /avatars/f8e53fff5a324591026114c431cd407e.svg
paper_authors.fullname: Yutong Bai
paper_authors.type: user
paper_authors.user: Emma02
paper_authors._id: 674fcf2daef6d44142cd7d87
paper_authors.name: Zhuoran Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:58:39.115Z
paper_authors._id: 646d769cda8e99940b71928e
paper_authors.avatarUrl: /avatars/acee495a23362aa39b3d3e75c9afd967.svg
paper_authors.fullname: Zhuoran Yang
paper_authors.type: user
paper_authors.user: zhuoranyang
paper_authors._id: 674fcf2daef6d44142cd7d88
paper_authors.name: Xiangyu Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:58:16.819Z
paper_authors._id: 666a8f24e2990b0cb16b7bf9
paper_authors.avatarUrl: /avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
paper_authors.fullname: Xiangyu Yue
paper_authors.type: user
paper_authors.user: xyyue
paper_title: AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand
    Audio-Visual Information?
paper_summary: Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini
  1.5 Pro, and Reka Core, have expanded their capabilities to include vision and
  audio modalities. While these models demonstrate impressive performance across
  a wide range of audio-visual applications, our proposed DeafTest reveals that
  MLLMs often struggle with simple tasks humans find trivial: 1) determining
  which of two sounds is louder, and 2) determining which of two sounds has a
  higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a
  comprehensive audio-visual benchmark designed to assess whether those MLLMs can
  truly understand the audio-visual information. This benchmark encompasses 4,555
  carefully crafted problems, each incorporating text, visual, and audio
  components. To successfully infer answers, models must effectively leverage
  clues from both visual and audio inputs. To ensure precise and objective
  evaluation of MLLM responses, we have structured the questions as
  multiple-choice, eliminating the need for human evaluation or LLM-assisted
  assessment. We benchmark a series of closed-source and open-source models and
  summarize the observations. By revealing the limitations of current models, we
  aim to provide useful insight for future dataset collection and model
  development.
paper_ai_summary: AV-Odyssey Bench evaluates the audio-visual understanding of large language models through a comprehensive benchmark of 4,555 multiple-choice questions.
paper_ai_keywords: multimodal large language models MLLMs GPT-4o Gemini 1.5 Pro Reka Core audio-visual benchmark audio-visual understanding multiple-choice questions