title: OmniBench: Towards The Future of Universal Omni-Language Models
summary: Recent advancements in multimodal large language models (MLLMs) have aimed to
  integrate and interpret data across diverse modalities. However, the capacity
  of these models to concurrently process and reason about multiple modalities
  remains inadequately explored, partly due to the lack of comprehensive
  modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to
  rigorously evaluate models' ability to recognize, interpret, and reason across
  visual, acoustic, and textual inputs simultaneously. We define models capable
  of such tri-modal processing as omni-language models (OLMs). OmniBench is
  distinguished by high-quality human annotations, ensuring that accurate
  responses require integrated understanding and reasoning across all three
  modalities. Our main findings reveal that: i) open-source OLMs exhibit critical
  limitations in instruction-following and reasoning capabilities within
  tri-modal contexts; and ii) the baseline models perform poorly (below 50%
  accuracy) even when provided with alternative textual representations of images
  and audio. These results suggest that the ability to construct a consistent
  context from text, image, and audio is often overlooked in existing MLLM
  training paradigms. We advocate for future research to focus on developing more
  robust tri-modal integration techniques and training strategies to enhance OLM
  performance across diverse modalities. The codes and live leaderboard could be
  found at https://m-a-p.ai/OmniBench.
paper_authors._id: 66f2a7c2707fbf6273d63eca
paper_authors.name: Yizhi Li
paper_authors._id: 66f2a7c2707fbf6273d63ecb
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:33:55.270Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66f2a7c2707fbf6273d63ecc
paper_authors.name: Yinghao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T18:24:24.145Z
paper_authors._id: 6410665d5364a661bee22524
paper_authors.avatarUrl: /avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
paper_authors.fullname: Yinghao Ma
paper_authors.type: user
paper_authors.user: nicolaus625
paper_authors._id: 66f2a7c2707fbf6273d63ecd
paper_authors.name: Ruibin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:51:04.494Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 66f2a7c2707fbf6273d63ece
paper_authors.name: Kang Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:51:11.857Z
paper_authors._id: 6578265ddea7e2122d02f6ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
paper_authors.fullname: kang zhu
paper_authors.type: user
paper_authors.user: kangz
paper_authors._id: 66f2a7c2707fbf6273d63ecf
paper_authors.name: Hangyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:51:18.375Z
paper_authors._id: 646b43deb1202bc77c1024a4
paper_authors.avatarUrl: /avatars/cf791574ab986bac274e7fbcf04e2a59.svg
paper_authors.fullname: hangyu guo
paper_authors.type: user
paper_authors.user: Rosiness
paper_authors._id: 66f2a7c2707fbf6273d63ed0
paper_authors.name: Yiming Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:51:51.973Z
paper_authors._id: 654fed4d7490049d621d0cd3
paper_authors.avatarUrl: /avatars/6bc1553d1571ab4e9ab80c58eeb9820c.svg
paper_authors.fullname: Yiming Liang
paper_authors.type: user
paper_authors.user: Yiming1234
paper_authors._id: 66f2a7c2707fbf6273d63ed1
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:52:01.110Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 66f2a7c2707fbf6273d63ed2
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:01:36.015Z
paper_authors._id: 64ccb9bfead94891d12aef42
paper_authors.avatarUrl: /avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
paper_authors.fullname: Yang Jian
paper_authors.type: user
paper_authors.user: CSJianYang
paper_authors._id: 66f2a7c2707fbf6273d63ed3
paper_authors.name: Siwei Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:52:23.793Z
paper_authors._id: 656d97b10bbc114fe64a96c5
paper_authors.avatarUrl: /avatars/fd23bae1d85c5b96c42064a5ddcfad41.svg
paper_authors.fullname: SiweiWu
paper_authors.type: user
paper_authors.user: SiweiWu
paper_authors._id: 66f2a7c2707fbf6273d63ed4
paper_authors.name: Xingwei Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T18:24:26.132Z
paper_authors._id: 646b4c9fdf2609a541c0866e
paper_authors.avatarUrl: /avatars/7ec6c709017dcf32d4ac49d1e3820328.svg
paper_authors.fullname: Qu
paper_authors.type: user
paper_authors.user: ScottQu
paper_authors._id: 66f2a7c2707fbf6273d63ed5
paper_authors.name: Jinjie Shi
paper_authors._id: 66f2a7c2707fbf6273d63ed6
paper_authors.name: Xinyue Zhang
paper_authors._id: 66f2a7c2707fbf6273d63ed7
paper_authors.name: Zhenzhu Yang
paper_authors._id: 66f2a7c2707fbf6273d63ed8
paper_authors.name: Xiangzhou Wang
paper_authors._id: 66f2a7c2707fbf6273d63ed9
paper_authors.name: Zhaoxiang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:53:26.760Z
paper_authors._id: 6588d5711b44d0e6942e8489
paper_authors.avatarUrl: /avatars/f6624b46071e4d1548ea719f26b90b17.svg
paper_authors.fullname: zhangzhaoxiang
paper_authors.type: user
paper_authors.user: zhangzhaoxiang
paper_authors._id: 66f2a7c2707fbf6273d63eda
paper_authors.name: Zachary Liu
paper_authors._id: 66f2a7c2707fbf6273d63edb
paper_authors.name: Emmanouil Benetos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T12:49:49.100Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 66f2a7c2707fbf6273d63edc
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:25:45.765Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 66f2a7c2707fbf6273d63edd
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T11:15:16.059Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_title: OmniBench: Towards The Future of Universal Omni-Language Models
paper_summary: Recent advancements in multimodal large language models (MLLMs) have aimed to
  integrate and interpret data across diverse modalities. However, the capacity
  of these models to concurrently process and reason about multiple modalities
  remains inadequately explored, partly due to the lack of comprehensive
  modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to
  rigorously evaluate models' ability to recognize, interpret, and reason across
  visual, acoustic, and textual inputs simultaneously. We define models capable
  of such tri-modal processing as omni-language models (OLMs). OmniBench is
  distinguished by high-quality human annotations, ensuring that accurate
  responses require integrated understanding and reasoning across all three
  modalities. Our main findings reveal that: i) open-source OLMs exhibit critical
  limitations in instruction-following and reasoning capabilities within
  tri-modal contexts; and ii) the baseline models perform poorly (below 50%
  accuracy) even when provided with alternative textual representations of images
  and audio. These results suggest that the ability to construct a consistent
  context from text, image, and audio is often overlooked in existing MLLM
  training paradigms. We advocate for future research to focus on developing more
  robust tri-modal integration techniques and training strategies to enhance OLM
  performance across diverse modalities. The codes and live leaderboard could be
  found at https://m-a-p.ai/OmniBench.
paper_ai_summary: OmniBench is a benchmark for evaluating multimodal large language models' ability to simultaneously process and reason across visual, acoustic, and textual inputs, revealing significant limitations in current models.
paper_ai_keywords: multimodal large language models OLMs OmniBench tri-modal processing instruction-following reasoning capabilities human annotations tri-modal contexts