title: MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
summary: Perceiving and generating diverse modalities are crucial for AI models to
  effectively learn from and engage with real-world signals, necessitating
  reliable evaluations for their development. We identify two major issues in
  current evaluations: (1) inconsistent standards, shaped by different
  communities with varying protocols and maturity levels; and (2) significant
  query, grading, and generalization biases. To address these, we introduce
  MixEval-X, the first any-to-any real-world benchmark designed to optimize and
  standardize evaluations across input and output modalities. We propose
  multi-modal benchmark mixture and adaptation-rectification pipelines to
  reconstruct real-world task distributions, ensuring evaluations generalize
  effectively to real-world use cases. Extensive meta-evaluations show our
  approach effectively aligns benchmark samples with real-world task
  distributions and the model rankings correlate strongly with that of
  crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
  leaderboards to rerank existing models and organizations and offer insights to
  enhance understanding of multi-modal evaluations and inform future research.
paper_authors._id: 6711de895b7360457e26dff7
paper_authors.name: Jinjie Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:55:08.722Z
paper_authors._id: 62a7362fd1e7a011fd4e31a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7362fd1e7a011fd4e31a7/ZY_mwH-sI0o05SHpLqwc7.png
paper_authors.fullname: Jinjie Ni
paper_authors.type: user
paper_authors.user: jinjieni
paper_authors._id: 6711de895b7360457e26dff8
paper_authors.name: Yifan Song
paper_authors._id: 6711de895b7360457e26dff9
paper_authors.name: Deepanway Ghosal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T09:31:24.961Z
paper_authors._id: 62eced5e7e89e8d34df1a1ea
paper_authors.avatarUrl: /avatars/99c9d8ba7e7722b7524d5d687cf96a25.svg
paper_authors.fullname: Deepanway Ghosal
paper_authors.type: user
paper_authors.user: dghosal
paper_authors._id: 6711de895b7360457e26dffa
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:14.198Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 6711de895b7360457e26dffb
paper_authors.name: David Junhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T10:33:07.038Z
paper_authors._id: 6468855763a564ba347e1efa
paper_authors.avatarUrl: /avatars/caa99c3318c1d6d21be44d2d32795a62.svg
paper_authors.fullname: David Junhao ZHANG
paper_authors.type: user
paper_authors.user: Junhao233
paper_authors._id: 6711de895b7360457e26dffc
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T14:55:42.911Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 6711de895b7360457e26dffd
paper_authors.name: Fuzhao Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T10:33:15.728Z
paper_authors._id: 64a52314978242bb77e05790
paper_authors.avatarUrl: /avatars/e97206c653de3446266c80c2943c0a6f.svg
paper_authors.fullname: Fuzhao Xue
paper_authors.type: user
paper_authors.user: fuzhao
paper_authors._id: 6711de895b7360457e26dffe
paper_authors.name: Zian Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:16.263Z
paper_authors._id: 6371e32aafbe42caa5ad9189
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371e32aafbe42caa5ad9189/Frqjr_L1BGyg4yWwVXKiv.jpeg
paper_authors.fullname: Zheng Zian(Andy)
paper_authors.type: user
paper_authors.user: OrionZheng
paper_authors._id: 6711de895b7360457e26dfff
paper_authors.name: Kaichen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:13.789Z
paper_authors._id: 64bb77e786e7fb5b8a317a43
paper_authors.avatarUrl: /avatars/4c1720db323082e644babeb827c6f5d4.svg
paper_authors.fullname: kcz
paper_authors.type: user
paper_authors.user: kcz358
paper_authors._id: 6711de895b7360457e26e000
paper_authors.name: Mahir Shah
paper_authors._id: 6711de895b7360457e26e001
paper_authors.name: Kabir Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:17.775Z
paper_authors._id: 648763974c7531fdad63de41
paper_authors.avatarUrl: /avatars/ca629fa7aa0749d85c1e3fa3f44f009e.svg
paper_authors.fullname: Kabir
paper_authors.type: user
paper_authors.user: kabear
paper_authors._id: 6711de895b7360457e26e002
paper_authors.name: Yang You
paper_authors._id: 6711de895b7360457e26e003
paper_authors.name: Michael Shieh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T09:56:45.883Z
paper_authors._id: 64d48e9ef2a3941669810f23
paper_authors.avatarUrl: /avatars/495e6785abc44d1facfcf385edd6a222.svg
paper_authors.fullname: Michael Shieh
paper_authors.type: user
paper_authors.user: Michael2008S
paper_title: MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
paper_summary: Perceiving and generating diverse modalities are crucial for AI models to
  effectively learn from and engage with real-world signals, necessitating
  reliable evaluations for their development. We identify two major issues in
  current evaluations: (1) inconsistent standards, shaped by different
  communities with varying protocols and maturity levels; and (2) significant
  query, grading, and generalization biases. To address these, we introduce
  MixEval-X, the first any-to-any real-world benchmark designed to optimize and
  standardize evaluations across input and output modalities. We propose
  multi-modal benchmark mixture and adaptation-rectification pipelines to
  reconstruct real-world task distributions, ensuring evaluations generalize
  effectively to real-world use cases. Extensive meta-evaluations show our
  approach effectively aligns benchmark samples with real-world task
  distributions and the model rankings correlate strongly with that of
  crowd-sourced real-world evaluations (up to 0.98). We provide comprehensive
  leaderboards to rerank existing models and organizations and offer insights to
  enhance understanding of multi-modal evaluations and inform future research.
paper_ai_keywords: multi-modal real-world benchmark benchmark mixture adaptation-rectification pipelines real-world task distributions crowd-sourced real-world evaluations leaderboards