title: MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
summary: We present MEGA-Bench, an evaluation suite that scales multimodal evaluation
  to over 500 real-world tasks, to address the highly heterogeneous daily use
  cases of end users. Our objective is to optimize for a set of high-quality data
  samples that cover a highly diverse and rich set of multimodal tasks, while
  enabling cost-effective and accurate model evaluation. In particular, we
  collected 505 realistic tasks encompassing over 8,000 samples from 16 expert
  annotators to extensively cover the multimodal task space. Instead of unifying
  these problems into standard multi-choice questions (like MMMU, MMBench, and
  MMT-Bench), we embrace a wide range of output formats like numbers, phrases,
  code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats,
  we developed over 40 metrics to evaluate these tasks. Unlike existing
  benchmarks, MEGA-Bench offers a fine-grained capability report across multiple
  dimensions (e.g., application, input type, output format, skill), allowing
  users to interact with and visualize model capabilities in depth. We evaluate a
  wide variety of frontier vision-language models on MEGA-Bench to understand
  their capabilities across these dimensions.
paper_authors._id: 670dd82d85a1548f5cdf99fd
paper_authors.name: Jiacheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:32:05.739Z
paper_authors._id: 655bca95360e4f90cb61ba83
paper_authors.avatarUrl: /avatars/1a187beb91a5e2fdc2303620b742aab1.svg
paper_authors.fullname: Jiacheng Chen
paper_authors.type: user
paper_authors.user: cccjc
paper_authors._id: 670dd82d85a1548f5cdf99fe
paper_authors.name: Tianhao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:45.081Z
paper_authors._id: 6703ec213df5fe425086ef73
paper_authors.avatarUrl: /avatars/e6f9dad6587ee0883ae10f8805ab7ea9.svg
paper_authors.fullname: Tianhao Liang
paper_authors.type: user
paper_authors.user: tianhao2k
paper_authors._id: 670dd82d85a1548f5cdf99ff
paper_authors.name: Sherman Siu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:52.899Z
paper_authors._id: 630d73387dacb93b335b00e9
paper_authors.avatarUrl: /avatars/2ded3550aedf77a58175a66319d01820.svg
paper_authors.fullname: Sherman Siu
paper_authors.type: user
paper_authors.user: shermansiu
paper_authors._id: 670dd82d85a1548f5cdf9a00
paper_authors.name: Zhengqing Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:33:34.951Z
paper_authors._id: 64d97c5bfd0b55d501ba00cf
paper_authors.avatarUrl: /avatars/47505f2a573acea7176a96f538226ecb.svg
paper_authors.fullname: Zhengqing Wang
paper_authors.type: user
paper_authors.user: EricW123456
paper_authors._id: 670dd82d85a1548f5cdf9a01
paper_authors.name: Kai Wang
paper_authors._id: 670dd82d85a1548f5cdf9a02
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:41:25.642Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: /avatars/d9c5cf3491243d1f2b1c5df1873ee8e7.svg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 670dd82d85a1548f5cdf9a03
paper_authors.name: Yuansheng Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:49.671Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 670dd82d85a1548f5cdf9a04
paper_authors.name: Wang Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:31.293Z
paper_authors._id: 6389568eec1f539adc0e0337
paper_authors.avatarUrl: /avatars/6ee5c9bea9989e4618ae8d334bd889a9.svg
paper_authors.fullname: Wang Zhu
paper_authors.type: user
paper_authors.user: Bill1235813
paper_authors._id: 670dd82d85a1548f5cdf9a05
paper_authors.name: Ziyan Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T13:06:01.261Z
paper_authors._id: 64778fb8168cb428e00f69b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
paper_authors.fullname: Ziyan Jiang
paper_authors.type: user
paper_authors.user: ziyjiang
paper_authors._id: 670dd82d85a1548f5cdf9a06
paper_authors.name: Bohan Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:24:09.935Z
paper_authors._id: 650267e7e751d03da933a24a
paper_authors.avatarUrl: /avatars/f047a047d1de304cd97027463541bdf3.svg
paper_authors.fullname: Bohan22
paper_authors.type: user
paper_authors.user: Bohan22
paper_authors._id: 670dd82d85a1548f5cdf9a07
paper_authors.name: Dongfu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:51.346Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 670dd82d85a1548f5cdf9a08
paper_authors.name: Xuan He
paper_authors._id: 670dd82d85a1548f5cdf9a09
paper_authors.name: Yuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:31:06.125Z
paper_authors._id: 64d47a7a508a6313e33faedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
paper_authors.fullname: Yuan Liu
paper_authors.type: user
paper_authors.user: YuanLiuuuuuu
paper_authors._id: 670dd82d85a1548f5cdf9a0a
paper_authors.name: Hexiang Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:30:44.118Z
paper_authors._id: 643441ccd55dea2d0ec2c309
paper_authors.avatarUrl: /avatars/82e99d445e2b513ad7270fa852adbcbb.svg
paper_authors.fullname: Hexiang Hu
paper_authors.type: user
paper_authors.user: hexianghu
paper_authors._id: 670dd82d85a1548f5cdf9a0b
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:54.823Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 670dd82d85a1548f5cdf9a0c
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-15T02:49:20.901Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
paper_summary: We present MEGA-Bench, an evaluation suite that scales multimodal evaluation
  to over 500 real-world tasks, to address the highly heterogeneous daily use
  cases of end users. Our objective is to optimize for a set of high-quality data
  samples that cover a highly diverse and rich set of multimodal tasks, while
  enabling cost-effective and accurate model evaluation. In particular, we
  collected 505 realistic tasks encompassing over 8,000 samples from 16 expert
  annotators to extensively cover the multimodal task space. Instead of unifying
  these problems into standard multi-choice questions (like MMMU, MMBench, and
  MMT-Bench), we embrace a wide range of output formats like numbers, phrases,
  code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats,
  we developed over 40 metrics to evaluate these tasks. Unlike existing
  benchmarks, MEGA-Bench offers a fine-grained capability report across multiple
  dimensions (e.g., application, input type, output format, skill), allowing
  users to interact with and visualize model capabilities in depth. We evaluate a
  wide variety of frontier vision-language models on MEGA-Bench to understand
  their capabilities across these dimensions.