title: CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
    Benchmark
summary: As the capabilities of large multimodal models (LMMs) continue to advance,
  evaluating the performance of LMMs emerges as an increasing need. Additionally,
  there is an even larger gap in evaluating the advanced knowledge and reasoning
  abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
  a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
  designed to evaluate LMMs on tasks demanding college-level subject knowledge
  and deliberate reasoning in a Chinese context. CMMMU is inspired by and
  strictly follows the annotation and analysis pattern of MMMU.
    CMMMU includes 12k manually collected multimodal questions from college
  exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
  Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
  Engineering, like its companion, MMMU. These questions span 30 subjects and
  comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
  tables, music sheets, and chemical structures.
    CMMMU focuses on complex perception and reasoning with domain-specific
  knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
  proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
  indicating a large space for improvement. CMMMU will boost the community to
  build the next-generation LMMs towards expert artificial intelligence and
  promote the democratization of LMMs by providing diverse language contexts.
paper_authors._id: 65af321777381188bd7548fd
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T09:29:10.355Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65af321777381188bd7548fe
paper_authors.name: Xinrun Du
paper_authors._id: 65af321777381188bd7548ff
paper_authors.name: Bei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T13:49:00.403Z
paper_authors._id: 64cc65d786d8dc0caa6ab3cd
paper_authors.avatarUrl: /avatars/23dc126f7332825e713807fb1c000ab6.svg
paper_authors.fullname: Bei Chen
paper_authors.type: user
paper_authors.user: BellaBei
paper_authors._id: 65af321777381188bd754900
paper_authors.name: Yiming Liang
paper_authors._id: 65af321777381188bd754901
paper_authors.name: Tongxu Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T08:49:33.842Z
paper_authors._id: 6421b07e918f0fd889f0a682
paper_authors.avatarUrl: /avatars/314b55c2428426c846d9449f98db4355.svg
paper_authors.fullname: Tongxu Luo
paper_authors.type: user
paper_authors.user: Zeno-Luo
paper_authors._id: 65af321777381188bd754902
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-23T16:36:09.819Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 65af321777381188bd754903
paper_authors.name: Kang Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:55:24.980Z
paper_authors._id: 6578265ddea7e2122d02f6ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
paper_authors.fullname: kang zhu
paper_authors.type: user
paper_authors.user: kangz
paper_authors._id: 65af321777381188bd754904
paper_authors.name: Yuyang Cheng
paper_authors._id: 65af321777381188bd754905
paper_authors.name: Chunpu Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:27:34.925Z
paper_authors._id: 6580333b64bab467b9bcb615
paper_authors.avatarUrl: /avatars/753afae2fefc748791550c733aa2c80b.svg
paper_authors.fullname: Chunpu Xu
paper_authors.type: user
paper_authors.user: cpxu
paper_authors._id: 65af321777381188bd754906
paper_authors.name: Shuyue Guo
paper_authors._id: 65af321777381188bd754907
paper_authors.name: Haoran Zhang
paper_authors._id: 65af321777381188bd754908
paper_authors.name: Xingwei Qu
paper_authors._id: 65af321777381188bd754909
paper_authors.name: Junjie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:09:03.576Z
paper_authors._id: 62579c55b98dcaa7e0de285d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
paper_authors.fullname: wangjunjie
paper_authors.type: user
paper_authors.user: wanng
paper_authors._id: 65af321777381188bd75490a
paper_authors.name: Ruibin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:29:41.058Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 65af321777381188bd75490b
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:54:24.108Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 65af321777381188bd75490c
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:32:59.886Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 65af321777381188bd75490d
paper_authors.name: Yudong Liu
paper_authors._id: 65af321777381188bd75490e
paper_authors.name: Yu-Hsuan Tsai
paper_authors._id: 65af321777381188bd75490f
paper_authors.name: Fengji Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:30:52.592Z
paper_authors._id: 64c8d4ac120a85440bfad55d
paper_authors.avatarUrl: /avatars/d65334288d69700250de51d0df344d9b.svg
paper_authors.fullname: Fengji Zhang
paper_authors.type: user
paper_authors.user: zfj1998
paper_authors._id: 65af321777381188bd754910
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:30:59.767Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 65af321777381188bd754911
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:32:10.735Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 65af321777381188bd754912
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:31:14.767Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 65af321777381188bd754913
paper_authors.name: Jie Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T08:32:02.954Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding
    Benchmark
paper_summary: As the capabilities of large multimodal models (LMMs) continue to advance,
  evaluating the performance of LMMs emerges as an increasing need. Additionally,
  there is an even larger gap in evaluating the advanced knowledge and reasoning
  abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU,
  a new Chinese Massive Multi-discipline Multimodal Understanding benchmark
  designed to evaluate LMMs on tasks demanding college-level subject knowledge
  and deliberate reasoning in a Chinese context. CMMMU is inspired by and
  strictly follows the annotation and analysis pattern of MMMU.
    CMMMU includes 12k manually collected multimodal questions from college
  exams, quizzes, and textbooks, covering six core disciplines: Art & Design,
  Business, Science, Health & Medicine, Humanities & Social Science, and Tech &
  Engineering, like its companion, MMMU. These questions span 30 subjects and
  comprise 39 highly heterogeneous image types, such as charts, diagrams, maps,
  tables, music sheets, and chemical structures.
    CMMMU focuses on complex perception and reasoning with domain-specific
  knowledge in the Chinese context. We evaluate 11 open-source LLMs and one
  proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%,
  indicating a large space for improvement. CMMMU will boost the community to
  build the next-generation LMMs towards expert artificial intelligence and
  promote the democratization of LMMs by providing diverse language contexts.
paper_ai_summary: CMMMU evaluates large multimodal models in college-level Chinese tasks across six disciplines, highlighting gaps and promoting expert-level artificial intelligence development.
paper_ai_keywords: large multimodal models CMMMU MMMU complex perception reasoning domain-specific knowledge expert artificial intelligence