title: JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding
    Benchmark for Culture-aware Evaluation
summary: Accelerating research on Large Multimodal Models (LMMs) in non-English
  languages is crucial for enhancing user experiences across broader populations.
  In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale
  Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the
  Japanese cultural context. To facilitate comprehensive culture-aware
  evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA)
  subset, where the culture-independent subjects (e.g., Math) are selected and
  translated into Japanese, enabling one-to-one comparison with its English
  counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly
  crafted subjects that reflect Japanese cultural context. Using the CA subset,
  we observe performance drop in many LMMs when evaluated in Japanese, which is
  purely attributable to language variation. Using the CS subset, we reveal their
  inadequate Japanese cultural understanding. Further, by combining both subsets,
  we identify that some LMMs perform well on the CA subset but not on the CS
  subset, exposing a shallow understanding of the Japanese language that lacks
  depth in cultural understanding. We hope this work will not only help advance
  LMM performance in Japanese but also serve as a guideline to create
  high-standard, culturally diverse benchmarks for multilingual LMM development.
  The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
paper_authors._id: 671887ca86e0b4dce87aedbd
paper_authors.name: Shota Onohara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:01:15.561Z
paper_authors._id: 64fa893f35a7fc7d4ff63321
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64fa893f35a7fc7d4ff63321/Gibk2euroCPycBFmf5I_p.jpeg
paper_authors.fullname: Shota Onohara
paper_authors.type: user
paper_authors.user: shtapm
paper_authors._id: 671887ca86e0b4dce87aedbe
paper_authors.name: Atsuyuki Miyai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:33:10.955Z
paper_authors._id: 6527b37c0ae663e384eb1b85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png
paper_authors.fullname: Atsuyuki Miyai
paper_authors.type: user
paper_authors.user: AtsuMiyai
paper_authors._id: 671887ca86e0b4dce87aedbf
paper_authors.name: Yuki Imajuku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:16.473Z
paper_authors._id: 65d86a59bdb95b4bbc744e10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d86a59bdb95b4bbc744e10/TUYziiCbQr8JN0LZSxoTY.jpeg
paper_authors.fullname: Yuki Imajuku
paper_authors.type: user
paper_authors.user: yuki-imajuku
paper_authors._id: 671887ca86e0b4dce87aedc0
paper_authors.name: Kazuki Egashira
paper_authors._id: 671887ca86e0b4dce87aedc1
paper_authors.name: Jeonghun Baek
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:01:40.973Z
paper_authors._id: 6523e5e8d14f5b11611eca95
paper_authors.avatarUrl: /avatars/a3f11aed9531849ee5670a95190b006a.svg
paper_authors.fullname: Jeonghun Baek
paper_authors.type: user
paper_authors.user: ku21fan
paper_authors._id: 671887ca86e0b4dce87aedc2
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:32.681Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 671887ca86e0b4dce87aedc3
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:01:31.738Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 671887ca86e0b4dce87aedc4
paper_authors.name: Kiyoharu Aizawa
paper_title: JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding
    Benchmark for Culture-aware Evaluation
paper_summary: Accelerating research on Large Multimodal Models (LMMs) in non-English
  languages is crucial for enhancing user experiences across broader populations.
  In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale
  Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the
  Japanese cultural context. To facilitate comprehensive culture-aware
  evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA)
  subset, where the culture-independent subjects (e.g., Math) are selected and
  translated into Japanese, enabling one-to-one comparison with its English
  counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly
  crafted subjects that reflect Japanese cultural context. Using the CA subset,
  we observe performance drop in many LMMs when evaluated in Japanese, which is
  purely attributable to language variation. Using the CS subset, we reveal their
  inadequate Japanese cultural understanding. Further, by combining both subsets,
  we identify that some LMMs perform well on the CA subset but not on the CS
  subset, exposing a shallow understanding of the Japanese language that lacks
  depth in cultural understanding. We hope this work will not only help advance
  LMM performance in Japanese but also serve as a guideline to create
  high-standard, culturally diverse benchmarks for multilingual LMM development.
  The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.