JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding
  Benchmark for Culture-aware Evaluation
Accelerating research on Large Multimodal Models (LMMs) in non-English
languages is crucial for enhancing user experiences across broader populations.
In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale
Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the
Japanese cultural context. To facilitate comprehensive culture-aware
evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA)
subset, where the culture-independent subjects (e.g., Math) are selected and
translated into Japanese, enabling one-to-one comparison with its English
counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly
crafted subjects that reflect Japanese cultural context. Using the CA subset,
we observe performance drop in many LMMs when evaluated in Japanese, which is
purely attributable to language variation. Using the CS subset, we reveal their
inadequate Japanese cultural understanding. Further, by combining both subsets,
we identify that some LMMs perform well on the CA subset but not on the CS
subset, exposing a shallow understanding of the Japanese language that lacks
depth in cultural understanding. We hope this work will not only help advance
LMM performance in Japanese but also serve as a guideline to create
high-standard, culturally diverse benchmarks for multilingual LMM development.
The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
671887ca86e0b4dce87aedbd
Shota Onohara
admin_assigned
2024-10-23T08:01:15.561Z
64fa893f35a7fc7d4ff63321
https://cdn-avatars.huggingface.co/v1/production/uploads/64fa893f35a7fc7d4ff63321/Gibk2euroCPycBFmf5I_p.jpeg
Shota Onohara
user
shtapm
671887ca86e0b4dce87aedbe
Atsuyuki Miyai
claimed_verified
2024-10-23T07:33:10.955Z
6527b37c0ae663e384eb1b85
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png
Atsuyuki Miyai
user
AtsuMiyai
671887ca86e0b4dce87aedbf
Yuki Imajuku
claimed_verified
2024-10-28T11:25:16.473Z
65d86a59bdb95b4bbc744e10
https://cdn-avatars.huggingface.co/v1/production/uploads/65d86a59bdb95b4bbc744e10/TUYziiCbQr8JN0LZSxoTY.jpeg
Yuki Imajuku
user
yuki-imajuku
671887ca86e0b4dce87aedc0
Kazuki Egashira
671887ca86e0b4dce87aedc1
Jeonghun Baek
admin_assigned
2024-10-23T08:01:40.973Z
6523e5e8d14f5b11611eca95
/avatars/a3f11aed9531849ee5670a95190b006a.svg
Jeonghun Baek
user
ku21fan
671887ca86e0b4dce87aedc2
Xiang Yue
claimed_verified
2024-12-09T14:48:32.681Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
671887ca86e0b4dce87aedc3
Graham Neubig
admin_assigned
2024-10-23T08:01:31.738Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
671887ca86e0b4dce87aedc4
Kiyoharu Aizawa
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding
  Benchmark for Culture-aware Evaluation
Accelerating research on Large Multimodal Models (LMMs) in non-English
languages is crucial for enhancing user experiences across broader populations.
In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale
Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the
Japanese cultural context. To facilitate comprehensive culture-aware
evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA)
subset, where the culture-independent subjects (e.g., Math) are selected and
translated into Japanese, enabling one-to-one comparison with its English
counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly
crafted subjects that reflect Japanese cultural context. Using the CA subset,
we observe performance drop in many LMMs when evaluated in Japanese, which is
purely attributable to language variation. Using the CS subset, we reveal their
inadequate Japanese cultural understanding. Further, by combining both subsets,
we identify that some LMMs perform well on the CA subset but not on the CS
subset, exposing a shallow understanding of the Japanese language that lacks
depth in cultural understanding. We hope this work will not only help advance
LMM performance in Japanese but also serve as a guideline to create
high-standard, culturally diverse benchmarks for multilingual LMM development.
The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
JMMMU, a Japanese benchmark for Large Multimodal Models, evaluates their performance on expert-level tasks and cultural understanding through culture-agnostic and culture-specific subsets.
Large Multimodal Models JMMMU Japanese benchmark culture-agnostic culture-specific Japanese cultural context multilingual LMM expert-level tasks