Can Large Language Models Help Multimodal Language Analysis? MMLA: A
  Comprehensive Benchmark
Multimodal language analysis is a rapidly evolving field that leverages
multiple modalities to enhance the understanding of high-level semantics
underlying human conversational utterances. Despite its significance, little
research has investigated the capability of multimodal large language models
(MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce
MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA
comprises over 61K multimodal utterances drawn from both staged and real-world
scenarios, covering six core dimensions of multimodal semantics: intent,
emotion, dialogue act, sentiment, speaking style, and communication behavior.
We evaluate eight mainstream branches of LLMs and MLLMs using three methods:
zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive
experiments reveal that even fine-tuned models achieve only about 60%~70%
accuracy, underscoring the limitations of current MLLMs in understanding
complex human language. We believe that MMLA will serve as a solid foundation
for exploring the potential of large language models in multimodal language
analysis and provide valuable resources to advance this field. The datasets and
code are open-sourced at https://github.com/thuiar/MMLA.
680c48805ec65044c2861a6a
Hanlei Zhang
claimed_verified
2025-04-26T08:52:58.965Z
669090c01e3f5b16ce22b535
https://cdn-avatars.huggingface.co/v1/production/uploads/669090c01e3f5b16ce22b535/AT-k66Mt5FtbImnhNQJ_J.jpeg
Hanlei Zhang
user
HanleiZhang
680c48805ec65044c2861a6b
Zhuohang Li
claimed_verified
2025-04-28T12:33:19.643Z
680ddad00a5976c000344004
/avatars/82913c7239fdb56e5830d5ddabfa90c1.svg
Zhuohang Li
user
lzh0275
680c48805ec65044c2861a6c
Yeshuang Zhu
680c48805ec65044c2861a6d
Hua Xu
680c48805ec65044c2861a6e
Peiwu Wang
admin_assigned
2025-04-29T12:33:16.395Z
66c3f954780d735f176f9460
/avatars/4b7319287718cb5978702429e8a398bb.svg
peiwuWang
user
TiansS
680c48805ec65044c2861a6f
Haige Zhu
admin_assigned
2025-04-29T12:33:22.432Z
66bd039f182d5a69ae84e135
/avatars/543ecc2365049eb602893d0a337969ed.svg
Haige Zhu
user
Evanhimself
680c48805ec65044c2861a70
Jie Zhou
680c48805ec65044c2861a71
Jinchao Zhang
Can Large Language Models Help Multimodal Language Analysis? MMLA: A
  Comprehensive Benchmark
Multimodal language analysis is a rapidly evolving field that leverages
multiple modalities to enhance the understanding of high-level semantics
underlying human conversational utterances. Despite its significance, little
research has investigated the capability of multimodal large language models
(MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce
MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA
comprises over 61K multimodal utterances drawn from both staged and real-world
scenarios, covering six core dimensions of multimodal semantics: intent,
emotion, dialogue act, sentiment, speaking style, and communication behavior.
We evaluate eight mainstream branches of LLMs and MLLMs using three methods:
zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive
experiments reveal that even fine-tuned models achieve only about 60%~70%
accuracy, underscoring the limitations of current MLLMs in understanding
complex human language. We believe that MMLA will serve as a solid foundation
for exploring the potential of large language models in multimodal language
analysis and provide valuable resources to advance this field. The datasets and
code are open-sourced at https://github.com/thuiar/MMLA.
MMLA benchmark assesses multimodal large language models' understanding of human language semantics across various core dimensions, highlighting limitations and offering a foundation for future research.
multimodal language models benchmark multimodal semantics zero-shot inference supervised fine-tuning instruction tuning sentiment speaking style communication behavior