title: Can Large Language Models Help Multimodal Language Analysis? MMLA: A
    Comprehensive Benchmark
summary: Multimodal language analysis is a rapidly evolving field that leverages
  multiple modalities to enhance the understanding of high-level semantics
  underlying human conversational utterances. Despite its significance, little
  research has investigated the capability of multimodal large language models
  (MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce
  MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA
  comprises over 61K multimodal utterances drawn from both staged and real-world
  scenarios, covering six core dimensions of multimodal semantics: intent,
  emotion, dialogue act, sentiment, speaking style, and communication behavior.
  We evaluate eight mainstream branches of LLMs and MLLMs using three methods:
  zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive
  experiments reveal that even fine-tuned models achieve only about 60%~70%
  accuracy, underscoring the limitations of current MLLMs in understanding
  complex human language. We believe that MMLA will serve as a solid foundation
  for exploring the potential of large language models in multimodal language
  analysis and provide valuable resources to advance this field. The datasets and
  code are open-sourced at https://github.com/thuiar/MMLA.
paper_authors._id: 680c48805ec65044c2861a6a
paper_authors.name: Hanlei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-26T08:52:58.965Z
paper_authors._id: 669090c01e3f5b16ce22b535
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669090c01e3f5b16ce22b535/AT-k66Mt5FtbImnhNQJ_J.jpeg
paper_authors.fullname: Hanlei Zhang
paper_authors.type: user
paper_authors.user: HanleiZhang
paper_authors._id: 680c48805ec65044c2861a6b
paper_authors.name: Zhuohang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:33:19.643Z
paper_authors._id: 680ddad00a5976c000344004
paper_authors.avatarUrl: /avatars/82913c7239fdb56e5830d5ddabfa90c1.svg
paper_authors.fullname: Zhuohang Li
paper_authors.type: user
paper_authors.user: lzh0275
paper_authors._id: 680c48805ec65044c2861a6c
paper_authors.name: Yeshuang Zhu
paper_authors._id: 680c48805ec65044c2861a6d
paper_authors.name: Hua Xu
paper_authors._id: 680c48805ec65044c2861a6e
paper_authors.name: Peiwu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:33:16.395Z
paper_authors._id: 66c3f954780d735f176f9460
paper_authors.avatarUrl: /avatars/4b7319287718cb5978702429e8a398bb.svg
paper_authors.fullname: peiwuWang
paper_authors.type: user
paper_authors.user: TiansS
paper_authors._id: 680c48805ec65044c2861a6f
paper_authors.name: Haige Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:33:22.432Z
paper_authors._id: 66bd039f182d5a69ae84e135
paper_authors.avatarUrl: /avatars/543ecc2365049eb602893d0a337969ed.svg
paper_authors.fullname: Haige Zhu
paper_authors.type: user
paper_authors.user: Evanhimself
paper_authors._id: 680c48805ec65044c2861a70
paper_authors.name: Jie Zhou
paper_authors._id: 680c48805ec65044c2861a71
paper_authors.name: Jinchao Zhang
paper_title: Can Large Language Models Help Multimodal Language Analysis? MMLA: A
    Comprehensive Benchmark
paper_summary: Multimodal language analysis is a rapidly evolving field that leverages
  multiple modalities to enhance the understanding of high-level semantics
  underlying human conversational utterances. Despite its significance, little
  research has investigated the capability of multimodal large language models
  (MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce
  MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA
  comprises over 61K multimodal utterances drawn from both staged and real-world
  scenarios, covering six core dimensions of multimodal semantics: intent,
  emotion, dialogue act, sentiment, speaking style, and communication behavior.
  We evaluate eight mainstream branches of LLMs and MLLMs using three methods:
  zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive
  experiments reveal that even fine-tuned models achieve only about 60%~70%
  accuracy, underscoring the limitations of current MLLMs in understanding
  complex human language. We believe that MMLA will serve as a solid foundation
  for exploring the potential of large language models in multimodal language
  analysis and provide valuable resources to advance this field. The datasets and
  code are open-sourced at https://github.com/thuiar/MMLA.
paper_ai_summary: MMLA benchmark assesses multimodal large language models' understanding of human language semantics across various core dimensions, highlighting limitations and offering a foundation for future research.
paper_ai_keywords: multimodal language models benchmark multimodal semantics zero-shot inference supervised fine-tuning instruction tuning sentiment speaking style communication behavior