title: LLaVAction: evaluating and training multi-modal large language models
    for action recognition
summary: Understanding human behavior requires measuring behavioral actions. Due to
  its complexity, behavior is best mapped onto a rich, semantic structure such as
  language. The recent development of multi-modal large language models (MLLMs)
  is a promising candidate for a wide range of action understanding tasks. In
  this work, we focus on evaluating and then improving MLLMs to perform action
  recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most
  challenging egocentric action datasets, to the form of video multiple question
  answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult
  incorrect answers as distractors, leading MLLMs struggle to recognize the
  correct actions. We propose a series of methods that greatly improve the MLLMs'
  ability to perform action recognition, achieving state-of-the-art on both the
  EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points
  in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other
  action-related video benchmarks such as EgoSchema, PerceptionTest,
  LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising
  path forward for complex action tasks. Code and models are available at:
  https://github.com/AdaptiveMotorControlLab/LLaVAction.
paper_authors._id: 67e47a92d762ffface602862
paper_authors.name: Shaokai Ye
paper_authors._id: 67e47a92d762ffface602863
paper_authors.name: Haozhe Qi
paper_authors._id: 67e47a92d762ffface602864
paper_authors.name: Alexander Mathis
paper_authors._id: 67e47a92d762ffface602865
paper_authors.name: Mackenzie W. Mathis
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-26T22:13:22.189Z
paper_authors._id: 62c2e4d672ff397f721ac6f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656944910507-62c2e4d672ff397f721ac6f7.jpeg
paper_authors.fullname: Mackenzie Mathis
paper_authors.type: user
paper_authors.user: mwmathis
paper_title: LLaVAction: evaluating and training multi-modal large language models
    for action recognition
paper_summary: Understanding human behavior requires measuring behavioral actions. Due to
  its complexity, behavior is best mapped onto a rich, semantic structure such as
  language. The recent development of multi-modal large language models (MLLMs)
  is a promising candidate for a wide range of action understanding tasks. In
  this work, we focus on evaluating and then improving MLLMs to perform action
  recognition. We reformulate EPIC-KITCHENS-100, one of the largest and most
  challenging egocentric action datasets, to the form of video multiple question
  answering (EPIC-KITCHENS-100-MQA). We show that when we sample difficult
  incorrect answers as distractors, leading MLLMs struggle to recognize the
  correct actions. We propose a series of methods that greatly improve the MLLMs'
  ability to perform action recognition, achieving state-of-the-art on both the
  EPIC-KITCHENS-100 validation set, as well as outperforming GPT-4o by 21 points
  in accuracy on EPIC-KITCHENS-100-MQA. Lastly, we show improvements on other
  action-related video benchmarks such as EgoSchema, PerceptionTest,
  LongVideoBench, VideoMME and MVBench, suggesting that MLLMs are a promising
  path forward for complex action tasks. Code and models are available at:
  https://github.com/AdaptiveMotorControlLab/LLaVAction.
paper_ai_summary: Multi-modal large language models enhanced with question-answering methods improve action recognition accuracy on several benchmarks, including EPIC-KITCHENS-100 and others.
paper_ai_keywords: multi-modal large language models MLLMs EPIC-KITCHENS-100 video multiple question answering EPIC-KITCHENS-100-MQA GPT-4o EgoSchema PerceptionTest LongVideoBench VideoMME MVBench