title: MotionLLM: Understanding Human Behaviors from Human Motions and Videos
summary: This study delves into the realm of multi-modality (i.e., video and motion
  modalities) human behavior understanding by leveraging the powerful
  capabilities of Large Language Models (LLMs). Diverging from recent LLMs
  designed for video-only or motion-only understanding, we argue that
  understanding human behavior necessitates joint modeling from both videos and
  motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
  and semantics effectively. In light of this, we present MotionLLM, a
  straightforward yet effective framework for human motion understanding,
  captioning, and reasoning. Specifically, MotionLLM adopts a unified
  video-motion training strategy that leverages the complementary advantages of
  existing coarse video-text data and fine-grained motion-text data to glean rich
  spatial-temporal insights. Furthermore, we collect a substantial dataset,
  MoVid, comprising diverse videos, motions, captions, and instructions.
  Additionally, we propose the MoVid-Bench, with carefully manual annotations,
  for better evaluation of human behavior understanding on video and motion.
  Extensive experiments show the superiority of MotionLLM in the caption,
  spatial-temporal comprehension, and reasoning ability.
paper_authors._id: 6659ef49b29989570b61e15d
paper_authors.name: Ling-Hao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:45:39.187Z
paper_authors._id: 63109a4d61cab0446e48c83b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63109a4d61cab0446e48c83b/JQlVkQp0ok586ND1GmB0w.png
paper_authors.fullname: Ling-Hao Chen
paper_authors.type: user
paper_authors.user: EvanTHU
paper_authors._id: 6659ef49b29989570b61e15e
paper_authors.name: Shunlin Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:45:45.873Z
paper_authors._id: 65378eba7f862897572020fc
paper_authors.avatarUrl: /avatars/5cbc4165ddf7c2c2ae886cd7fa7d1186.svg
paper_authors.fullname: Shunlin Lu
paper_authors.type: user
paper_authors.user: shunlinl
paper_authors._id: 6659ef49b29989570b61e15f
paper_authors.name: Ailing Zeng
paper_authors._id: 6659ef49b29989570b61e160
paper_authors.name: Hao Zhang
paper_authors._id: 6659ef49b29989570b61e161
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:46:33.967Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 6659ef49b29989570b61e162
paper_authors.name: Ruimao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:46:45.207Z
paper_authors._id: 65324848220f490dbaa3b8c9
paper_authors.avatarUrl: /avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Ruimao
paper_authors._id: 6659ef49b29989570b61e163
paper_authors.name: Lei Zhang
paper_title: MotionLLM: Understanding Human Behaviors from Human Motions and Videos
paper_summary: This study delves into the realm of multi-modality (i.e., video and motion
  modalities) human behavior understanding by leveraging the powerful
  capabilities of Large Language Models (LLMs). Diverging from recent LLMs
  designed for video-only or motion-only understanding, we argue that
  understanding human behavior necessitates joint modeling from both videos and
  motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
  and semantics effectively. In light of this, we present MotionLLM, a
  straightforward yet effective framework for human motion understanding,
  captioning, and reasoning. Specifically, MotionLLM adopts a unified
  video-motion training strategy that leverages the complementary advantages of
  existing coarse video-text data and fine-grained motion-text data to glean rich
  spatial-temporal insights. Furthermore, we collect a substantial dataset,
  MoVid, comprising diverse videos, motions, captions, and instructions.
  Additionally, we propose the MoVid-Bench, with carefully manual annotations,
  for better evaluation of human behavior understanding on video and motion.
  Extensive experiments show the superiority of MotionLLM in the caption,
  spatial-temporal comprehension, and reasoning ability.
paper_ai_keywords: multi-modality Large Language Models (LLMs) video-only motion-only joint modeling SMPL sequences body part dynamics semantics questionLLM MotionLLM unified video-motion training coarse video-text data fine-grained motion-text data spatial-temporal insights MoVid dataset diverse videos motions captions instructions MoVid-B manual annotations spatial-temporal comprehension reasoning (Note: There seems to be a typo here, should be "model") reasoning ability