title: MotionGPT: Human Motion as a Foreign Language
summary: Though the advancement of pre-trained large language models unfolds, the
  exploration of building a unified model for language and other multi-modal
  data, such as motion, remains challenging and untouched so far. Fortunately,
  human motion displays a semantic coupling akin to human language, often
  perceived as a form of body language. By fusing language data with large-scale
  motion models, motion-language pre-training that can enhance the performance of
  motion-related tasks becomes feasible. Driven by this insight, we propose
  MotionGPT, a unified, versatile, and user-friendly motion-language model to
  handle multiple motion-relevant tasks. Specifically, we employ the discrete
  vector quantization for human motion and transfer 3D motion into motion tokens,
  similar to the generation process of word tokens. Building upon this "motion
  vocabulary", we perform language modeling on both motion and text in a unified
  manner, treating human motion as a specific language. Moreover, inspired by
  prompt learning, we pre-train MotionGPT with a mixture of motion-language data
  and fine-tune it on prompt-based question-and-answer tasks. Extensive
  experiments demonstrate that MotionGPT achieves state-of-the-art performances
  on multiple motion tasks including text-driven motion generation, motion
  captioning, motion prediction, and motion in-between.
paper_authors._id: 649a519c23dc99ea5486e98a
paper_authors.name: Biao Jiang
paper_authors._id: 649a519c23dc99ea5486e98b
paper_authors.name: Xin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:34.646Z
paper_authors._id: 63b543b2bda8d44adf2d7d0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: ChrisChenXin
paper_authors._id: 649a519c23dc99ea5486e98c
paper_authors.name: Wen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:32:47.919Z
paper_authors._id: 63198e8802fb322037332f2d
paper_authors.avatarUrl: /avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
paper_authors.fullname: Wen Liu
paper_authors.type: user
paper_authors.user: doubility123
paper_authors._id: 649a519c23dc99ea5486e98d
paper_authors.name: Jingyi Yu
paper_authors._id: 649a519c23dc99ea5486e98e
paper_authors.name: Gang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T07:41:30.624Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_authors._id: 649a519c23dc99ea5486e98f
paper_authors.name: Tao Chen
paper_title: MotionGPT: Human Motion as a Foreign Language
paper_summary: Though the advancement of pre-trained large language models unfolds, the
  exploration of building a unified model for language and other multi-modal
  data, such as motion, remains challenging and untouched so far. Fortunately,
  human motion displays a semantic coupling akin to human language, often
  perceived as a form of body language. By fusing language data with large-scale
  motion models, motion-language pre-training that can enhance the performance of
  motion-related tasks becomes feasible. Driven by this insight, we propose
  MotionGPT, a unified, versatile, and user-friendly motion-language model to
  handle multiple motion-relevant tasks. Specifically, we employ the discrete
  vector quantization for human motion and transfer 3D motion into motion tokens,
  similar to the generation process of word tokens. Building upon this "motion
  vocabulary", we perform language modeling on both motion and text in a unified
  manner, treating human motion as a specific language. Moreover, inspired by
  prompt learning, we pre-train MotionGPT with a mixture of motion-language data
  and fine-tune it on prompt-based question-and-answer tasks. Extensive
  experiments demonstrate that MotionGPT achieves state-of-the-art performances
  on multiple motion tasks including text-driven motion generation, motion
  captioning, motion prediction, and motion in-between.
paper_ai_keywords: discrete vector quantization motion tokens language modeling prompt learning text-driven motion generation motion captioning motion prediction motion in-between