MotionGPT: Human Motion as a Foreign Language
Though the advancement of pre-trained large language models unfolds, the
exploration of building a unified model for language and other multi-modal
data, such as motion, remains challenging and untouched so far. Fortunately,
human motion displays a semantic coupling akin to human language, often
perceived as a form of body language. By fusing language data with large-scale
motion models, motion-language pre-training that can enhance the performance of
motion-related tasks becomes feasible. Driven by this insight, we propose
MotionGPT, a unified, versatile, and user-friendly motion-language model to
handle multiple motion-relevant tasks. Specifically, we employ the discrete
vector quantization for human motion and transfer 3D motion into motion tokens,
similar to the generation process of word tokens. Building upon this "motion
vocabulary", we perform language modeling on both motion and text in a unified
manner, treating human motion as a specific language. Moreover, inspired by
prompt learning, we pre-train MotionGPT with a mixture of motion-language data
and fine-tune it on prompt-based question-and-answer tasks. Extensive
experiments demonstrate that MotionGPT achieves state-of-the-art performances
on multiple motion tasks including text-driven motion generation, motion
captioning, motion prediction, and motion in-between.
649a519c23dc99ea5486e98a
Biao Jiang
649a519c23dc99ea5486e98b
Xin Chen
claimed_verified
2023-06-28T07:41:34.646Z
63b543b2bda8d44adf2d7d0a
https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
Chen
user
ChrisChenXin
649a519c23dc99ea5486e98c
Wen Liu
claimed_verified
2024-01-22T10:32:47.919Z
63198e8802fb322037332f2d
/avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
Wen Liu
user
doubility123
649a519c23dc99ea5486e98d
Jingyi Yu
649a519c23dc99ea5486e98e
Gang Yu
claimed_verified
2023-06-28T07:41:30.624Z
63417332c5565a4b8d43a0d8
https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
Gang Yu
user
skicy
649a519c23dc99ea5486e98f
Tao Chen
MotionGPT: Human Motion as a Foreign Language
Though the advancement of pre-trained large language models unfolds, the
exploration of building a unified model for language and other multi-modal
data, such as motion, remains challenging and untouched so far. Fortunately,
human motion displays a semantic coupling akin to human language, often
perceived as a form of body language. By fusing language data with large-scale
motion models, motion-language pre-training that can enhance the performance of
motion-related tasks becomes feasible. Driven by this insight, we propose
MotionGPT, a unified, versatile, and user-friendly motion-language model to
handle multiple motion-relevant tasks. Specifically, we employ the discrete
vector quantization for human motion and transfer 3D motion into motion tokens,
similar to the generation process of word tokens. Building upon this "motion
vocabulary", we perform language modeling on both motion and text in a unified
manner, treating human motion as a specific language. Moreover, inspired by
prompt learning, we pre-train MotionGPT with a mixture of motion-language data
and fine-tune it on prompt-based question-and-answer tasks. Extensive
experiments demonstrate that MotionGPT achieves state-of-the-art performances
on multiple motion tasks including text-driven motion generation, motion
captioning, motion prediction, and motion in-between.
MotionGPT, a unified motion-language model using discrete vector quantization, achieves top performance in various motion-related tasks by treating motion as a language similar to text.
discrete vector quantization motion tokens language modeling prompt learning text-driven motion generation motion captioning motion prediction motion in-between