title: UniMuMo: Unified Text, Music and Motion Generation
summary: We introduce UniMuMo, a unified multimodal model capable of taking arbitrary
  text, music, and motion data as input conditions to generate outputs across all
  three modalities. To address the lack of time-synchronized data, we align
  unpaired music and motion data based on rhythmic patterns to leverage existing
  large-scale music-only and motion-only datasets. By converting music, motion,
  and text into token-based representation, our model bridges these modalities
  through a unified encoder-decoder transformer architecture. To support multiple
  generation tasks within a single framework, we introduce several architectural
  improvements. We propose encoding motion with a music codebook, mapping motion
  into the same feature space as music. We introduce a music-motion parallel
  generation scheme that unifies all music and motion generation tasks into a
  single transformer decoder architecture with a single training task of
  music-motion joint generation. Moreover, the model is designed by fine-tuning
  existing pre-trained single-modality models, significantly reducing
  computational demands. Extensive experiments demonstrate that UniMuMo achieves
  competitive results on all unidirectional generation benchmarks across music,
  motion, and text modalities. Quantitative results are available in the
  https://hanyangclarence.github.io/unimumo_demo/{project page}.
paper_authors._id: 6704b563755b74099845d572
paper_authors.name: Han Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T12:46:20.473Z
paper_authors._id: 63ef2578436d28737ee8925c
paper_authors.avatarUrl: /avatars/cf854d419673d55a6822b7e156dade88.svg
paper_authors.fullname: YH
paper_authors.type: user
paper_authors.user: ClarenceY
paper_authors._id: 6704b563755b74099845d573
paper_authors.name: Kun Su
paper_authors._id: 6704b563755b74099845d574
paper_authors.name: Yutong Zhang
paper_authors._id: 6704b563755b74099845d575
paper_authors.name: Jiaben Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:57:34.574Z
paper_authors._id: 64f788ea9980b96c33e2b8fe
paper_authors.avatarUrl: /avatars/b9fc9a93550df79033f86758fc8cbc5d.svg
paper_authors.fullname: Jiaben Chen
paper_authors.type: user
paper_authors.user: Jiaben
paper_authors._id: 6704b563755b74099845d576
paper_authors.name: Kaizhi Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:57:28.773Z
paper_authors._id: 664a316b658c482f96eba4c0
paper_authors.avatarUrl: /avatars/0447a713eec179e423613b33e668bf68.svg
paper_authors.fullname: Kaizhi Qian
paper_authors.type: user
paper_authors.user: auspicious3000
paper_authors._id: 6704b563755b74099845d577
paper_authors.name: Gaowen Liu
paper_authors._id: 6704b563755b74099845d578
paper_authors.name: Chuang Gan
paper_title: UniMuMo: Unified Text, Music and Motion Generation
paper_summary: We introduce UniMuMo, a unified multimodal model capable of taking arbitrary
  text, music, and motion data as input conditions to generate outputs across all
  three modalities. To address the lack of time-synchronized data, we align
  unpaired music and motion data based on rhythmic patterns to leverage existing
  large-scale music-only and motion-only datasets. By converting music, motion,
  and text into token-based representation, our model bridges these modalities
  through a unified encoder-decoder transformer architecture. To support multiple
  generation tasks within a single framework, we introduce several architectural
  improvements. We propose encoding motion with a music codebook, mapping motion
  into the same feature space as music. We introduce a music-motion parallel
  generation scheme that unifies all music and motion generation tasks into a
  single transformer decoder architecture with a single training task of
  music-motion joint generation. Moreover, the model is designed by fine-tuning
  existing pre-trained single-modality models, significantly reducing
  computational demands. Extensive experiments demonstrate that UniMuMo achieves
  competitive results on all unidirectional generation benchmarks across music,
  motion, and text modalities. Quantitative results are available in the
  https://hanyangclarence.github.io/unimumo_demo/{project page}.
paper_ai_summary: UniMuMo generates outputs across text, music, and motion modalities using a unified encoder-decoder transformer architecture and aligned unpaired data based on rhythmic patterns.
paper_ai_keywords: Unified multimodal model token-based representation encoder-decoder transformer architecture music codebook music-motion parallel generation single transformer decoder architecture music-motion joint generation pre-trained single-modality models