title: X-Dancer: Expressive Music to Human Dance Video Generation
summary: We present X-Dancer, a novel zero-shot music-driven image animation pipeline
  that creates diverse and long-range lifelike human dance videos from a single
  static image. As its core, we introduce a unified transformer-diffusion
  framework, featuring an autoregressive transformer model that synthesize
  extended and music-synchronized token sequences for 2D body, head and hands
  poses, which then guide a diffusion model to produce coherent and realistic
  dance video frames. Unlike traditional methods that primarily generate human
  motion in 3D, X-Dancer addresses data limitations and enhances scalability by
  modeling a wide spectrum of 2D dance motions, capturing their nuanced alignment
  with musical beats through readily available monocular videos. To achieve this,
  we first build a spatially compositional token representation from 2D human
  pose labels associated with keypoint confidences, encoding both large
  articulated body movements (e.g., upper and lower body) and fine-grained
  motions (e.g., head and hands). We then design a music-to-motion transformer
  model that autoregressively generates music-aligned dance pose token sequences,
  incorporating global attention to both musical style and prior motion context.
  Finally we leverage a diffusion backbone to animate the reference image with
  these synthesized pose tokens through AdaIN, forming a fully differentiable
  end-to-end framework. Experimental results demonstrate that X-Dancer is able to
  produce both diverse and characterized dance videos, substantially
  outperforming state-of-the-art methods in term of diversity, expressiveness and
  realism. Code and model will be available for research purposes.
paper_authors._id: 67bd526001d5bfa0abfcc5ba
paper_authors.name: Zeyuan Chen
paper_authors._id: 67bd526001d5bfa0abfcc5bb
paper_authors.name: Hongyi Xu
paper_authors._id: 67bd526001d5bfa0abfcc5bc
paper_authors.name: Guoxian Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:33:13.695Z
paper_authors._id: 63086a237dc1b1a54cc6c24d
paper_authors.avatarUrl: /avatars/477b94134edc4c18c8f769ecbb7d8091.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: guoxiansong
paper_authors._id: 67bd526001d5bfa0abfcc5bd
paper_authors.name: You Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:33:21.430Z
paper_authors._id: 6408dfd4b6a334f53e24023c
paper_authors.avatarUrl: /avatars/b7e3fa4fbec6313e94ff3384b74dabfc.svg
paper_authors.fullname: You Xie
paper_authors.type: user
paper_authors.user: youxie
paper_authors._id: 67bd526001d5bfa0abfcc5be
paper_authors.name: Chenxu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:33:27.827Z
paper_authors._id: 64f58e61abc51e6b0f885575
paper_authors.avatarUrl: /avatars/53ec28d045e708570e0e34f44aaba7a7.svg
paper_authors.fullname: Chenxu Zhang
paper_authors.type: user
paper_authors.user: ChenxuZhang528
paper_authors._id: 67bd526001d5bfa0abfcc5bf
paper_authors.name: Xin Chen
paper_authors._id: 67bd526001d5bfa0abfcc5c0
paper_authors.name: Chao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:33:39.754Z
paper_authors._id: 64cf8caa0b71aea8be5c97db
paper_authors.avatarUrl: /avatars/d486466afa3b1d58abd85725930b9298.svg
paper_authors.fullname: Chao Wang
paper_authors.type: user
paper_authors.user: chaowang
paper_authors._id: 67bd526001d5bfa0abfcc5c1
paper_authors.name: Di Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:33:46.334Z
paper_authors._id: 64a5d8219f3b568c202b3137
paper_authors.avatarUrl: /avatars/eef6fb7c70d272555a53183c0e50dbaf.svg
paper_authors.fullname: Di Chang
paper_authors.type: user
paper_authors.user: Boese0601
paper_authors._id: 67bd526001d5bfa0abfcc5c2
paper_authors.name: Linjie Luo
paper_title: X-Dancer: Expressive Music to Human Dance Video Generation
paper_summary: We present X-Dancer, a novel zero-shot music-driven image animation pipeline
  that creates diverse and long-range lifelike human dance videos from a single
  static image. As its core, we introduce a unified transformer-diffusion
  framework, featuring an autoregressive transformer model that synthesize
  extended and music-synchronized token sequences for 2D body, head and hands
  poses, which then guide a diffusion model to produce coherent and realistic
  dance video frames. Unlike traditional methods that primarily generate human
  motion in 3D, X-Dancer addresses data limitations and enhances scalability by
  modeling a wide spectrum of 2D dance motions, capturing their nuanced alignment
  with musical beats through readily available monocular videos. To achieve this,
  we first build a spatially compositional token representation from 2D human
  pose labels associated with keypoint confidences, encoding both large
  articulated body movements (e.g., upper and lower body) and fine-grained
  motions (e.g., head and hands). We then design a music-to-motion transformer
  model that autoregressively generates music-aligned dance pose token sequences,
  incorporating global attention to both musical style and prior motion context.
  Finally we leverage a diffusion backbone to animate the reference image with
  these synthesized pose tokens through AdaIN, forming a fully differentiable
  end-to-end framework. Experimental results demonstrate that X-Dancer is able to
  produce both diverse and characterized dance videos, substantially
  outperforming state-of-the-art methods in term of diversity, expressiveness and
  realism. Code and model will be available for research purposes.
paper_ai_keywords: autoregressive transformer model diffusion model unified transformer-diffusion framework music-synchronized token sequences continuous transformer-diffusion framework monocular videos spatially compositional token representation keypoint confidences **Unified Transformer-Diffusion Framework** **Autoregressive Transformer Model** **Diffusion Model** **Music-Synchronized Token Sequences** **Continuous Transformer-Diffusion Framework** **Monocular Videos** **Spatially Compositional Token Representation** **Keypoint Confidences** unified transformer-diffusion framework autoregressive transformer model diffusion model music-synchronized token sequences continuous transformer-diffusion framework monocular videos spatially compositional token representation keypoint confidences