title: MagicDance: Realistic Human Dance Video Generation with Motions & Facial
    Expressions Transfer
summary: In this work, we propose MagicDance, a diffusion-based model for 2D human
  motion and facial expression transfer on challenging human dance videos.
  Specifically, we aim to generate human dance videos of any target identity
  driven by novel pose sequences while keeping the identity unchanged. To this
  end, we propose a two-stage training strategy to disentangle human motions and
  appearance (e.g., facial expressions, skin tone and dressing), consisting of
  the pretraining of an appearance-control block and fine-tuning of an
  appearance-pose-joint-control block over human dance poses of the same dataset.
  Our novel design enables robust appearance control with temporally consistent
  upper body, facial attributes, and even background. The model also generalizes
  well on unseen human identities and complex motion sequences without the need
  for any fine-tuning with additional data with diverse human attributes by
  leveraging the prior knowledge of image diffusion models. Moreover, the
  proposed model is easy to use and can be considered as a plug-in
  module/extension to Stable Diffusion. We also demonstrate the model's ability
  for zero-shot 2D animation generation, enabling not only the appearance
  transfer from one identity to another but also allowing for cartoon-like
  stylization given only pose inputs. Extensive experiments demonstrate our
  superior performance on the TikTok dataset.
paper_authors._id: 655d825272ea9f3a3b829f20
paper_authors.name: Di Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:00:30.654Z
paper_authors._id: 64a5d8219f3b568c202b3137
paper_authors.avatarUrl: /avatars/eef6fb7c70d272555a53183c0e50dbaf.svg
paper_authors.fullname: Di Chang
paper_authors.type: user
paper_authors.user: Boese0601
paper_authors._id: 655d825272ea9f3a3b829f21
paper_authors.name: Yichun Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:00:41.562Z
paper_authors._id: 6309ad81b105f8675bd5a796
paper_authors.avatarUrl: /avatars/01c2186924486da4606e128e83709164.svg
paper_authors.fullname: Shi
paper_authors.type: user
paper_authors.user: Yichun
paper_authors._id: 655d825272ea9f3a3b829f22
paper_authors.name: Quankai Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:00:48.943Z
paper_authors._id: 64c12bb1f149752445c39472
paper_authors.avatarUrl: /avatars/8a534f5b850065f72dbe2e036181f7a9.svg
paper_authors.fullname: Quankai Gao
paper_authors.type: user
paper_authors.user: Quankai
paper_authors._id: 655d825272ea9f3a3b829f23
paper_authors.name: Jessica Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:00:56.246Z
paper_authors._id: 64d042822f1f9578a0324917
paper_authors.avatarUrl: /avatars/fea77fe4c09f38dfb649e445dc047370.svg
paper_authors.fullname: Jessica Fu
paper_authors.type: user
paper_authors.user: fujessc
paper_authors._id: 655d825272ea9f3a3b829f24
paper_authors.name: Hongyi Xu
paper_authors._id: 655d825272ea9f3a3b829f25
paper_authors.name: Guoxian Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:01:36.966Z
paper_authors._id: 63086a237dc1b1a54cc6c24d
paper_authors.avatarUrl: /avatars/477b94134edc4c18c8f769ecbb7d8091.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: guoxiansong
paper_authors._id: 655d825272ea9f3a3b829f26
paper_authors.name: Qing Yan
paper_authors._id: 655d825272ea9f3a3b829f27
paper_authors.name: Xiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:42:33.352Z
paper_authors._id: 62b3d2490385d6048985b42b
paper_authors.avatarUrl: /avatars/1c4e534d086d53ef1057fbc00334350a.svg
paper_authors.fullname: Xiao Yang
paper_authors.type: user
paper_authors.user: seravee008
paper_authors._id: 655d825272ea9f3a3b829f28
paper_authors.name: Mohammad Soleymani
paper_title: MagicDance: Realistic Human Dance Video Generation with Motions & Facial
    Expressions Transfer
paper_summary: In this work, we propose MagicDance, a diffusion-based model for 2D human
  motion and facial expression transfer on challenging human dance videos.
  Specifically, we aim to generate human dance videos of any target identity
  driven by novel pose sequences while keeping the identity unchanged. To this
  end, we propose a two-stage training strategy to disentangle human motions and
  appearance (e.g., facial expressions, skin tone and dressing), consisting of
  the pretraining of an appearance-control block and fine-tuning of an
  appearance-pose-joint-control block over human dance poses of the same dataset.
  Our novel design enables robust appearance control with temporally consistent
  upper body, facial attributes, and even background. The model also generalizes
  well on unseen human identities and complex motion sequences without the need
  for any fine-tuning with additional data with diverse human attributes by
  leveraging the prior knowledge of image diffusion models. Moreover, the
  proposed model is easy to use and can be considered as a plug-in
  module/extension to Stable Diffusion. We also demonstrate the model's ability
  for zero-shot 2D animation generation, enabling not only the appearance
  transfer from one identity to another but also allowing for cartoon-like
  stylization given only pose inputs. Extensive experiments demonstrate our
  superior performance on the TikTok dataset.
paper_ai_keywords: diffusion-based model appearance-control block appearance-pose-joint-control block temporally consistent image diffusion models Stable Diffusion zero-shot 2D animation generation