title: Champ: Controllable and Consistent Human Image Animation with 3D
    Parametric Guidance
summary: In this study, we introduce a methodology for human image animation by
  leveraging a 3D human parametric model within a latent diffusion framework to
  enhance shape alignment and motion guidance in curernt human generative
  techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear)
  model as the 3D human parametric model to establish a unified representation of
  body shape and pose. This facilitates the accurate capture of intricate human
  geometry and motion characteristics from source videos. Specifically, we
  incorporate rendered depth images, normal maps, and semantic maps obtained from
  SMPL sequences, alongside skeleton-based motion guidance, to enrich the
  conditions to the latent diffusion model with comprehensive 3D shape and
  detailed pose attributes. A multi-layer motion fusion module, integrating
  self-attention mechanisms, is employed to fuse the shape and motion latent
  representations in the spatial domain. By representing the 3D human parametric
  model as the motion guidance, we can perform parametric shape alignment of the
  human body between the reference image and the source video motion.
  Experimental evaluations conducted on benchmark datasets demonstrate the
  methodology's superior ability to generate high-quality human animations that
  accurately capture both pose and shape variations. Furthermore, our approach
  also exhibits superior generalization capabilities on the proposed wild
  dataset. Project page: https://fudan-generative-vision.github.io/champ.
paper_authors._id: 660107379e1cf5eb41e51cfd
paper_authors.name: Shenhao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:16:18.584Z
paper_authors._id: 660022fd7a3a9bbfcd4f27ee
paper_authors.avatarUrl: /avatars/6d6472a52c6d5ddc7d9a8df62d70e567.svg
paper_authors.fullname: Shenhao Zhu
paper_authors.type: user
paper_authors.user: ShenhaoZhu
paper_authors._id: 660107379e1cf5eb41e51cfe
paper_authors.name: Junming Leo Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-25T05:20:33.398Z
paper_authors._id: 65d802be7eed9113f6f3e20b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d802be7eed9113f6f3e20b/TdjqI_GBn3Bc9bvxgqnDF.jpeg
paper_authors.fullname: Junming Leo Chen
paper_authors.type: user
paper_authors.user: Leooo333
paper_authors._id: 660107379e1cf5eb41e51cff
paper_authors.name: Zuozhuo Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:16:30.544Z
paper_authors._id: 6316fe2a6d322c52490f90e8
paper_authors.avatarUrl: /avatars/8c896b3733e183657009a426834a9dab.svg
paper_authors.fullname: Dai Zuozhuo
paper_authors.type: user
paper_authors.user: zyand
paper_authors._id: 660107379e1cf5eb41e51d00
paper_authors.name: Yinghui Xu
paper_authors._id: 660107379e1cf5eb41e51d01
paper_authors.name: Xun Cao
paper_authors._id: 660107379e1cf5eb41e51d02
paper_authors.name: Yao Yao
paper_authors._id: 660107379e1cf5eb41e51d03
paper_authors.name: Hao Zhu
paper_authors._id: 660107379e1cf5eb41e51d04
paper_authors.name: Siyu Zhu
paper_title: Champ: Controllable and Consistent Human Image Animation with 3D
    Parametric Guidance
paper_summary: In this study, we introduce a methodology for human image animation by
  leveraging a 3D human parametric model within a latent diffusion framework to
  enhance shape alignment and motion guidance in curernt human generative
  techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear)
  model as the 3D human parametric model to establish a unified representation of
  body shape and pose. This facilitates the accurate capture of intricate human
  geometry and motion characteristics from source videos. Specifically, we
  incorporate rendered depth images, normal maps, and semantic maps obtained from
  SMPL sequences, alongside skeleton-based motion guidance, to enrich the
  conditions to the latent diffusion model with comprehensive 3D shape and
  detailed pose attributes. A multi-layer motion fusion module, integrating
  self-attention mechanisms, is employed to fuse the shape and motion latent
  representations in the spatial domain. By representing the 3D human parametric
  model as the motion guidance, we can perform parametric shape alignment of the
  human body between the reference image and the source video motion.
  Experimental evaluations conducted on benchmark datasets demonstrate the
  methodology's superior ability to generate high-quality human animations that
  accurately capture both pose and shape variations. Furthermore, our approach
  also exhibits superior generalization capabilities on the proposed wild
  dataset. Project page: https://fudan-generative-vision.github.io/champ.
paper_ai_keywords: 3D human parametric model latent diffusion framework SMPL (Skinned Multi-Person Linear) model shape alignment motion guidance rendered depth images normal maps semantic maps skeleton-based motion guidance multi-layer motion fusion module self-attention mechanisms shape latent representations motion latent representations spatial domain hash tables parametric shape alignment benchmark datasets high-quality human animations pose variations wild dataset