title: Synthesizing Moving People with 3D Control
summary: In this paper, we present a diffusion model-based framework for animating
  people from a single image for a given target 3D motion sequence. Our approach
  has two core components: a) learning priors about invisible parts of the human
  body and clothing, and b) rendering novel body poses with proper clothing and
  texture. For the first part, we learn an in-filling diffusion model to
  hallucinate unseen parts of a person given a single image. We train this model
  on texture map space, which makes it more sample-efficient since it is
  invariant to pose and viewpoint. Second, we develop a diffusion-based rendering
  pipeline, which is controlled by 3D human poses. This produces realistic
  renderings of novel poses of the person, including clothing, hair, and
  plausible in-filling of unseen regions. This disentangled approach allows our
  method to generate a sequence of images that are faithful to the target motion
  in the 3D pose and, to the input image in terms of visual similarity. In
  addition to that, the 3D control allows various synthetic camera trajectories
  to render a person. Our experiments show that our method is resilient in
  generating prolonged motions and varied challenging and complex poses compared
  to prior methods. Please check our website for more details:
  https://boyiliee.github.io/3DHM.github.io/.
paper_authors._id: 65ade45aa8f716b32e027084
paper_authors.name: Boyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T13:11:55.391Z
paper_authors._id: 620dd3888528f797e88cb9b5
paper_authors.avatarUrl: /avatars/af04728788d78fe7d6375e19e32a535e.svg
paper_authors.fullname: Boyi Li
paper_authors.type: user
paper_authors.user: Boyiliee
paper_authors._id: 65ade45aa8f716b32e027085
paper_authors.name: Jathushan Rajasegaran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T13:12:02.287Z
paper_authors._id: 63895b3a43d8b0797a0d0406
paper_authors.avatarUrl: /avatars/04d9952dff70f85d4e481ec80ae818cb.svg
paper_authors.fullname: Jathushan Rajasegaran
paper_authors.type: user
paper_authors.user: brjathu
paper_authors._id: 65ade45aa8f716b32e027086
paper_authors.name: Yossi Gandelsman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:28:47.504Z
paper_authors._id: 62f6942205ca68c0e0ff7b97
paper_authors.avatarUrl: /avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
paper_authors.fullname: Yossi Gandelsman
paper_authors.type: user
paper_authors.user: yossig
paper_authors._id: 65ade45aa8f716b32e027087
paper_authors.name: Alexei A. Efros
paper_authors._id: 65ade45aa8f716b32e027088
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T13:12:15.276Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_title: Synthesizing Moving People with 3D Control
paper_summary: In this paper, we present a diffusion model-based framework for animating
  people from a single image for a given target 3D motion sequence. Our approach
  has two core components: a) learning priors about invisible parts of the human
  body and clothing, and b) rendering novel body poses with proper clothing and
  texture. For the first part, we learn an in-filling diffusion model to
  hallucinate unseen parts of a person given a single image. We train this model
  on texture map space, which makes it more sample-efficient since it is
  invariant to pose and viewpoint. Second, we develop a diffusion-based rendering
  pipeline, which is controlled by 3D human poses. This produces realistic
  renderings of novel poses of the person, including clothing, hair, and
  plausible in-filling of unseen regions. This disentangled approach allows our
  method to generate a sequence of images that are faithful to the target motion
  in the 3D pose and, to the input image in terms of visual similarity. In
  addition to that, the 3D control allows various synthetic camera trajectories
  to render a person. Our experiments show that our method is resilient in
  generating prolonged motions and varied challenging and complex poses compared
  to prior methods. Please check our website for more details:
  https://boyiliee.github.io/3DHM.github.io/.
paper_ai_summary: A diffusion model-based framework animates people from a single image using learned priors for unseen parts and a diffusion-based rendering pipeline controlled by 3D human poses.
paper_ai_keywords: diffusion model in-filling diffusion model texture map space diffusion-based rendering pipeline 3D human poses realistic renderings visual similarity synthetic camera trajectories