title: VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
summary: We propose VLOGGER, a method for audio-driven human video generation from a
  single input image of a person, which builds on the success of recent
  generative diffusion models. Our method consists of 1) a stochastic
  human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture
  that augments text-to-image models with both spatial and temporal controls.
  This supports the generation of high quality video of variable length, easily
  controllable through high-level representations of human faces and bodies. In
  contrast to previous work, our method does not require training for each
  person, does not rely on face detection and cropping, generates the complete
  image (not just the face or the lips), and considers a broad spectrum of
  scenarios (e.g. visible torso or diverse subject identities) that are critical
  to correctly synthesize humans who communicate. We also curate MENTOR, a new
  and diverse dataset with 3d pose and expression annotations, one order of
  magnitude larger than previous ones (800,000 identities) and with dynamic
  gestures, on which we train and ablate our main technical contributions.
    VLOGGER outperforms state-of-the-art methods in three public benchmarks,
  considering image quality, identity preservation and temporal consistency while
  also generating upper-body gestures. We analyze the performance of VLOGGER with
  respect to multiple diversity metrics, showing that our architectural choices
  and the use of MENTOR benefit training a fair and unbiased model at scale.
  Finally we show applications in video editing and personalization.
paper_authors._id: 65f26b48b06ca1ff489e81fd
paper_authors.name: Enric Corona
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:05:47.063Z
paper_authors._id: 64a4fe7a80d9ac0155ae911a
paper_authors.avatarUrl: /avatars/25c96e0ab25eeb183ac278b2f8ff6262.svg
paper_authors.fullname: Enric Corona
paper_authors.type: user
paper_authors.user: enriccorona
paper_authors._id: 65f26b48b06ca1ff489e81fe
paper_authors.name: Andrei Zanfir
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:05:53.300Z
paper_authors._id: 648cd24dab8e3024f5bbe93b
paper_authors.avatarUrl: /avatars/c8bd4b51155cd37fb7e76c97b843d461.svg
paper_authors.fullname: Andrei Zanfir
paper_authors.type: user
paper_authors.user: Andreiz
paper_authors._id: 65f26b48b06ca1ff489e81ff
paper_authors.name: Eduard Gabriel Bazavan
paper_authors._id: 65f26b48b06ca1ff489e8200
paper_authors.name: Nikos Kolotouros
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:06:08.779Z
paper_authors._id: 648cd29f8f46d7284544083b
paper_authors.avatarUrl: /avatars/53163398f1a629b9838548b808af53f7.svg
paper_authors.fullname: Nikos Kolotouros
paper_authors.type: user
paper_authors.user: kolotouros
paper_authors._id: 65f26b48b06ca1ff489e8201
paper_authors.name: Thiemo Alldieck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:06:14.770Z
paper_authors._id: 648c964d39d2584ee47af19c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c964d39d2584ee47af19c/5UEkzDTMY8I3svKjR7kxN.jpeg
paper_authors.fullname: Thiemo Alldieck
paper_authors.type: user
paper_authors.user: thiemoall
paper_authors._id: 65f26b48b06ca1ff489e8202
paper_authors.name: Cristian Sminchisescu
paper_title: VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
paper_summary: We propose VLOGGER, a method for audio-driven human video generation from a
  single input image of a person, which builds on the success of recent
  generative diffusion models. Our method consists of 1) a stochastic
  human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture
  that augments text-to-image models with both spatial and temporal controls.
  This supports the generation of high quality video of variable length, easily
  controllable through high-level representations of human faces and bodies. In
  contrast to previous work, our method does not require training for each
  person, does not rely on face detection and cropping, generates the complete
  image (not just the face or the lips), and considers a broad spectrum of
  scenarios (e.g. visible torso or diverse subject identities) that are critical
  to correctly synthesize humans who communicate. We also curate MENTOR, a new
  and diverse dataset with 3d pose and expression annotations, one order of
  magnitude larger than previous ones (800,000 identities) and with dynamic
  gestures, on which we train and ablate our main technical contributions.
    VLOGGER outperforms state-of-the-art methods in three public benchmarks,
  considering image quality, identity preservation and temporal consistency while
  also generating upper-body gestures. We analyze the performance of VLOGGER with
  respect to multiple diversity metrics, showing that our architectural choices
  and the use of MENTOR benefit training a fair and unbiased model at scale.
  Finally we show applications in video editing and personalization.
paper_ai_summary: VLOGGER generates audio-driven human videos from a single image using a diffusion-based method that includes 3D motion and text-to-image models, outperforming existing methods in quality, identity, and consistency.
paper_ai_keywords: stochastic human-to-3d-motion diffusion model diffusion-based architecture text-to-image models spatial controls temporal controls VLOGGER MENTOR 3d pose expression annotations image quality identity preservation temporal consistency upper-body gestures diversity metrics fair model unbiased model video editing personalization