title: From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
summary: We present a framework for generating full-bodied photorealistic avatars that
  gesture according to the conversational dynamics of a dyadic interaction. Given
  speech audio, we output multiple possibilities of gestural motion for an
  individual, including face, body, and hands. The key behind our method is in
  combining the benefits of sample diversity from vector quantization with the
  high-frequency details obtained through diffusion to generate more dynamic,
  expressive motion. We visualize the generated motion using highly
  photorealistic avatars that can express crucial nuances in gestures (e.g.
  sneers and smirks). To facilitate this line of research, we introduce a
  first-of-its-kind multi-view conversational dataset that allows for
  photorealistic reconstruction. Experiments show our model generates appropriate
  and diverse gestures, outperforming both diffusion- and VQ-only methods.
  Furthermore, our perceptual evaluation highlights the importance of
  photorealism (vs. meshes) in accurately assessing subtle motion details in
  conversational gestures. Code and dataset available online.
paper_authors._id: 659620bab6b4c4e5049410e8
paper_authors.name: Evonne Ng
paper_authors._id: 659620bab6b4c4e5049410e9
paper_authors.name: Javier Romero
paper_authors._id: 659620bab6b4c4e5049410ea
paper_authors.name: Timur Bagautdinov
paper_authors._id: 659620bab6b4c4e5049410eb
paper_authors.name: Shaojie Bai
paper_authors._id: 659620bab6b4c4e5049410ec
paper_authors.name: Trevor Darrell
paper_authors._id: 659620bab6b4c4e5049410ed
paper_authors.name: Angjoo Kanazawa
paper_authors._id: 659620bab6b4c4e5049410ee
paper_authors.name: Alexander Richard
paper_title: From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
paper_summary: We present a framework for generating full-bodied photorealistic avatars that
  gesture according to the conversational dynamics of a dyadic interaction. Given
  speech audio, we output multiple possibilities of gestural motion for an
  individual, including face, body, and hands. The key behind our method is in
  combining the benefits of sample diversity from vector quantization with the
  high-frequency details obtained through diffusion to generate more dynamic,
  expressive motion. We visualize the generated motion using highly
  photorealistic avatars that can express crucial nuances in gestures (e.g.
  sneers and smirks). To facilitate this line of research, we introduce a
  first-of-its-kind multi-view conversational dataset that allows for
  photorealistic reconstruction. Experiments show our model generates appropriate
  and diverse gestures, outperforming both diffusion- and VQ-only methods.
  Furthermore, our perceptual evaluation highlights the importance of
  photorealism (vs. meshes) in accurately assessing subtle motion details in
  conversational gestures. Code and dataset available online.
paper_ai_summary: A framework combines vector quantization and diffusion models to generate diverse, dynamic, and photorealistic avatars responding to speech in a conversational setting using a novel multi-view dataset.
paper_ai_keywords: vector quantization diffusion photorealistic avatars gestural motion face body hands multi-view conversational dataset perceptual evaluation