title: TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
summary: We introduce a novel framework that learns a dynamic neural radiance field
  (NeRF) for full-body talking humans from monocular videos. Prior work
  represents only the body pose or the face. However, humans communicate with
  their full body, combining body pose, hand gestures, as well as facial
  expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network
  that represents the holistic 4D human motion. Given a monocular video of a
  subject, we learn corresponding modules for the body, face, and hands, that are
  combined together to generate the final result. To capture complex finger
  articulation, we learn an additional deformation field for the hands. Our
  multi-identity representation enables simultaneous training for multiple
  subjects, as well as robust animation under completely unseen poses. It can
  also generalize to novel identities, given only a short video as input. We
  demonstrate state-of-the-art performance for animating full-body talking
  humans, with fine-grained hand articulation and facial expressions.
paper_authors._id: 66f4c08a279ccb2be378d6c6
paper_authors.name: Aggelina Chatziagapi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T10:31:10.377Z
paper_authors._id: 66ef430a133a4761fa261e78
paper_authors.avatarUrl: /avatars/fecf5806ff97d86688e310004a04cf96.svg
paper_authors.fullname: Aggelina Chatziagapi
paper_authors.type: user
paper_authors.user: aggelina
paper_authors._id: 66f4c08a279ccb2be378d6c7
paper_authors.name: Bindita Chaudhuri
paper_authors._id: 66f4c08a279ccb2be378d6c8
paper_authors.name: Amit Kumar
paper_authors._id: 66f4c08a279ccb2be378d6c9
paper_authors.name: Rakesh Ranjan
paper_authors._id: 66f4c08a279ccb2be378d6ca
paper_authors.name: Dimitris Samaras
paper_authors._id: 66f4c08a279ccb2be378d6cb
paper_authors.name: Nikolaos Sarafianos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T10:33:08.169Z
paper_authors._id: 66999bab3197305664859a8e
paper_authors.avatarUrl: /avatars/199f92ccdf0397c13bfbe69595bcb8e1.svg
paper_authors.fullname: Nikolaos Sarafianos
paper_authors.type: user
paper_authors.user: nsarafianos
paper_title: TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans
paper_summary: We introduce a novel framework that learns a dynamic neural radiance field
  (NeRF) for full-body talking humans from monocular videos. Prior work
  represents only the body pose or the face. However, humans communicate with
  their full body, combining body pose, hand gestures, as well as facial
  expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network
  that represents the holistic 4D human motion. Given a monocular video of a
  subject, we learn corresponding modules for the body, face, and hands, that are
  combined together to generate the final result. To capture complex finger
  articulation, we learn an additional deformation field for the hands. Our
  multi-identity representation enables simultaneous training for multiple
  subjects, as well as robust animation under completely unseen poses. It can
  also generalize to novel identities, given only a short video as input. We
  demonstrate state-of-the-art performance for animating full-body talking
  humans, with fine-grained hand articulation and facial expressions.
paper_ai_summary: TalkinNeRF is a novel NeRF-based framework that captures full-body human motion and communicates information through body pose, hand gestures, and facial expressions using a unified model.
paper_ai_keywords: dynamic neural radiance field NeRF full-body talking humans monocular videos body pose hand gestures facial expressions unified NeRF-based network holistic 4D human motion deformation field multi-identity representation