title: Controllable Dynamic Appearance for Neural 3D Portraits
summary: Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
  reconstruct and reanimate dynamic portrait scenes with control over head-pose,
  facial expressions and viewing direction. However, training such models assumes
  photometric consistency over the deformed region e.g. the face must be evenly
  lit as it deforms with changing head-pose and facial expression. Such
  photometric consistency across frames of a video is hard to maintain, even in
  studio environments, thus making the created reanimatable neural portraits
  prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
  system that enables the creation of fully controllable 3D portraits in
  real-world capture conditions. CoDyNeRF learns to approximate illumination
  dependent effects via a dynamic appearance model in the canonical space that is
  conditioned on predicted surface normals and the facial expressions and
  head-pose deformations. The surface normals prediction is guided using 3DMM
  normals that act as a coarse prior for the normals of the human head, where
  direct prediction of normals is hard due to rigid and non-rigid deformations
  induced by head-pose and facial expression changes. Using only a
  smartphone-captured short video of a subject for training, we demonstrate the
  effectiveness of our method on free view synthesis of a portrait scene with
  explicit head pose and expression controls, and realistic lighting effects. The
  project page can be found here:
  http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html
paper_authors._id: 650bd01d1f2949b99e65d252
paper_authors.name: ShahRukh Athar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-21T13:17:37.470Z
paper_authors._id: 650c3a2bd439bbbbadf40436
paper_authors.avatarUrl: /avatars/a5a7c04a03580ae59c28eb6f90d43b47.svg
paper_authors.fullname: ShahRukh Athar
paper_authors.type: user
paper_authors.user: Sathar
paper_authors._id: 650bd01d1f2949b99e65d253
paper_authors.name: Zhixin Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T10:10:44.639Z
paper_authors._id: 62a8efd508a7ea93ff18785a
paper_authors.avatarUrl: /avatars/ff259233d437833a304329bb973a5a04.svg
paper_authors.fullname: Zhixin Shu
paper_authors.type: user
paper_authors.user: zhixinshu
paper_authors._id: 650bd01d1f2949b99e65d254
paper_authors.name: Zexiang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:55:38.136Z
paper_authors._id: 65565b1b8616d8d740abb976
paper_authors.avatarUrl: /avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
paper_authors.fullname: Zexiang Xu
paper_authors.type: user
paper_authors.user: zexiangxu
paper_authors._id: 650bd01d1f2949b99e65d255
paper_authors.name: Fuji Luan
paper_authors._id: 650bd01d1f2949b99e65d256
paper_authors.name: Sai Bi
paper_authors._id: 650bd01d1f2949b99e65d257
paper_authors.name: Kalyan Sunkavalli
paper_authors._id: 650bd01d1f2949b99e65d258
paper_authors.name: Dimitris Samaras
paper_title: Controllable Dynamic Appearance for Neural 3D Portraits
paper_summary: Recent advances in Neural Radiance Fields (NeRFs) have made it possible to
  reconstruct and reanimate dynamic portrait scenes with control over head-pose,
  facial expressions and viewing direction. However, training such models assumes
  photometric consistency over the deformed region e.g. the face must be evenly
  lit as it deforms with changing head-pose and facial expression. Such
  photometric consistency across frames of a video is hard to maintain, even in
  studio environments, thus making the created reanimatable neural portraits
  prone to artifacts during reanimation. In this work, we propose CoDyNeRF, a
  system that enables the creation of fully controllable 3D portraits in
  real-world capture conditions. CoDyNeRF learns to approximate illumination
  dependent effects via a dynamic appearance model in the canonical space that is
  conditioned on predicted surface normals and the facial expressions and
  head-pose deformations. The surface normals prediction is guided using 3DMM
  normals that act as a coarse prior for the normals of the human head, where
  direct prediction of normals is hard due to rigid and non-rigid deformations
  induced by head-pose and facial expression changes. Using only a
  smartphone-captured short video of a subject for training, we demonstrate the
  effectiveness of our method on free view synthesis of a portrait scene with
  explicit head pose and expression controls, and realistic lighting effects. The
  project page can be found here:
  http://shahrukhathar.github.io/2023/08/22/CoDyNeRF.html
paper_ai_keywords: Neural Radiance Fields NeRFs reanimate head-pose facial expressions viewing direction photometric consistency deformed region canonical space dynamic independence model surface normals 3DMM normals free view synthesis head pose controls expression controls realistic lighting effects