title: Relightable Gaussian Codec Avatars
summary: The fidelity of relighting is bounded by both geometry and appearance
  representations. For geometry, both mesh and volumetric approaches have
  difficulty modeling intricate structures like 3D hair geometry. For appearance,
  existing relighting models are limited in fidelity and often too slow to render
  in real-time with high-resolution continuous environments. In this work, we
  present Relightable Gaussian Codec Avatars, a method to build high-fidelity
  relightable head avatars that can be animated to generate novel expressions.
  Our geometry model based on 3D Gaussians can capture 3D-consistent
  sub-millimeter details such as hair strands and pores on dynamic face
  sequences. To support diverse materials of human heads such as the eyes, skin,
  and hair in a unified manner, we present a novel relightable appearance model
  based on learnable radiance transfer. Together with global illumination-aware
  spherical harmonics for the diffuse components, we achieve real-time relighting
  with spatially all-frequency reflections using spherical Gaussians. This
  appearance model can be efficiently relit under both point light and continuous
  illumination. We further improve the fidelity of eye reflections and enable
  explicit gaze control by introducing relightable explicit eye models. Our
  method outperforms existing approaches without compromising real-time
  performance. We also demonstrate real-time relighting of avatars on a tethered
  consumer VR headset, showcasing the efficiency and fidelity of our avatars.
paper_authors._id: 65712957c8018fe6408b90b4
paper_authors.name: Shunsuke Saito
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:30:04.043Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_authors._id: 65712957c8018fe6408b90b5
paper_authors.name: Gabriel Schwartz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:30:10.712Z
paper_authors._id: 63b261b9d5f79ac67d2cace3
paper_authors.avatarUrl: /avatars/fa560a2e7732cf9e4cd85db3496fd6b2.svg
paper_authors.fullname: Gabriel Schwartz 
paper_authors.type: user
paper_authors.user: GBielXONE02
paper_authors._id: 65712957c8018fe6408b90b6
paper_authors.name: Tomas Simon
paper_authors._id: 65712957c8018fe6408b90b7
paper_authors.name: Junxuan Li
paper_authors._id: 65712957c8018fe6408b90b8
paper_authors.name: Giljoo Nam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T15:10:38.283Z
paper_authors._id: 6571de376270264216b92e5f
paper_authors.avatarUrl: /avatars/81a8b82b6584aa896317638b88c5d671.svg
paper_authors.fullname: Giljoo Nam
paper_authors.type: user
paper_authors.user: giljoonam
paper_title: Relightable Gaussian Codec Avatars
paper_summary: The fidelity of relighting is bounded by both geometry and appearance
  representations. For geometry, both mesh and volumetric approaches have
  difficulty modeling intricate structures like 3D hair geometry. For appearance,
  existing relighting models are limited in fidelity and often too slow to render
  in real-time with high-resolution continuous environments. In this work, we
  present Relightable Gaussian Codec Avatars, a method to build high-fidelity
  relightable head avatars that can be animated to generate novel expressions.
  Our geometry model based on 3D Gaussians can capture 3D-consistent
  sub-millimeter details such as hair strands and pores on dynamic face
  sequences. To support diverse materials of human heads such as the eyes, skin,
  and hair in a unified manner, we present a novel relightable appearance model
  based on learnable radiance transfer. Together with global illumination-aware
  spherical harmonics for the diffuse components, we achieve real-time relighting
  with spatially all-frequency reflections using spherical Gaussians. This
  appearance model can be efficiently relit under both point light and continuous
  illumination. We further improve the fidelity of eye reflections and enable
  explicit gaze control by introducing relightable explicit eye models. Our
  method outperforms existing approaches without compromising real-time
  performance. We also demonstrate real-time relighting of avatars on a tethered
  consumer VR headset, showcasing the efficiency and fidelity of our avatars.
paper_ai_summary: Relightable Gaussian Codec Avatars model high-fidelity head avatars with real-time relighting capabilities using 3D Gaussians for geometry and learnable radiance transfer for appearance.
paper_ai_keywords: 3D Gaussians relightable appearance model learnable radiance transfer global illumination-aware spherical harmonics spherical Gaussians explicit eye models real-time relighting tethered consumer VR headset