Relightable Gaussian Codec Avatars
The fidelity of relighting is bounded by both geometry and appearance
representations. For geometry, both mesh and volumetric approaches have
difficulty modeling intricate structures like 3D hair geometry. For appearance,
existing relighting models are limited in fidelity and often too slow to render
in real-time with high-resolution continuous environments. In this work, we
present Relightable Gaussian Codec Avatars, a method to build high-fidelity
relightable head avatars that can be animated to generate novel expressions.
Our geometry model based on 3D Gaussians can capture 3D-consistent
sub-millimeter details such as hair strands and pores on dynamic face
sequences. To support diverse materials of human heads such as the eyes, skin,
and hair in a unified manner, we present a novel relightable appearance model
based on learnable radiance transfer. Together with global illumination-aware
spherical harmonics for the diffuse components, we achieve real-time relighting
with spatially all-frequency reflections using spherical Gaussians. This
appearance model can be efficiently relit under both point light and continuous
illumination. We further improve the fidelity of eye reflections and enable
explicit gaze control by introducing relightable explicit eye models. Our
method outperforms existing approaches without compromising real-time
performance. We also demonstrate real-time relighting of avatars on a tethered
consumer VR headset, showcasing the efficiency and fidelity of our avatars.
65712957c8018fe6408b90b4
Shunsuke Saito
admin_assigned
2023-12-07T11:30:04.043Z
630688edd37ce67e0e50e548
/avatars/1681e465d7649f67f94e1b69d236cb1e.svg
Shunsuke Saito
user
psyth
65712957c8018fe6408b90b5
Gabriel Schwartz
admin_assigned
2023-12-07T11:30:10.712Z
63b261b9d5f79ac67d2cace3
/avatars/fa560a2e7732cf9e4cd85db3496fd6b2.svg
Gabriel Schwartz 
user
GBielXONE02
65712957c8018fe6408b90b6
Tomas Simon
65712957c8018fe6408b90b7
Junxuan Li
65712957c8018fe6408b90b8
Giljoo Nam
claimed_verified
2023-12-07T15:10:38.283Z
6571de376270264216b92e5f
/avatars/81a8b82b6584aa896317638b88c5d671.svg
Giljoo Nam
user
giljoonam
Relightable Gaussian Codec Avatars
The fidelity of relighting is bounded by both geometry and appearance
representations. For geometry, both mesh and volumetric approaches have
difficulty modeling intricate structures like 3D hair geometry. For appearance,
existing relighting models are limited in fidelity and often too slow to render
in real-time with high-resolution continuous environments. In this work, we
present Relightable Gaussian Codec Avatars, a method to build high-fidelity
relightable head avatars that can be animated to generate novel expressions.
Our geometry model based on 3D Gaussians can capture 3D-consistent
sub-millimeter details such as hair strands and pores on dynamic face
sequences. To support diverse materials of human heads such as the eyes, skin,
and hair in a unified manner, we present a novel relightable appearance model
based on learnable radiance transfer. Together with global illumination-aware
spherical harmonics for the diffuse components, we achieve real-time relighting
with spatially all-frequency reflections using spherical Gaussians. This
appearance model can be efficiently relit under both point light and continuous
illumination. We further improve the fidelity of eye reflections and enable
explicit gaze control by introducing relightable explicit eye models. Our
method outperforms existing approaches without compromising real-time
performance. We also demonstrate real-time relighting of avatars on a tethered
consumer VR headset, showcasing the efficiency and fidelity of our avatars.
Relightable Gaussian Codec Avatars model high-fidelity head avatars with real-time relighting capabilities using 3D Gaussians for geometry and learnable radiance transfer for appearance.
3D Gaussians relightable appearance model learnable radiance transfer global illumination-aware spherical harmonics spherical Gaussians explicit eye models real-time relighting tethered consumer VR headset