NPGA: Neural Parametric Gaussian Avatars
The creation of high-fidelity, digital versions of human heads is an
important stepping stone in the process of further integrating virtual
components into our everyday lives. Constructing such avatars is a challenging
research problem, due to a high demand for photo-realism and real-time
rendering performance. In this work, we propose Neural Parametric Gaussian
Avatars (NPGA), a data-driven approach to create high-fidelity, controllable
avatars from multi-view video recordings. We build our method around 3D
Gaussian Splatting for its highly efficient rendering and to inherit the
topological flexibility of point clouds. In contrast to previous work, we
condition our avatars' dynamics on the rich expression space of neural
parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we
distill the backward deformation field of our underlying NPHM into forward
deformations which are compatible with rasterization-based rendering. All
remaining fine-scale, expression-dependent details are learned from the
multi-view videos. To increase the representational capacity of our avatars, we
augment the canonical Gaussian point cloud using per-primitive latent features
which govern its dynamic behavior. To regularize this increased dynamic
expressivity, we propose Laplacian terms on the latent features and predicted
dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating
that NPGA significantly outperforms the previous state-of-the-art avatars on
the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate
animation capabilities from real-world monocular videos.
66588f4fad7e5b2e764fdc87
Simon Giebenhain
admin_assigned
2024-05-31T11:06:12.381Z
64dc981b96fbcd6b19f687b5
/avatars/6eca33e3fcca750d4f6e9b0f7d724d8e.svg
Giebenhain
user
SimonGiebenhain
66588f4fad7e5b2e764fdc88
Tobias Kirschstein
admin_assigned
2024-05-31T11:06:18.062Z
64806288cacb1c4a06946084
/avatars/5101d2234279257d534d9e9d2f69527f.svg
Tobias Kirschstein
user
tobias-kirschstein
66588f4fad7e5b2e764fdc89
Martin Rünz
66588f4fad7e5b2e764fdc8a
Lourdes Agapito
66588f4fad7e5b2e764fdc8b
Matthias Nießner
NPGA: Neural Parametric Gaussian Avatars
The creation of high-fidelity, digital versions of human heads is an
important stepping stone in the process of further integrating virtual
components into our everyday lives. Constructing such avatars is a challenging
research problem, due to a high demand for photo-realism and real-time
rendering performance. In this work, we propose Neural Parametric Gaussian
Avatars (NPGA), a data-driven approach to create high-fidelity, controllable
avatars from multi-view video recordings. We build our method around 3D
Gaussian Splatting for its highly efficient rendering and to inherit the
topological flexibility of point clouds. In contrast to previous work, we
condition our avatars' dynamics on the rich expression space of neural
parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we
distill the backward deformation field of our underlying NPHM into forward
deformations which are compatible with rasterization-based rendering. All
remaining fine-scale, expression-dependent details are learned from the
multi-view videos. To increase the representational capacity of our avatars, we
augment the canonical Gaussian point cloud using per-primitive latent features
which govern its dynamic behavior. To regularize this increased dynamic
expressivity, we propose Laplacian terms on the latent features and predicted
dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating
that NPGA significantly outperforms the previous state-of-the-art avatars on
the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate
animation capabilities from real-world monocular videos.
Neural Parametric Gaussian Avatars (NPGA) create high-fidelity, real-time avatars from multi-view video recordings using 3D Gaussian Splatting and neural parametric head models, outperforming previous methods in self-reenactment and animation.
Neural Parametric Gaussian Avatars NPGA 3D Gaussian Splatting neural parametric head models NPHM mesh-based 3DMMs backward deformation field forward deformations rasterization-based rendering per-primitive latent features Laplacian terms