title: NPGA: Neural Parametric Gaussian Avatars
summary: The creation of high-fidelity, digital versions of human heads is an
  important stepping stone in the process of further integrating virtual
  components into our everyday lives. Constructing such avatars is a challenging
  research problem, due to a high demand for photo-realism and real-time
  rendering performance. In this work, we propose Neural Parametric Gaussian
  Avatars (NPGA), a data-driven approach to create high-fidelity, controllable
  avatars from multi-view video recordings. We build our method around 3D
  Gaussian Splatting for its highly efficient rendering and to inherit the
  topological flexibility of point clouds. In contrast to previous work, we
  condition our avatars' dynamics on the rich expression space of neural
  parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we
  distill the backward deformation field of our underlying NPHM into forward
  deformations which are compatible with rasterization-based rendering. All
  remaining fine-scale, expression-dependent details are learned from the
  multi-view videos. To increase the representational capacity of our avatars, we
  augment the canonical Gaussian point cloud using per-primitive latent features
  which govern its dynamic behavior. To regularize this increased dynamic
  expressivity, we propose Laplacian terms on the latent features and predicted
  dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating
  that NPGA significantly outperforms the previous state-of-the-art avatars on
  the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate
  animation capabilities from real-world monocular videos.
paper_authors._id: 66588f4fad7e5b2e764fdc87
paper_authors.name: Simon Giebenhain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:06:12.381Z
paper_authors._id: 64dc981b96fbcd6b19f687b5
paper_authors.avatarUrl: /avatars/6eca33e3fcca750d4f6e9b0f7d724d8e.svg
paper_authors.fullname: Giebenhain
paper_authors.type: user
paper_authors.user: SimonGiebenhain
paper_authors._id: 66588f4fad7e5b2e764fdc88
paper_authors.name: Tobias Kirschstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:06:18.062Z
paper_authors._id: 64806288cacb1c4a06946084
paper_authors.avatarUrl: /avatars/5101d2234279257d534d9e9d2f69527f.svg
paper_authors.fullname: Tobias Kirschstein
paper_authors.type: user
paper_authors.user: tobias-kirschstein
paper_authors._id: 66588f4fad7e5b2e764fdc89
paper_authors.name: Martin Rünz
paper_authors._id: 66588f4fad7e5b2e764fdc8a
paper_authors.name: Lourdes Agapito
paper_authors._id: 66588f4fad7e5b2e764fdc8b
paper_authors.name: Matthias Nießner
paper_title: NPGA: Neural Parametric Gaussian Avatars
paper_summary: The creation of high-fidelity, digital versions of human heads is an
  important stepping stone in the process of further integrating virtual
  components into our everyday lives. Constructing such avatars is a challenging
  research problem, due to a high demand for photo-realism and real-time
  rendering performance. In this work, we propose Neural Parametric Gaussian
  Avatars (NPGA), a data-driven approach to create high-fidelity, controllable
  avatars from multi-view video recordings. We build our method around 3D
  Gaussian Splatting for its highly efficient rendering and to inherit the
  topological flexibility of point clouds. In contrast to previous work, we
  condition our avatars' dynamics on the rich expression space of neural
  parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we
  distill the backward deformation field of our underlying NPHM into forward
  deformations which are compatible with rasterization-based rendering. All
  remaining fine-scale, expression-dependent details are learned from the
  multi-view videos. To increase the representational capacity of our avatars, we
  augment the canonical Gaussian point cloud using per-primitive latent features
  which govern its dynamic behavior. To regularize this increased dynamic
  expressivity, we propose Laplacian terms on the latent features and predicted
  dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating
  that NPGA significantly outperforms the previous state-of-the-art avatars on
  the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate
  animation capabilities from real-world monocular videos.
paper_ai_summary: Neural Parametric Gaussian Avatars (NPGA) create high-fidelity, real-time avatars from multi-view video recordings using 3D Gaussian Splatting and neural parametric head models, outperforming previous methods in self-reenactment and animation.
paper_ai_keywords: Neural Parametric Gaussian Avatars NPGA 3D Gaussian Splatting neural parametric head models NPHM mesh-based 3DMMs backward deformation field forward deformations rasterization-based rendering per-primitive latent features Laplacian terms