title: Instant Facial Gaussians Translator for Relightable and Interactable
    Facial Rendering
summary: We propose GauFace, a novel Gaussian Splatting representation, tailored for
  efficient animation and rendering of physically-based facial assets. Leveraging
  strong geometric priors and constrained optimization, GauFace ensures a neat
  and structured Gaussian representation, delivering high fidelity and real-time
  facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.
    Then, we introduce TransGS, a diffusion transformer that instantly translates
  physically-based facial assets into the corresponding GauFace representations.
  Specifically, we adopt a patch-based pipeline to handle the vast number of
  Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme
  with UV positional encoding to ensure the throughput and rendering quality of
  GauFace assets generated by our TransGS. Once trained, TransGS can instantly
  translate facial assets with lighting conditions to GauFace representation,
  With the rich conditioning modalities, it also enables editing and animation
  capabilities reminiscent of traditional CG pipelines.
    We conduct extensive evaluations and user studies, compared to traditional
  offline and online renderers, as well as recent neural rendering methods, which
  demonstrate the superior performance of our approach for facial asset
  rendering. We also showcase diverse immersive applications of facial assets
  using our TransGS approach and GauFace representation, across various platforms
  like PCs, phones and even VR headsets.
paper_authors._id: 66e258a25f29feaa90934c4d
paper_authors.name: Dafei Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T10:25:39.572Z
paper_authors._id: 65b8c8b3fc836d9cd5b8a6ec
paper_authors.avatarUrl: /avatars/9c2490aebcca7b16396302f4f96acdfd.svg
paper_authors.fullname: Dafei Qin
paper_authors.type: user
paper_authors.user: dafeiqin
paper_authors._id: 66e258a25f29feaa90934c4e
paper_authors.name: Hongyang Lin
paper_authors._id: 66e258a25f29feaa90934c4f
paper_authors.name: Qixuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T10:25:54.414Z
paper_authors._id: 661d10eee7b0ab12bc6721d1
paper_authors.avatarUrl: /avatars/a35c91825baa90d4b0703a1e06ccc5f4.svg
paper_authors.fullname: Qixuan Zhang
paper_authors.type: user
paper_authors.user: Haaribo
paper_authors._id: 66e258a25f29feaa90934c50
paper_authors.name: Kaichun Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T10:25:59.975Z
paper_authors._id: 64c8c537cb2f1bf0e7c3e99d
paper_authors.avatarUrl: /avatars/ce8df2802840120022ec8ebaea4f525e.svg
paper_authors.fullname: Kaichun Qiao
paper_authors.type: user
paper_authors.user: DolphinQiao
paper_authors._id: 66e258a25f29feaa90934c51
paper_authors.name: Longwen Zhang
paper_authors._id: 66e258a25f29feaa90934c52
paper_authors.name: Zijun Zhao
paper_authors._id: 66e258a25f29feaa90934c53
paper_authors.name: Jun Saito
paper_authors._id: 66e258a25f29feaa90934c54
paper_authors.name: Jingyi Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T10:27:04.227Z
paper_authors._id: 63ffcec6e7767a89533b8f80
paper_authors.avatarUrl: /avatars/b9e8673943741c2e7b80b15a061bb951.svg
paper_authors.fullname: JINGYI Yu
paper_authors.type: user
paper_authors.user: Daluuu
paper_authors._id: 66e258a25f29feaa90934c55
paper_authors.name: Lan Xu
paper_authors._id: 66e258a25f29feaa90934c56
paper_authors.name: Taku Komura
paper_title: Instant Facial Gaussians Translator for Relightable and Interactable
    Facial Rendering
paper_summary: We propose GauFace, a novel Gaussian Splatting representation, tailored for
  efficient animation and rendering of physically-based facial assets. Leveraging
  strong geometric priors and constrained optimization, GauFace ensures a neat
  and structured Gaussian representation, delivering high fidelity and real-time
  facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.
    Then, we introduce TransGS, a diffusion transformer that instantly translates
  physically-based facial assets into the corresponding GauFace representations.
  Specifically, we adopt a patch-based pipeline to handle the vast number of
  Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme
  with UV positional encoding to ensure the throughput and rendering quality of
  GauFace assets generated by our TransGS. Once trained, TransGS can instantly
  translate facial assets with lighting conditions to GauFace representation,
  With the rich conditioning modalities, it also enables editing and animation
  capabilities reminiscent of traditional CG pipelines.
    We conduct extensive evaluations and user studies, compared to traditional
  offline and online renderers, as well as recent neural rendering methods, which
  demonstrate the superior performance of our approach for facial asset
  rendering. We also showcase diverse immersive applications of facial assets
  using our TransGS approach and GauFace representation, across various platforms
  like PCs, phones and even VR headsets.
paper_ai_keywords: Gaussian Splatting TransGS diffusion transformer patch-based pipeline pixel-aligned sampling UV positional encoding traditional CG pipelines neural rendering methods