title: Fast Registration of Photorealistic Avatars for VR Facial Animation
summary: Virtual Reality (VR) bares promise of social interactions that can feel more
  immersive than other media. Key to this is the ability to accurately animate a
  photorealistic avatar of one's likeness while wearing a VR headset. Although
  high quality registration of person-specific avatars to headset-mounted camera
  (HMC) images is possible in an offline setting, the performance of generic
  realtime models are significantly degraded. Online registration is also
  challenging due to oblique camera views and differences in modality. In this
  work, we first show that the domain gap between the avatar and headset-camera
  images is one of the primary sources of difficulty, where a transformer-based
  architecture achieves high accuracy on domain-consistent data, but degrades
  when the domain-gap is re-introduced. Building on this finding, we develop a
  system design that decouples the problem into two parts: 1) an iterative
  refinement module that takes in-domain inputs, and 2) a generic avatar-guided
  image-to-image style transfer module that is conditioned on current estimation
  of expression and head pose. These two modules reinforce each other, as image
  style transfer becomes easier when close-to-ground-truth examples are shown,
  and better domain-gap removal helps registration. Our system produces
  high-quality results efficiently, obviating the need for costly offline
  registration to generate personalized labels. We validate the accuracy and
  efficiency of our approach through extensive experiments on a commodity
  headset, demonstrating significant improvements over direct regression methods
  as well as offline registration.
paper_authors._id: 65af5993a134c07ddedaf223
paper_authors.name: Chaitanya Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:25:16.873Z
paper_authors._id: 66f10fa5f9cf1bf7e6798d15
paper_authors.avatarUrl: /avatars/ff416c6778091eaf13f6c233a5e42244.svg
paper_authors.fullname: Chaitanya Patel
paper_authors.type: user
paper_authors.user: chaitanya100100
paper_authors._id: 65af5993a134c07ddedaf224
paper_authors.name: Shaojie Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T11:19:30.637Z
paper_authors._id: 646a08304c1cd18b4975dfbe
paper_authors.avatarUrl: /avatars/dfca388bb89deb82e874cd633c832a2c.svg
paper_authors.fullname: BaiShaojie
paper_authors.type: user
paper_authors.user: otnw
paper_authors._id: 65af5993a134c07ddedaf225
paper_authors.name: Te-Li Wang
paper_authors._id: 65af5993a134c07ddedaf226
paper_authors.name: Jason Saragih
paper_authors._id: 65af5993a134c07ddedaf227
paper_authors.name: Shih-En Wei
paper_title: Fast Registration of Photorealistic Avatars for VR Facial Animation
paper_summary: Virtual Reality (VR) bares promise of social interactions that can feel more
  immersive than other media. Key to this is the ability to accurately animate a
  photorealistic avatar of one's likeness while wearing a VR headset. Although
  high quality registration of person-specific avatars to headset-mounted camera
  (HMC) images is possible in an offline setting, the performance of generic
  realtime models are significantly degraded. Online registration is also
  challenging due to oblique camera views and differences in modality. In this
  work, we first show that the domain gap between the avatar and headset-camera
  images is one of the primary sources of difficulty, where a transformer-based
  architecture achieves high accuracy on domain-consistent data, but degrades
  when the domain-gap is re-introduced. Building on this finding, we develop a
  system design that decouples the problem into two parts: 1) an iterative
  refinement module that takes in-domain inputs, and 2) a generic avatar-guided
  image-to-image style transfer module that is conditioned on current estimation
  of expression and head pose. These two modules reinforce each other, as image
  style transfer becomes easier when close-to-ground-truth examples are shown,
  and better domain-gap removal helps registration. Our system produces
  high-quality results efficiently, obviating the need for costly offline
  registration to generate personalized labels. We validate the accuracy and
  efficiency of our approach through extensive experiments on a commodity
  headset, demonstrating significant improvements over direct regression methods
  as well as offline registration.
paper_ai_summary: A system for real-time avatar animation using VR cameras decouples domain gap removal and iterative refinement, improving accuracy and efficiency over existing methods.
paper_ai_keywords: transformer-based architecture image-to-image style transfer domain gap iterative refinement expression head pose