title: Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone
    Capture
summary: Creating photorealistic avatars for individuals traditionally involves
  extensive capture sessions with complex and expensive studio devices like the
  LightStage system. While recent strides in neural representations have enabled
  the generation of photorealistic and animatable 3D avatars from quick phone
  scans, they have the capture-time lighting baked-in, lack facial details and
  have missing regions in areas such as the back of the ears. Thus, they lag in
  quality compared to studio-captured avatars. In this paper, we propose a method
  that bridges this gap by generating studio-like illuminated texture maps from
  short, monocular phone captures. We do this by parameterizing the phone texture
  maps using the W^+ space of a StyleGAN2, enabling near-perfect
  reconstruction. Then, we finetune a StyleGAN2 by sampling in the W^+
  parameterized space using a very small set of studio-captured textures as an
  adversarial training signal. To further enhance the realism and accuracy of
  facial details, we super-resolve the output of the StyleGAN2 using carefully
  designed diffusion model that is guided by image gradients of the
  phone-captured texture map. Once trained, our method excels at producing
  studio-like facial texture maps from casual monocular smartphone videos.
  Demonstrating its capabilities, we showcase the generation of photorealistic,
  uniformly lit, complete avatars from monocular phone captures.
  http://shahrukhathar.github.io/2024/07/22/Bridging.html{The project page
  can be found here.}
paper_authors._id: 66a85d31b812afa43f9a83b3
paper_authors.name: ShahRukh Athar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:05:24.124Z
paper_authors._id: 650c3a2bd439bbbbadf40436
paper_authors.avatarUrl: /avatars/a5a7c04a03580ae59c28eb6f90d43b47.svg
paper_authors.fullname: ShahRukh Athar
paper_authors.type: user
paper_authors.user: Sathar
paper_authors._id: 66a85d31b812afa43f9a83b4
paper_authors.name: Shunsuke Saito
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-23T13:33:23.377Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_authors._id: 66a85d31b812afa43f9a83b5
paper_authors.name: Zhengyu Yang
paper_authors._id: 66a85d31b812afa43f9a83b6
paper_authors.name: Stanislav Pidhorsky
paper_authors._id: 66a85d31b812afa43f9a83b7
paper_authors.name: Chen Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T09:06:15.742Z
paper_authors._id: 66a606790ab6a8dc67910b6c
paper_authors.avatarUrl: /avatars/d4b55d788b69efdfe0be5a1afe571d51.svg
paper_authors.fullname: cao
paper_authors.type: user
paper_authors.user: chencao95
paper_title: Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone
    Capture
paper_summary: Creating photorealistic avatars for individuals traditionally involves
  extensive capture sessions with complex and expensive studio devices like the
  LightStage system. While recent strides in neural representations have enabled
  the generation of photorealistic and animatable 3D avatars from quick phone
  scans, they have the capture-time lighting baked-in, lack facial details and
  have missing regions in areas such as the back of the ears. Thus, they lag in
  quality compared to studio-captured avatars. In this paper, we propose a method
  that bridges this gap by generating studio-like illuminated texture maps from
  short, monocular phone captures. We do this by parameterizing the phone texture
  maps using the W^+ space of a StyleGAN2, enabling near-perfect
  reconstruction. Then, we finetune a StyleGAN2 by sampling in the W^+
  parameterized space using a very small set of studio-captured textures as an
  adversarial training signal. To further enhance the realism and accuracy of
  facial details, we super-resolve the output of the StyleGAN2 using carefully
  designed diffusion model that is guided by image gradients of the
  phone-captured texture map. Once trained, our method excels at producing
  studio-like facial texture maps from casual monocular smartphone videos.
  Demonstrating its capabilities, we showcase the generation of photorealistic,
  uniformly lit, complete avatars from monocular phone captures.
  http://shahrukhathar.github.io/2024/07/22/Bridging.html{The project page
  can be found here.}
paper_ai_keywords: Style-mdash; StyleGAN2 $W^+$ diffusion model image gradients