title: FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural
    Radiance Fields
summary: As recent advances in Neural Radiance Fields (NeRF) have enabled
  high-fidelity 3D face reconstruction and novel view synthesis, its manipulation
  also became an essential task in 3D vision. However, existing manipulation
  methods require extensive human labor, such as a user-provided semantic mask
  and manual attribute search unsuitable for non-expert users. Instead, our
  approach is designed to require a single text to manipulate a face
  reconstructed with NeRF. To do so, we first train a scene manipulator, a latent
  code-conditional deformable NeRF, over a dynamic scene to control a face
  deformation using the latent code. However, representing a scene deformation
  with a single latent code is unfavorable for compositing local deformations
  observed in different instances. As so, our proposed Position-conditional
  Anchor Compositor (PAC) learns to represent a manipulated scene with spatially
  varying latent codes. Their renderings with the scene manipulator are then
  optimized to yield high cosine similarity to a target text in CLIP embedding
  space for text-driven manipulation. To the best of our knowledge, our approach
  is the first to address the text-driven manipulation of a face reconstructed
  with NeRF. Extensive results, comparisons, and ablation studies demonstrate the
  effectiveness of our approach.
paper_authors._id: 64be087f76a6e2efccb80b0c
paper_authors.name: Sungwon Hwang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-24T07:42:16.208Z
paper_authors._id: 642fcfc0a043f0ac7deeaae0
paper_authors.avatarUrl: /avatars/6cc46dd480cdc0d86c7a509e22782a13.svg
paper_authors.fullname: Sungwon Hwang
paper_authors.type: user
paper_authors.user: sungwon95
paper_authors._id: 64be087f76a6e2efccb80b0d
paper_authors.name: Junha Hyung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-24T07:47:54.356Z
paper_authors._id: 6331ed964db0a767bbee4706
paper_authors.avatarUrl: /avatars/2ef3fa6083ca5208f4b87c91345267d9.svg
paper_authors.fullname: Junha Hyung
paper_authors.type: user
paper_authors.user: JunhaH
paper_authors._id: 64be087f76a6e2efccb80b0e
paper_authors.name: Daejin Kim
paper_authors._id: 64be087f76a6e2efccb80b0f
paper_authors.name: Min-Jung Kim
paper_authors._id: 64be087f76a6e2efccb80b10
paper_authors.name: Jaegul Choo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-24T11:31:32.437Z
paper_authors._id: 64be3127805e5b64572da65c
paper_authors.avatarUrl: /avatars/edd3e94ba9e375827cc75b164602bcac.svg
paper_authors.fullname: Jaegul Choo
paper_authors.type: user
paper_authors.user: joyfull78
paper_title: FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural
    Radiance Fields
paper_summary: As recent advances in Neural Radiance Fields (NeRF) have enabled
  high-fidelity 3D face reconstruction and novel view synthesis, its manipulation
  also became an essential task in 3D vision. However, existing manipulation
  methods require extensive human labor, such as a user-provided semantic mask
  and manual attribute search unsuitable for non-expert users. Instead, our
  approach is designed to require a single text to manipulate a face
  reconstructed with NeRF. To do so, we first train a scene manipulator, a latent
  code-conditional deformable NeRF, over a dynamic scene to control a face
  deformation using the latent code. However, representing a scene deformation
  with a single latent code is unfavorable for compositing local deformations
  observed in different instances. As so, our proposed Position-conditional
  Anchor Compositor (PAC) learns to represent a manipulated scene with spatially
  varying latent codes. Their renderings with the scene manipulator are then
  optimized to yield high cosine similarity to a target text in CLIP embedding
  space for text-driven manipulation. To the best of our knowledge, our approach
  is the first to address the text-driven manipulation of a face reconstructed
  with NeRF. Extensive results, comparisons, and ablation studies demonstrate the
  effectiveness of our approach.
paper_ai_summary: The proposed Position-conditional Anchor Compositor (PAC) enables text-driven manipulation of NeRF-reconstructed faces using spatially varying latent codes and CLIP embedding optimization, requiring minimal human input.
paper_ai_keywords: Neural Radiance Fields NeRF 3D face reconstruction novel view synthesis scene manipulator latent code-conditional deformable NeRF Position-conditional Anchor Compositor PAC scene deformation spatially varying latent codes CLIP embedding space text-driven manipulation