title: Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
    Fields using Geometry-Guided Text-to-Image Diffusion Model
summary: Recent advances in diffusion models such as ControlNet have enabled
  geometrically controllable, high-fidelity text-to-image generation. However,
  none of them addresses the question of adding such controllability to
  text-to-3D generation. In response, we propose Text2Control3D, a controllable
  text-to-3D avatar generation method whose facial expression is controllable
  given a monocular video casually captured with hand-held camera. Our main
  strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
  optimized with a set of controlled viewpoint-aware images that we generate from
  ControlNet, whose condition input is the depth map extracted from the input
  video. When generating the viewpoint-aware images, we utilize cross-reference
  attention to inject well-controlled, referential facial expression and
  appearance via cross attention. We also conduct low-pass filtering of Gaussian
  latent of the diffusion model in order to ameliorate the viewpoint-agnostic
  texture problem we observed from our empirical analysis, where the
  viewpoint-aware images contain identical textures on identical pixel positions
  that are incomprehensible in 3D. Finally, to train NeRF with the images that
  are viewpoint-aware yet are not strictly consistent in geometry, our approach
  considers per-image geometric variation as a view of deformation from a shared
  3D canonical space. Consequently, we construct the 3D avatar in a canonical
  space of deformable NeRF by learning a set of per-image deformation via
  deformation field table. We demonstrate the empirical results and discuss the
  effectiveness of our method.
paper_authors._id: 64fa8cfde0dc35986b83ce50
paper_authors.name: Sungwon Hwang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-08T09:33:14.311Z
paper_authors._id: 642fcfc0a043f0ac7deeaae0
paper_authors.avatarUrl: /avatars/6cc46dd480cdc0d86c7a509e22782a13.svg
paper_authors.fullname: Sungwon Hwang
paper_authors.type: user
paper_authors.user: sungwon95
paper_authors._id: 64fa8cfde0dc35986b83ce51
paper_authors.name: Junha Hyung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:25:19.461Z
paper_authors._id: 6331ed964db0a767bbee4706
paper_authors.avatarUrl: /avatars/2ef3fa6083ca5208f4b87c91345267d9.svg
paper_authors.fullname: Junha Hyung
paper_authors.type: user
paper_authors.user: JunhaH
paper_authors._id: 64fa8cfde0dc35986b83ce52
paper_authors.name: Jaegul Choo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:25:28.411Z
paper_authors._id: 64be3127805e5b64572da65c
paper_authors.avatarUrl: /avatars/edd3e94ba9e375827cc75b164602bcac.svg
paper_authors.fullname: Jaegul Choo
paper_authors.type: user
paper_authors.user: joyfull78
paper_title: Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
    Fields using Geometry-Guided Text-to-Image Diffusion Model
paper_summary: Recent advances in diffusion models such as ControlNet have enabled
  geometrically controllable, high-fidelity text-to-image generation. However,
  none of them addresses the question of adding such controllability to
  text-to-3D generation. In response, we propose Text2Control3D, a controllable
  text-to-3D avatar generation method whose facial expression is controllable
  given a monocular video casually captured with hand-held camera. Our main
  strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
  optimized with a set of controlled viewpoint-aware images that we generate from
  ControlNet, whose condition input is the depth map extracted from the input
  video. When generating the viewpoint-aware images, we utilize cross-reference
  attention to inject well-controlled, referential facial expression and
  appearance via cross attention. We also conduct low-pass filtering of Gaussian
  latent of the diffusion model in order to ameliorate the viewpoint-agnostic
  texture problem we observed from our empirical analysis, where the
  viewpoint-aware images contain identical textures on identical pixel positions
  that are incomprehensible in 3D. Finally, to train NeRF with the images that
  are viewpoint-aware yet are not strictly consistent in geometry, our approach
  considers per-image geometric variation as a view of deformation from a shared
  3D canonical space. Consequently, we construct the 3D avatar in a canonical
  space of deformable NeRF by learning a set of per-image deformation via
  deformation field table. We demonstrate the empirical results and discuss the
  effectiveness of our method.
paper_ai_keywords: Neural fields Diffusion models ControlNet Geometrically controllable High-fidelity text-to-image generation Text-to-3D generation Controllable text-to-3D avatar generation Facial expression controllability Monocular video Neural Radiance Fields (NeRF) Controlled viewpoint-aware images Depth map Cross-reference attention Gaussian latent Viewpoint-agnostic texture problem Per-image geometric variation View deformation from a shared 3D canonical space Deformable NeRF Per-image deformation Deformation field table