title: FlashTex: Fast Relightable Mesh Texturing with LightControlNet
summary: Manually creating textures for 3D meshes is time-consuming, even for expert
  visual content creators. We propose a fast approach for automatically texturing
  an input 3D mesh based on a user-provided text prompt. Importantly, our
  approach disentangles lighting from surface material/reflectance in the
  resulting texture so that the mesh can be properly relit and rendered in any
  lighting environment. We introduce LightControlNet, a new text-to-image model
  based on the ControlNet architecture, which allows the specification of the
  desired lighting as a conditioning image to the model. Our text-to-texture
  pipeline then constructs the texture in two stages. The first stage produces a
  sparse set of visually consistent reference views of the mesh using
  LightControlNet. The second stage applies a texture optimization based on Score
  Distillation Sampling (SDS) that works with LightControlNet to increase the
  texture quality while disentangling surface material from lighting. Our
  pipeline is significantly faster than previous text-to-texture methods, while
  producing high-quality and relightable textures.
paper_authors._id: 65d586cfb17d8c0681f3998c
paper_authors.name: Kangle Deng
paper_authors._id: 65d586cfb17d8c0681f3998d
paper_authors.name: Timothy Omernick
paper_authors._id: 65d586cfb17d8c0681f3998e
paper_authors.name: Alexander Weiss
paper_authors._id: 65d586cfb17d8c0681f3998f
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:34:53.192Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_authors._id: 65d586cfb17d8c0681f39990
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:35:22.450Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_authors._id: 65d586cfb17d8c0681f39991
paper_authors.name: Tinghui Zhou
paper_authors._id: 65d586cfb17d8c0681f39992
paper_authors.name: Maneesh Agrawala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:36:41.426Z
paper_authors._id: 642bb3e44f379cdc3deb26cd
paper_authors.avatarUrl: /avatars/556e792f013bcbe3748d7c9c719070d8.svg
paper_authors.fullname: Maneesh Agrawala
paper_authors.type: user
paper_authors.user: magrawala
paper_title: FlashTex: Fast Relightable Mesh Texturing with LightControlNet
paper_summary: Manually creating textures for 3D meshes is time-consuming, even for expert
  visual content creators. We propose a fast approach for automatically texturing
  an input 3D mesh based on a user-provided text prompt. Importantly, our
  approach disentangles lighting from surface material/reflectance in the
  resulting texture so that the mesh can be properly relit and rendered in any
  lighting environment. We introduce LightControlNet, a new text-to-image model
  based on the ControlNet architecture, which allows the specification of the
  desired lighting as a conditioning image to the model. Our text-to-texture
  pipeline then constructs the texture in two stages. The first stage produces a
  sparse set of visually consistent reference views of the mesh using
  LightControlNet. The second stage applies a texture optimization based on Score
  Distillation Sampling (SDS) that works with LightControlNet to increase the
  texture quality while disentangling surface material from lighting. Our
  pipeline is significantly faster than previous text-to-texture methods, while
  producing high-quality and relightable textures.
paper_ai_keywords: LightControlNet ControlNet text-to-image model conditioning image text-to-texture pipeline Score Distillation Sampling (SDS)