title: Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation
summary: Recently, text-to-3D approaches have achieved high-fidelity 3D content
  generation using text description. However, the generated objects are
  stochastic and lack fine-grained control. Sketches provide a cheap approach to
  introduce such fine-grained control. Nevertheless, it is challenging to achieve
  flexible control from these sketches due to their abstraction and ambiguity. In
  this paper, we present a multi-view sketch-guided text-to-3D generation
  framework (namely, Sketch2NeRF) to add sketch control to 3D generation.
  Specifically, our method leverages pretrained 2D diffusion models (e.g., Stable
  Diffusion and ControlNet) to supervise the optimization of a 3D scene
  represented by a neural radiance field (NeRF). We propose a novel synchronized
  generation and reconstruction method to effectively optimize the NeRF. In the
  experiments, we collected two kinds of multi-view sketch datasets to evaluate
  the proposed method. We demonstrate that our method can synthesize 3D
  consistent contents with fine-grained sketch control while being high-fidelity
  to text prompts. Extensive results show that our method achieves
  state-of-the-art performance in terms of sketch similarity and text alignment.
paper_authors._id: 65b31715c8a577067d293198
paper_authors.name: Minglin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:02:56.795Z
paper_authors._id: 632685e7ab50d3a02a3524e8
paper_authors.avatarUrl: /avatars/54a7b2a20ea9661b188a23a9e4fbdfd4.svg
paper_authors.fullname: Minglin Chen
paper_authors.type: user
paper_authors.user: mlchen
paper_authors._id: 65b31715c8a577067d293199
paper_authors.name: Longguang Wang
paper_authors._id: 65b31715c8a577067d29319a
paper_authors.name: Weihao Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:03:33.317Z
paper_authors._id: 649e7793f8565ca45f6a18d1
paper_authors.avatarUrl: /avatars/50b75a435a3baf5ec0de7259f19fb9f8.svg
paper_authors.fullname: Weihao Yuan
paper_authors.type: user
paper_authors.user: willsky
paper_authors._id: 65b31715c8a577067d29319b
paper_authors.name: Yukun Wang
paper_authors._id: 65b31715c8a577067d29319c
paper_authors.name: Zhe Sheng
paper_authors._id: 65b31715c8a577067d29319d
paper_authors.name: Yisheng He
paper_authors._id: 65b31715c8a577067d29319e
paper_authors.name: Zilong Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:04:25.750Z
paper_authors._id: 6582878f97c071ac05da130b
paper_authors.avatarUrl: /avatars/b4265ec4eaafff5b0f7b44f983a1cb0e.svg
paper_authors.fullname: Zilong Dong
paper_authors.type: user
paper_authors.user: zjudzl
paper_authors._id: 65b31715c8a577067d29319f
paper_authors.name: Liefeng Bo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:04:37.381Z
paper_authors._id: 63d0cc736b985b0f25d0412c
paper_authors.avatarUrl: /avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
paper_authors.fullname: Bo
paper_authors.type: user
paper_authors.user: Liefeng
paper_authors._id: 65b31715c8a577067d2931a0
paper_authors.name: Yulan Guo
paper_title: Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation
paper_summary: Recently, text-to-3D approaches have achieved high-fidelity 3D content
  generation using text description. However, the generated objects are
  stochastic and lack fine-grained control. Sketches provide a cheap approach to
  introduce such fine-grained control. Nevertheless, it is challenging to achieve
  flexible control from these sketches due to their abstraction and ambiguity. In
  this paper, we present a multi-view sketch-guided text-to-3D generation
  framework (namely, Sketch2NeRF) to add sketch control to 3D generation.
  Specifically, our method leverages pretrained 2D diffusion models (e.g., Stable
  Diffusion and ControlNet) to supervise the optimization of a 3D scene
  represented by a neural radiance field (NeRF). We propose a novel synchronized
  generation and reconstruction method to effectively optimize the NeRF. In the
  experiments, we collected two kinds of multi-view sketch datasets to evaluate
  the proposed method. We demonstrate that our method can synthesize 3D
  consistent contents with fine-grained sketch control while being high-fidelity
  to text prompts. Extensive results show that our method achieves
  state-of-the-art performance in terms of sketch similarity and text alignment.
paper_ai_keywords: diffusion models Stable Diffusion ControlNet neural radiance field NeRF synchronization generation reconstruction multi-view sketch datasets sketch consistency text prompts