title: Coin3D: Controllable and Interactive 3D Assets Generation with
    Proxy-Guided Conditioning
summary: As humans, we aspire to create media content that is both freely willed and
  readily controlled. Thanks to the prominent development of generative
  techniques, we now can easily utilize 2D diffusion methods to synthesize images
  controlled by raw sketch or designated human poses, and even progressively
  edit/regenerate local regions with masked inpainting. However, similar
  workflows in 3D modeling tasks are still unavailable due to the lack of
  controllability and efficiency in 3D generation. In this paper, we present a
  novel controllable and interactive 3D assets modeling framework, named Coin3D.
  Coin3D allows users to control the 3D generation using a coarse geometry proxy
  assembled from basic shapes, and introduces an interactive generation workflow
  to support seamless local part editing while delivering responsive 3D object
  previewing within a few seconds. To this end, we develop several techniques,
  including the 3D adapter that applies volumetric coarse shape control to the
  diffusion model, proxy-bounded editing strategy for precise part editing,
  progressive volume cache to support responsive preview, and volume-SDS to
  ensure consistent mesh reconstruction. Extensive experiments of interactive
  generation and editing on diverse shape proxies demonstrate that our method
  achieves superior controllability and flexibility in the 3D assets generation
  task.
paper_authors._id: 6644e03e366e9c68806db777
paper_authors.name: Wenqi Dong
paper_authors._id: 6644e03e366e9c68806db778
paper_authors.name: Bangbang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-16T10:43:06.250Z
paper_authors._id: 63d748ff6f49aa82306b7e48
paper_authors.avatarUrl: /avatars/9f0b8b8a09b14d76e52ed1bd312e6b63.svg
paper_authors.fullname: BB Yang
paper_authors.type: user
paper_authors.user: ybbbbt
paper_authors._id: 6644e03e366e9c68806db779
paper_authors.name: Lin Ma
paper_authors._id: 6644e03e366e9c68806db77a
paper_authors.name: Xiao Liu
paper_authors._id: 6644e03e366e9c68806db77b
paper_authors.name: Liyuan Cui
paper_authors._id: 6644e03e366e9c68806db77c
paper_authors.name: Hujun Bao
paper_authors._id: 6644e03e366e9c68806db77d
paper_authors.name: Yuewen Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T17:37:03.774Z
paper_authors._id: 641a74bdf5e9c66105015385
paper_authors.avatarUrl: /avatars/13f966357de0ef5bb192ae566ad85ca0.svg
paper_authors.fullname: Zack Ma
paper_authors.type: user
paper_authors.user: yuewenma
paper_authors._id: 6644e03e366e9c68806db77e
paper_authors.name: Zhaopeng Cui
paper_title: Coin3D: Controllable and Interactive 3D Assets Generation with
    Proxy-Guided Conditioning
paper_summary: As humans, we aspire to create media content that is both freely willed and
  readily controlled. Thanks to the prominent development of generative
  techniques, we now can easily utilize 2D diffusion methods to synthesize images
  controlled by raw sketch or designated human poses, and even progressively
  edit/regenerate local regions with masked inpainting. However, similar
  workflows in 3D modeling tasks are still unavailable due to the lack of
  controllability and efficiency in 3D generation. In this paper, we present a
  novel controllable and interactive 3D assets modeling framework, named Coin3D.
  Coin3D allows users to control the 3D generation using a coarse geometry proxy
  assembled from basic shapes, and introduces an interactive generation workflow
  to support seamless local part editing while delivering responsive 3D object
  previewing within a few seconds. To this end, we develop several techniques,
  including the 3D adapter that applies volumetric coarse shape control to the
  diffusion model, proxy-bounded editing strategy for precise part editing,
  progressive volume cache to support responsive preview, and volume-SDS to
  ensure consistent mesh reconstruction. Extensive experiments of interactive
  generation and editing on diverse shape proxies demonstrate that our method
  achieves superior controllability and flexibility in the 3D assets generation
  task.
paper_ai_summary: Coin3D is an interactive 3D modeling framework that uses a coarse geometry proxy to control 3D generation, supports seamless local editing, and provides responsive previews using diffusion models and various novel techniques.
paper_ai_keywords: generative techniques 2D diffusion methods masked inpainting 3D generation controllable and interactive 3D assets modeling Coin3D coarse geometry proxy interactive generation workflow 3D adapter proxy-bounded editing strategy progressive volume cache volume-SDS mesh reconstruction interactive generation local editing responsive preview