Coin3D: Controllable and Interactive 3D Assets Generation with
  Proxy-Guided Conditioning
As humans, we aspire to create media content that is both freely willed and
readily controlled. Thanks to the prominent development of generative
techniques, we now can easily utilize 2D diffusion methods to synthesize images
controlled by raw sketch or designated human poses, and even progressively
edit/regenerate local regions with masked inpainting. However, similar
workflows in 3D modeling tasks are still unavailable due to the lack of
controllability and efficiency in 3D generation. In this paper, we present a
novel controllable and interactive 3D assets modeling framework, named Coin3D.
Coin3D allows users to control the 3D generation using a coarse geometry proxy
assembled from basic shapes, and introduces an interactive generation workflow
to support seamless local part editing while delivering responsive 3D object
previewing within a few seconds. To this end, we develop several techniques,
including the 3D adapter that applies volumetric coarse shape control to the
diffusion model, proxy-bounded editing strategy for precise part editing,
progressive volume cache to support responsive preview, and volume-SDS to
ensure consistent mesh reconstruction. Extensive experiments of interactive
generation and editing on diverse shape proxies demonstrate that our method
achieves superior controllability and flexibility in the 3D assets generation
task.
6644e03e366e9c68806db777
Wenqi Dong
6644e03e366e9c68806db778
Bangbang Yang
claimed_verified
2024-05-16T10:43:06.250Z
63d748ff6f49aa82306b7e48
/avatars/9f0b8b8a09b14d76e52ed1bd312e6b63.svg
BB Yang
user
ybbbbt
6644e03e366e9c68806db779
Lin Ma
6644e03e366e9c68806db77a
Xiao Liu
6644e03e366e9c68806db77b
Liyuan Cui
6644e03e366e9c68806db77c
Hujun Bao
6644e03e366e9c68806db77d
Yuewen Ma
admin_assigned
2024-05-15T17:37:03.774Z
641a74bdf5e9c66105015385
/avatars/13f966357de0ef5bb192ae566ad85ca0.svg
Zack Ma
user
yuewenma
6644e03e366e9c68806db77e
Zhaopeng Cui
Coin3D: Controllable and Interactive 3D Assets Generation with
  Proxy-Guided Conditioning
As humans, we aspire to create media content that is both freely willed and
readily controlled. Thanks to the prominent development of generative
techniques, we now can easily utilize 2D diffusion methods to synthesize images
controlled by raw sketch or designated human poses, and even progressively
edit/regenerate local regions with masked inpainting. However, similar
workflows in 3D modeling tasks are still unavailable due to the lack of
controllability and efficiency in 3D generation. In this paper, we present a
novel controllable and interactive 3D assets modeling framework, named Coin3D.
Coin3D allows users to control the 3D generation using a coarse geometry proxy
assembled from basic shapes, and introduces an interactive generation workflow
to support seamless local part editing while delivering responsive 3D object
previewing within a few seconds. To this end, we develop several techniques,
including the 3D adapter that applies volumetric coarse shape control to the
diffusion model, proxy-bounded editing strategy for precise part editing,
progressive volume cache to support responsive preview, and volume-SDS to
ensure consistent mesh reconstruction. Extensive experiments of interactive
generation and editing on diverse shape proxies demonstrate that our method
achieves superior controllability and flexibility in the 3D assets generation
task.
Coin3D is an interactive 3D modeling framework that uses a coarse geometry proxy to control 3D generation, supports seamless local editing, and provides responsive previews using diffusion models and various novel techniques.
generative techniques 2D diffusion methods masked inpainting 3D generation controllable and interactive 3D assets modeling Coin3D coarse geometry proxy interactive generation workflow 3D adapter proxy-bounded editing strategy progressive volume cache volume-SDS mesh reconstruction interactive generation local editing responsive preview