Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
Although the recent rapid evolution of 3D generative neural networks greatly
improves 3D shape generation, it is still not convenient for ordinary users to
create 3D shapes and control the local geometry of generated shapes. To address
these challenges, we propose a diffusion-based 3D generation framework --
locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch
image input. Our method is built on a two-stage diffusion model. The first
stage, named occupancy-diffusion, aims to generate a low-resolution occupancy
field to approximate the shape shell. The second stage, named SDF-diffusion,
synthesizes a high-resolution signed distance field within the occupied voxels
determined by the first stage to extract fine geometry. Our model is empowered
by a novel view-aware local attention mechanism for image-conditioned shape
generation, which takes advantage of 2D image patch features to guide 3D voxel
feature learning, greatly improving local controllability and model
generalizability. Through extensive experiments in sketch-conditioned and
category-conditioned 3D shape generation tasks, we validate and demonstrate the
ability of our method to provide plausible and diverse 3D shapes, as well as
its superior controllability and generalizability over existing work. Our code
and trained models are available at
https://zhengxinyang.github.io/projects/LAS-Diffusion.html
64599c9ca82daa98729dd4c2
Xin-Yang Zheng
64599c9ca82daa98729dd4c3
Hao Pan
64599c9ca82daa98729dd4c4
Peng-Shuai Wang
extracted_confirmed
2023-05-09T01:07:49.301Z
6333aabd5bcc1a4fe14d73de
https://cdn-avatars.huggingface.co/v1/production/uploads/6333aabd5bcc1a4fe14d73de/UclHAK80-kRtRPNJ8o3Dt.jpeg
Peng-Shuai Wang
user
wangps
64599c9ca82daa98729dd4c5
Xin Tong
64599c9ca82daa98729dd4c6
Yang Liu
64599c9ca82daa98729dd4c7
Heung-Yeung Shum
Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
Although the recent rapid evolution of 3D generative neural networks greatly
improves 3D shape generation, it is still not convenient for ordinary users to
create 3D shapes and control the local geometry of generated shapes. To address
these challenges, we propose a diffusion-based 3D generation framework --
locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch
image input. Our method is built on a two-stage diffusion model. The first
stage, named occupancy-diffusion, aims to generate a low-resolution occupancy
field to approximate the shape shell. The second stage, named SDF-diffusion,
synthesizes a high-resolution signed distance field within the occupied voxels
determined by the first stage to extract fine geometry. Our model is empowered
by a novel view-aware local attention mechanism for image-conditioned shape
generation, which takes advantage of 2D image patch features to guide 3D voxel
feature learning, greatly improving local controllability and model
generalizability. Through extensive experiments in sketch-conditioned and
category-conditioned 3D shape generation tasks, we validate and demonstrate the
ability of our method to provide plausible and diverse 3D shapes, as well as
its superior controllability and generalizability over existing work. Our code
and trained models are available at
https://zhengxinyang.github.io/projects/LAS-Diffusion.html
A diffusion-based framework offering improved local controllability and generalizability for 3D shape generation from 2D sketches.
diffusion-based 3D generation locally attentional SDF diffusion occupancy-diffusion SDF-diffusion view-aware local attention mechanism image-conditioned shape generation signed distance field 3D shape generation sketch-conditioned category-conditioned