title: Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
summary: Although the recent rapid evolution of 3D generative neural networks greatly
  improves 3D shape generation, it is still not convenient for ordinary users to
  create 3D shapes and control the local geometry of generated shapes. To address
  these challenges, we propose a diffusion-based 3D generation framework --
  locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch
  image input. Our method is built on a two-stage diffusion model. The first
  stage, named occupancy-diffusion, aims to generate a low-resolution occupancy
  field to approximate the shape shell. The second stage, named SDF-diffusion,
  synthesizes a high-resolution signed distance field within the occupied voxels
  determined by the first stage to extract fine geometry. Our model is empowered
  by a novel view-aware local attention mechanism for image-conditioned shape
  generation, which takes advantage of 2D image patch features to guide 3D voxel
  feature learning, greatly improving local controllability and model
  generalizability. Through extensive experiments in sketch-conditioned and
  category-conditioned 3D shape generation tasks, we validate and demonstrate the
  ability of our method to provide plausible and diverse 3D shapes, as well as
  its superior controllability and generalizability over existing work. Our code
  and trained models are available at
  https://zhengxinyang.github.io/projects/LAS-Diffusion.html
paper_authors._id: 64599c9ca82daa98729dd4c2
paper_authors.name: Xin-Yang Zheng
paper_authors._id: 64599c9ca82daa98729dd4c3
paper_authors.name: Hao Pan
paper_authors._id: 64599c9ca82daa98729dd4c4
paper_authors.name: Peng-Shuai Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-09T01:07:49.301Z
paper_authors._id: 6333aabd5bcc1a4fe14d73de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6333aabd5bcc1a4fe14d73de/UclHAK80-kRtRPNJ8o3Dt.jpeg
paper_authors.fullname: Peng-Shuai Wang
paper_authors.type: user
paper_authors.user: wangps
paper_authors._id: 64599c9ca82daa98729dd4c5
paper_authors.name: Xin Tong
paper_authors._id: 64599c9ca82daa98729dd4c6
paper_authors.name: Yang Liu
paper_authors._id: 64599c9ca82daa98729dd4c7
paper_authors.name: Heung-Yeung Shum
paper_title: Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
paper_summary: Although the recent rapid evolution of 3D generative neural networks greatly
  improves 3D shape generation, it is still not convenient for ordinary users to
  create 3D shapes and control the local geometry of generated shapes. To address
  these challenges, we propose a diffusion-based 3D generation framework --
  locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch
  image input. Our method is built on a two-stage diffusion model. The first
  stage, named occupancy-diffusion, aims to generate a low-resolution occupancy
  field to approximate the shape shell. The second stage, named SDF-diffusion,
  synthesizes a high-resolution signed distance field within the occupied voxels
  determined by the first stage to extract fine geometry. Our model is empowered
  by a novel view-aware local attention mechanism for image-conditioned shape
  generation, which takes advantage of 2D image patch features to guide 3D voxel
  feature learning, greatly improving local controllability and model
  generalizability. Through extensive experiments in sketch-conditioned and
  category-conditioned 3D shape generation tasks, we validate and demonstrate the
  ability of our method to provide plausible and diverse 3D shapes, as well as
  its superior controllability and generalizability over existing work. Our code
  and trained models are available at
  https://zhengxinyang.github.io/projects/LAS-Diffusion.html
paper_ai_keywords: diffusion-based 3D generation framework locally attentional SDF diffusion two-stage diffusion model occupancy-diffusion low-resolution occupancy field shape shell SDF-diffusion high-resolution signed distance field view-aware local attention mechanism image-conditioned shape generation 2D image patch features 3D voxel feature learning plausible and diverse 3D shapes sketch-conditioned category-conditioned