title: PartGen: Part-level 3D Generation and Reconstruction with Multi-View
    Diffusion Models
summary: Text- or image-to-3D generators and 3D scanners can now produce 3D assets
  with high-quality shapes and textures. These assets typically consist of a
  single, fused representation, like an implicit neural field, a Gaussian
  mixture, or a mesh, without any useful structure. However, most applications
  and creative workflows require assets to be made of several meaningful parts
  that can be manipulated independently. To address this gap, we introduce
  PartGen, a novel approach that generates 3D objects composed of meaningful
  parts starting from text, an image, or an unstructured 3D object. First, given
  multiple views of a 3D object, generated or rendered, a multi-view diffusion
  model extracts a set of plausible and view-consistent part segmentations,
  dividing the object into parts. Then, a second multi-view diffusion model takes
  each part separately, fills in the occlusions, and uses those completed views
  for 3D reconstruction by feeding them to a 3D reconstruction network. This
  completion process considers the context of the entire object to ensure that
  the parts integrate cohesively. The generative completion model can make up for
  the information missing due to occlusions; in extreme cases, it can hallucinate
  entirely invisible parts based on the input 3D asset. We evaluate our method on
  generated and real 3D assets and show that it outperforms segmentation and
  part-extraction baselines by a large margin. We also showcase downstream
  applications such as 3D part editing.
paper_authors._id: 676baa15295f85d93eb48f24
paper_authors.name: Minghao Chen
paper_authors._id: 676baa15295f85d93eb48f25
paper_authors.name: Roman Shapovalov
paper_authors._id: 676baa15295f85d93eb48f26
paper_authors.name: Iro Laina
paper_authors._id: 676baa15295f85d93eb48f27
paper_authors.name: Tom Monnier
paper_authors._id: 676baa15295f85d93eb48f28
paper_authors.name: Jianyuan Wang
paper_authors._id: 676baa15295f85d93eb48f29
paper_authors.name: David Novotny
paper_authors._id: 676baa15295f85d93eb48f2a
paper_authors.name: Andrea Vedaldi
paper_title: PartGen: Part-level 3D Generation and Reconstruction with Multi-View
    Diffusion Models
paper_summary: Text- or image-to-3D generators and 3D scanners can now produce 3D assets
  with high-quality shapes and textures. These assets typically consist of a
  single, fused representation, like an implicit neural field, a Gaussian
  mixture, or a mesh, without any useful structure. However, most applications
  and creative workflows require assets to be made of several meaningful parts
  that can be manipulated independently. To address this gap, we introduce
  PartGen, a novel approach that generates 3D objects composed of meaningful
  parts starting from text, an image, or an unstructured 3D object. First, given
  multiple views of a 3D object, generated or rendered, a multi-view diffusion
  model extracts a set of plausible and view-consistent part segmentations,
  dividing the object into parts. Then, a second multi-view diffusion model takes
  each part separately, fills in the occlusions, and uses those completed views
  for 3D reconstruction by feeding them to a 3D reconstruction network. This
  completion process considers the context of the entire object to ensure that
  the parts integrate cohesively. The generative completion model can make up for
  the information missing due to occlusions; in extreme cases, it can hallucinate
  entirely invisible parts based on the input 3D asset. We evaluate our method on
  generated and real 3D assets and show that it outperforms segmentation and
  part-extraction baselines by a large margin. We also showcase downstream
  applications such as 3D part editing.
paper_ai_summary: PartGen generates 3D objects composed of meaningful parts from text, images, or unstructured 3D assets using multi-view diffusion models for segmentation and completion.
paper_ai_keywords: diffusion models part segmentations 3D reconstruction multi-view diffusion model 3D part editing