title: Part123: Part-aware 3D Reconstruction from a Single-view Image
summary: Recently, the emergence of diffusion models has opened up new opportunities
  for single-view reconstruction. However, all the existing methods represent the
  target object as a closed mesh devoid of any structural information, thus
  neglecting the part-based structure, which is crucial for many downstream
  applications, of the reconstructed shape. Moreover, the generated meshes
  usually suffer from large noises, unsmooth surfaces, and blurry textures,
  making it challenging to obtain satisfactory part segments using 3D
  segmentation techniques. In this paper, we present Part123, a novel framework
  for part-aware 3D reconstruction from a single-view image. We first use
  diffusion models to generate multiview-consistent images from a given image,
  and then leverage Segment Anything Model (SAM), which demonstrates powerful
  generalization ability on arbitrary objects, to generate multiview segmentation
  masks. To effectively incorporate 2D part-based information into 3D
  reconstruction and handle inconsistency, we introduce contrastive learning into
  a neural rendering framework to learn a part-aware feature space based on the
  multiview segmentation masks. A clustering-based algorithm is also developed to
  automatically derive 3D part segmentation results from the reconstructed
  models. Experiments show that our method can generate 3D models with
  high-quality segmented parts on various objects. Compared to existing
  unstructured reconstruction methods, the part-aware 3D models from our method
  benefit some important applications, including feature-preserving
  reconstruction, primitive fitting, and 3D shape editing.
paper_authors._id: 66554dce61422fc897949f6b
paper_authors.name: Anran Liu
paper_authors._id: 66554dce61422fc897949f6c
paper_authors.name: Cheng Lin
paper_authors._id: 66554dce61422fc897949f6d
paper_authors.name: Yuan Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-28T04:22:39.242Z
paper_authors._id: 63f5dc824b831cc179b90e8f
paper_authors.avatarUrl: /avatars/2c2d22c6b605a9c412a9c37960efeeb2.svg
paper_authors.fullname: Yuan Liu
paper_authors.type: user
paper_authors.user: liuyuan-pal
paper_authors._id: 66554dce61422fc897949f6e
paper_authors.name: Xiaoxiao Long
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:36:32.555Z
paper_authors._id: 648ebb653c8611c2ba1dbede
paper_authors.avatarUrl: /avatars/dddaf7ae8d76e733b30d9952563ea22b.svg
paper_authors.fullname: xiaoxiaolongli
paper_authors.type: user
paper_authors.user: xiaoxiaolong
paper_authors._id: 66554dce61422fc897949f6f
paper_authors.name: Zhiyang Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T10:36:41.632Z
paper_authors._id: 645223fb01d7bd9555ea399a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
paper_authors.fullname: Zhiyang Dou
paper_authors.type: user
paper_authors.user: frankzydou
paper_authors._id: 66554dce61422fc897949f70
paper_authors.name: Hao-Xiang Guo
paper_authors._id: 66554dce61422fc897949f71
paper_authors.name: Ping Luo
paper_authors._id: 66554dce61422fc897949f72
paper_authors.name: Wenping Wang
paper_title: Part123: Part-aware 3D Reconstruction from a Single-view Image
paper_summary: Recently, the emergence of diffusion models has opened up new opportunities
  for single-view reconstruction. However, all the existing methods represent the
  target object as a closed mesh devoid of any structural information, thus
  neglecting the part-based structure, which is crucial for many downstream
  applications, of the reconstructed shape. Moreover, the generated meshes
  usually suffer from large noises, unsmooth surfaces, and blurry textures,
  making it challenging to obtain satisfactory part segments using 3D
  segmentation techniques. In this paper, we present Part123, a novel framework
  for part-aware 3D reconstruction from a single-view image. We first use
  diffusion models to generate multiview-consistent images from a given image,
  and then leverage Segment Anything Model (SAM), which demonstrates powerful
  generalization ability on arbitrary objects, to generate multiview segmentation
  masks. To effectively incorporate 2D part-based information into 3D
  reconstruction and handle inconsistency, we introduce contrastive learning into
  a neural rendering framework to learn a part-aware feature space based on the
  multiview segmentation masks. A clustering-based algorithm is also developed to
  automatically derive 3D part segmentation results from the reconstructed
  models. Experiments show that our method can generate 3D models with
  high-quality segmented parts on various objects. Compared to existing
  unstructured reconstruction methods, the part-aware 3D models from our method
  benefit some important applications, including feature-preserving
  reconstruction, primitive fitting, and 3D shape editing.
paper_ai_keywords: diffusion models single-view reconstruction closed mesh part-based structure multiview-consistent images Segment Anything Model (SAM) multiview segmentation masks contrastive learning neural rendering framework part-aware feature space clustering-based algorithm 3D part segmentation feature-preserving reconstruction primitive fitting 3D shape editing