Part123: Part-aware 3D Reconstruction from a Single-view Image
Recently, the emergence of diffusion models has opened up new opportunities
for single-view reconstruction. However, all the existing methods represent the
target object as a closed mesh devoid of any structural information, thus
neglecting the part-based structure, which is crucial for many downstream
applications, of the reconstructed shape. Moreover, the generated meshes
usually suffer from large noises, unsmooth surfaces, and blurry textures,
making it challenging to obtain satisfactory part segments using 3D
segmentation techniques. In this paper, we present Part123, a novel framework
for part-aware 3D reconstruction from a single-view image. We first use
diffusion models to generate multiview-consistent images from a given image,
and then leverage Segment Anything Model (SAM), which demonstrates powerful
generalization ability on arbitrary objects, to generate multiview segmentation
masks. To effectively incorporate 2D part-based information into 3D
reconstruction and handle inconsistency, we introduce contrastive learning into
a neural rendering framework to learn a part-aware feature space based on the
multiview segmentation masks. A clustering-based algorithm is also developed to
automatically derive 3D part segmentation results from the reconstructed
models. Experiments show that our method can generate 3D models with
high-quality segmented parts on various objects. Compared to existing
unstructured reconstruction methods, the part-aware 3D models from our method
benefit some important applications, including feature-preserving
reconstruction, primitive fitting, and 3D shape editing.
66554dce61422fc897949f6b
Anran Liu
66554dce61422fc897949f6c
Cheng Lin
66554dce61422fc897949f6d
Yuan Liu
extracted_confirmed
2024-05-28T04:22:39.242Z
63f5dc824b831cc179b90e8f
/avatars/2c2d22c6b605a9c412a9c37960efeeb2.svg
Yuan Liu
user
liuyuan-pal
66554dce61422fc897949f6e
Xiaoxiao Long
admin_assigned
2024-05-28T10:36:32.555Z
648ebb653c8611c2ba1dbede
/avatars/dddaf7ae8d76e733b30d9952563ea22b.svg
xiaoxiaolongli
user
xiaoxiaolong
66554dce61422fc897949f6f
Zhiyang Dou
admin_assigned
2024-05-28T10:36:41.632Z
645223fb01d7bd9555ea399a
https://cdn-avatars.huggingface.co/v1/production/uploads/645223fb01d7bd9555ea399a/fVR7XmGg6pMSRKx9sEdvT.png
Zhiyang Dou
user
frankzydou
66554dce61422fc897949f70
Hao-Xiang Guo
66554dce61422fc897949f71
Ping Luo
66554dce61422fc897949f72
Wenping Wang
Part123: Part-aware 3D Reconstruction from a Single-view Image
Recently, the emergence of diffusion models has opened up new opportunities
for single-view reconstruction. However, all the existing methods represent the
target object as a closed mesh devoid of any structural information, thus
neglecting the part-based structure, which is crucial for many downstream
applications, of the reconstructed shape. Moreover, the generated meshes
usually suffer from large noises, unsmooth surfaces, and blurry textures,
making it challenging to obtain satisfactory part segments using 3D
segmentation techniques. In this paper, we present Part123, a novel framework
for part-aware 3D reconstruction from a single-view image. We first use
diffusion models to generate multiview-consistent images from a given image,
and then leverage Segment Anything Model (SAM), which demonstrates powerful
generalization ability on arbitrary objects, to generate multiview segmentation
masks. To effectively incorporate 2D part-based information into 3D
reconstruction and handle inconsistency, we introduce contrastive learning into
a neural rendering framework to learn a part-aware feature space based on the
multiview segmentation masks. A clustering-based algorithm is also developed to
automatically derive 3D part segmentation results from the reconstructed
models. Experiments show that our method can generate 3D models with
high-quality segmented parts on various objects. Compared to existing
unstructured reconstruction methods, the part-aware 3D models from our method
benefit some important applications, including feature-preserving
reconstruction, primitive fitting, and 3D shape editing.
Part123 framework uses diffusion models and contrastive learning within neural rendering to produce part-aware 3D reconstructions from single-view images with high-quality segmentation.
diffusion models Segment Anything Model (SAM) contrastive learning neural rendering 3D reconstruction part-wise segmentation feature-preserving reconstruction primitive fitting 3D shape editing