Point-Cloud Completion with Pretrained Text-to-image Diffusion Models
Point-cloud data collected in real-world applications are often incomplete.
Data is typically missing due to objects being observed from partial
viewpoints, which only capture a specific perspective or angle. Additionally,
data can be incomplete due to occlusion and low-resolution sampling. Existing
completion approaches rely on datasets of predefined objects to guide the
completion of noisy and incomplete, point clouds. However, these approaches
perform poorly when tested on Out-Of-Distribution (OOD) objects, that are
poorly represented in the training dataset. Here we leverage recent advances in
text-guided image generation, which lead to major breakthroughs in text-guided
shape generation. We describe an approach called SDS-Complete that uses a
pre-trained text-to-image diffusion model and leverages the text semantics of a
given incomplete point cloud of an object, to obtain a complete surface
representation. SDS-Complete can complete a variety of objects using test-time
optimization without expensive collection of 3D information. We evaluate SDS
Complete on incomplete scanned objects, captured by real-world depth sensors
and LiDAR scanners. We find that it effectively reconstructs objects that are
absent from common datasets, reducing Chamfer loss by 50% on average compared
with current methods. Project page: https://sds-complete.github.io/
64927091b4ab18f4467522bf
Yoni Kasten
admin_assigned
2023-06-21T15:50:02.284Z
642294c112b4b51aae368d30
/avatars/017c19910ba01d6bd9cd864132652448.svg
Yoni Kasten
user
ykasten
64927091b4ab18f4467522c0
Ohad Rahamim
claimed_verified
2025-01-01T20:15:14.988Z
63c59c3a6d132b995fedface
/avatars/4e18b19e477cb683ce1ba3ae6ab77d8e.svg
Ohad rahamim
user
ohad204
64927091b4ab18f4467522c1
Gal Chechik
claimed_verified
2023-06-21T20:46:26.908Z
6493393f357b252af72196c5
https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
Gal Chechik
user
galchechik
Point-Cloud Completion with Pretrained Text-to-image Diffusion Models
Point-cloud data collected in real-world applications are often incomplete.
Data is typically missing due to objects being observed from partial
viewpoints, which only capture a specific perspective or angle. Additionally,
data can be incomplete due to occlusion and low-resolution sampling. Existing
completion approaches rely on datasets of predefined objects to guide the
completion of noisy and incomplete, point clouds. However, these approaches
perform poorly when tested on Out-Of-Distribution (OOD) objects, that are
poorly represented in the training dataset. Here we leverage recent advances in
text-guided image generation, which lead to major breakthroughs in text-guided
shape generation. We describe an approach called SDS-Complete that uses a
pre-trained text-to-image diffusion model and leverages the text semantics of a
given incomplete point cloud of an object, to obtain a complete surface
representation. SDS-Complete can complete a variety of objects using test-time
optimization without expensive collection of 3D information. We evaluate SDS
Complete on incomplete scanned objects, captured by real-world depth sensors
and LiDAR scanners. We find that it effectively reconstructs objects that are
absent from common datasets, reducing Chamfer loss by 50% on average compared
with current methods. Project page: https://sds-complete.github.io/
SDS-Complete uses a text-to-image diffusion model to reconstruct incomplete point clouds, achieving reduced Chamfer loss on out-of-distribution objects.
text-to-image diffusion model text semantics test-time optimization Chamfer loss