  Training a 3D scene understanding model requires complicated human
annotations, which are laborious to collect and result in a model only encoding
close-set object semantics. In contrast, vision-language pre-training models
(e.g., CLIP) have shown remarkable open-world reasoning properties. To this
end, we propose directly transferring CLIP's feature space to 3D scene
understanding model without any form of supervision. We first modify CLIP's
input and forwarding process so that it can be adapted to extract dense pixel
features for 3D scene contents. We then project multi-view image features to
the point cloud and train a 3D scene understanding model with feature
distillation. Without any annotations or additional training, our model
achieves promising annotation-free semantic segmentation results on
open-vocabulary semantics and long-tailed concepts. Besides, serving as a
cross-modal pre-training framework, our method can be used to improve data
efficiency during fine-tuning. Our model outperforms previous SOTA methods in
various zero-shot and data-efficient learning benchmarks. Most importantly, our
model successfully inherits CLIP's rich-structured knowledge, allowing 3D scene
understanding models to recognize not only object concepts but also open-world
semantics.

Computer Vision and Pattern Recognition