A Review of 3D Object Detection with Vision-Language Models

This paper presents a groundbreaking and comprehensive review, the first of its kind, focused on 3D object detection with Vision-Language Models (VLMs), a rapidly advancing frontier in multimodal AI. Using a hybrid search strategy combining academic databases and AI-powered engines, we curated and analyzed over 100 state-of-the-art papers. Our study begins by contextualizing 3D object detection within traditional pipelines, examining methods like PointNet++, PV-RCNN, and VoteNet that utilize point clouds and voxel grids for geometric inference. We then trace the shift toward VLM-driven systems, where models such as CLIP, PaLM-E, and RoboFlamingo-Plus enhance spatial understanding through language-guided reasoning, zero-shot generalization, and instruction-based interaction. We investigate the architectural foundations enabling this transition, including pretraining techniques, spatial alignment modules, and cross-modal fusion strategies. Visualizations and benchmark comparisons reveal VLMs’ unique capabilities in semantic abstraction and open-vocabulary detection, despite trade-offs in speed and annotation cost. Our comparative synthesis highlights key challenges such as spatial misalignment, occlusion sensitivity, and limited real-time viability, alongside emerging solutions like 3D scene graphs, synthetic captioning, and multimodal reinforcement learning. This review not only consolidates the technical landscape of VLM-based 3D detection but also provides a forward-looking roadmap, identifying promising innovations and deployment opportunities. It serves as a foundational reference for researchers seeking to harness the power of language-guided 3D perception in robotics, AR, and embodied AI. A project associated with this review and evaluation has been created at Github Link: https://github.com/r4hul77/Awesome-3D-Detection-Based-on-VLMs