DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
<|reference_start|>DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models: A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.<|reference_end|>
@article{tian2024drivevlm:,
    title={DriveVLM: The Convergence of Autonomous Driving and Large
  Vision-Language Models},
    author={Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong
  Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, Hang Zhao},
    journal={arXiv preprint arXiv:2402.12289},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.12289},
    primaryClass={cs.CV}
}