Embodied Understanding of Driving Scenarios
<|reference_start|>Embodied Understanding of Driving Scenarios: Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared.<|reference_end|>
@article{zhou2024embodied,
    title={Embodied Understanding of Driving Scenarios},
    author={Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu,
  Hongzi Zhu, Minyi Guo, Yu Qiao, Hongyang Li},
    journal={arXiv preprint arXiv:2403.04593},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04593},
    primaryClass={cs.CV}
}