Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases
<|reference_start|>Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases: Large Vision-Language Models (LVLMs) have received widespread attention in advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on the multi-faceted capabilities in natural circumstances, lacking automated and quantifiable assessment for self-driving, let alone the severe road corner cases. In this paper, we propose CODA-LM, the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure to prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotation for human annotators, and for LVLM evaluation, we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover, with CODA-LM, we build CODA-VLM, a new driving LVLM surpassing all the open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.<|reference_end|>
@article{chen2024automated,
    title={Automated Evaluation of Large Vision-Language Models on Self-driving
  Corner Cases},
    author={Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan
  Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, Dit-Yan Yeung, Huchuan
  Lu, Xu Jia},
    journal={arXiv preprint arXiv:2404.10595},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10595},
    primaryClass={cs.CV}
}