title: On the Road with GPT-4V(ision): Early Explorations of Visual-Language
    Model on Autonomous Driving
summary: The pursuit of autonomous driving technology hinges on the sophisticated
  integration of perception, decision-making, and control systems. Traditional
  approaches, both data-driven and rule-based, have been hindered by their
  inability to grasp the nuance of complex driving environments and the
  intentions of other road users. This has been a significant bottleneck,
  particularly in the development of common sense reasoning and nuanced scene
  understanding necessary for safe and reliable autonomous driving. The advent of
  Visual Language Models (VLM) represents a novel frontier in realizing fully
  autonomous vehicle driving. This report provides an exhaustive evaluation of
  the latest state-of-the-art VLM, \modelnamefull, and its application in
  autonomous driving scenarios. We explore the model's abilities to understand
  and reason about driving scenes, make decisions, and ultimately act in the
  capacity of a driver. Our comprehensive tests span from basic scene recognition
  to complex causal reasoning and real-time decision-making under varying
  conditions. Our findings reveal that \modelname demonstrates superior
  performance in scene understanding and causal reasoning compared to existing
  autonomous systems. It showcases the potential to handle out-of-distribution
  scenarios, recognize intentions, and make informed decisions in real driving
  contexts. However, challenges remain, particularly in direction discernment,
  traffic light recognition, vision grounding, and spatial reasoning tasks. These
  limitations underscore the need for further research and development. Project
  is now available on GitHub for interested parties to access and utilize:
  https://github.com/PJLab-ADG/GPT4V-AD-Exploration
paper_authors._id: 654dbabb0011607c31213778
paper_authors.name: Licheng Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:44:06.784Z
paper_authors._id: 64a7c43ae940d769194055df
paper_authors.avatarUrl: /avatars/441ccadd62e039fb8cb112f138ed917d.svg
paper_authors.fullname: Licheng Wen
paper_authors.type: user
paper_authors.user: Wayne-lc
paper_authors._id: 654dbabb0011607c31213779
paper_authors.name: Xuemeng Yang
paper_authors._id: 654dbabb0011607c3121377a
paper_authors.name: Daocheng Fu
paper_authors._id: 654dbabb0011607c3121377b
paper_authors.name: Xiaofeng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:32:24.336Z
paper_authors._id: 6426616ea5ec4a5cbc535634
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/hH6JsxnXeakH3mBTeNNmO.jpeg
paper_authors.fullname: JeffWang
paper_authors.type: user
paper_authors.user: Jeff-Wang
paper_authors._id: 654dbabb0011607c3121377c
paper_authors.name: Pinlong Cai
paper_authors._id: 654dbabb0011607c3121377d
paper_authors.name: Xin Li
paper_authors._id: 654dbabb0011607c3121377e
paper_authors.name: Tao Ma
paper_authors._id: 654dbabb0011607c3121377f
paper_authors.name: Yingxuan Li
paper_authors._id: 654dbabb0011607c31213780
paper_authors.name: Linran Xu
paper_authors._id: 654dbabb0011607c31213781
paper_authors.name: Dengke Shang
paper_authors._id: 654dbabb0011607c31213782
paper_authors.name: Zheng Zhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-05T07:00:29.237Z
paper_authors._id: 656e9b562cd7a3e348011d26
paper_authors.avatarUrl: /avatars/bcca51bdc27c664f8f132420e6ed99fa.svg
paper_authors.fullname: Zheng Zhu
paper_authors.type: user
paper_authors.user: ZhengZhu
paper_authors._id: 654dbabb0011607c31213783
paper_authors.name: Shaoyan Sun
paper_authors._id: 654dbabb0011607c31213784
paper_authors.name: Yeqi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:46:32.780Z
paper_authors._id: 634ff92b4a9fc01602f4f379
paper_authors.avatarUrl: /avatars/d121a9599ed05016c251e67b280fc882.svg
paper_authors.fullname: Skura
paper_authors.type: user
paper_authors.user: baiyeqi
paper_authors._id: 654dbabb0011607c31213785
paper_authors.name: Xinyu Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:46:16.994Z
paper_authors._id: 64521a2a7ba5cb58fabefcae
paper_authors.avatarUrl: /avatars/caff912e95348c36b34dfc4f1509d42b.svg
paper_authors.fullname: XinyuCai
paper_authors.type: user
paper_authors.user: XinyuCai
paper_authors._id: 654dbabb0011607c31213786
paper_authors.name: Min Dou
paper_authors._id: 654dbabb0011607c31213787
paper_authors.name: Shuanglu Hu
paper_authors._id: 654dbabb0011607c31213788
paper_authors.name: Botian Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:45:52.609Z
paper_authors._id: 643df87f7cd64d872cb9fabd
paper_authors.avatarUrl: /avatars/c53bfabcee08de448dde973915e8b31d.svg
paper_authors.fullname: Botian Shi
paper_authors.type: user
paper_authors.user: friskit
paper_title: On the Road with GPT-4V(ision): Early Explorations of Visual-Language
    Model on Autonomous Driving
paper_summary: The pursuit of autonomous driving technology hinges on the sophisticated
  integration of perception, decision-making, and control systems. Traditional
  approaches, both data-driven and rule-based, have been hindered by their
  inability to grasp the nuance of complex driving environments and the
  intentions of other road users. This has been a significant bottleneck,
  particularly in the development of common sense reasoning and nuanced scene
  understanding necessary for safe and reliable autonomous driving. The advent of
  Visual Language Models (VLM) represents a novel frontier in realizing fully
  autonomous vehicle driving. This report provides an exhaustive evaluation of
  the latest state-of-the-art VLM, \modelnamefull, and its application in
  autonomous driving scenarios. We explore the model's abilities to understand
  and reason about driving scenes, make decisions, and ultimately act in the
  capacity of a driver. Our comprehensive tests span from basic scene recognition
  to complex causal reasoning and real-time decision-making under varying
  conditions. Our findings reveal that \modelname demonstrates superior
  performance in scene understanding and causal reasoning compared to existing
  autonomous systems. It showcases the potential to handle out-of-distribution
  scenarios, recognize intentions, and make informed decisions in real driving
  contexts. However, challenges remain, particularly in direction discernment,
  traffic light recognition, vision grounding, and spatial reasoning tasks. These
  limitations underscore the need for further research and development. Project
  is now available on GitHub for interested parties to access and utilize:
  https://github.com/PJLab-ADG/GPT4V-AD-Exploration
paper_ai_summary: Visual Language Models (VLMs) demonstrate superior scene understanding and causal reasoning in autonomous driving but face challenges in specific vision tasks like direction discernment and traffic light recognition.
paper_ai_keywords: Visual Language Models VLMs scene understanding causal reasoning autonomous driving real-time decision-making out-of-distribution scenarios intention recognition spatial reasoning vision grounding