title: V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with
    Multi-Modal Large Language Models
summary: Current autonomous driving vehicles rely mainly on their individual sensors
  to understand surrounding scenes and plan for future trajectories, which can be
  unreliable when the sensors are malfunctioning or occluded. To address this
  problem, cooperative perception methods via vehicle-to-vehicle (V2V)
  communication have been proposed, but they have tended to focus on detection
  and tracking. How those approaches contribute to overall cooperative planning
  performance is still under-explored. Inspired by recent progress using Large
  Language Models (LLMs) to build autonomous driving systems, we propose a novel
  problem setting that integrates an LLM into cooperative autonomous driving,
  with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and
  benchmark. We also propose our baseline method Vehicle-to-Vehicle Large
  Language Model (V2V-LLM), which uses an LLM to fuse perception information from
  multiple connected autonomous vehicles (CAVs) and answer driving-related
  questions: grounding, notable object identification, and planning. Experimental
  results show that our proposed V2V-LLM can be a promising unified model
  architecture for performing various tasks in cooperative autonomous driving,
  and outperforms other baseline methods that use different fusion approaches.
  Our work also creates a new research direction that can improve the safety of
  future autonomous driving systems. Our project website:
  https://eddyhkchiu.github.io/v2vllm.github.io/ .
paper_authors._id: 67b2d7e86a002d59a415fc99
paper_authors.name: Hsu-kuang Chiu
paper_authors._id: 67b2d7e86a002d59a415fc9a
paper_authors.name: Ryo Hachiuma
paper_authors._id: 67b2d7e86a002d59a415fc9b
paper_authors.name: Chien-Yi Wang
paper_authors._id: 67b2d7e86a002d59a415fc9c
paper_authors.name: Stephen F. Smith
paper_authors._id: 67b2d7e86a002d59a415fc9d
paper_authors.name: Yu-Chiang Frank Wang
paper_authors._id: 67b2d7e86a002d59a415fc9e
paper_authors.name: Min-Hung Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:44.929Z
paper_authors._id: 64ae22dd1aee69ece065cdcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
paper_authors.fullname: Min-Hung Chen
paper_authors.type: user
paper_authors.user: cmhungsteve
paper_title: V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with
    Multi-Modal Large Language Models
paper_summary: Current autonomous driving vehicles rely mainly on their individual sensors
  to understand surrounding scenes and plan for future trajectories, which can be
  unreliable when the sensors are malfunctioning or occluded. To address this
  problem, cooperative perception methods via vehicle-to-vehicle (V2V)
  communication have been proposed, but they have tended to focus on detection
  and tracking. How those approaches contribute to overall cooperative planning
  performance is still under-explored. Inspired by recent progress using Large
  Language Models (LLMs) to build autonomous driving systems, we propose a novel
  problem setting that integrates an LLM into cooperative autonomous driving,
  with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and
  benchmark. We also propose our baseline method Vehicle-to-Vehicle Large
  Language Model (V2V-LLM), which uses an LLM to fuse perception information from
  multiple connected autonomous vehicles (CAVs) and answer driving-related
  questions: grounding, notable object identification, and planning. Experimental
  results show that our proposed V2V-LLM can be a promising unified model
  architecture for performing various tasks in cooperative autonomous driving,
  and outperforms other baseline methods that use different fusion approaches.
  Our work also creates a new research direction that can improve the safety of
  future autonomous driving systems. Our project website:
  https://eddyhkchiu.github.io/v2vllm.github.io/ .
paper_ai_keywords: Large Language Models (LLMs) Vehicle-to-Vehicle (V2V) communication cooperative autonomous driving Vehicle-to-Vehicle Question-Answering (V2V-QA) Vehicle-to-Vehicle Large Language Model (V2V-LLM) connected autonomous vehicles (CAVs) grounding notable object identification planning