V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with
  Multi-Modal Large Language Models
Current autonomous driving vehicles rely mainly on their individual sensors
to understand surrounding scenes and plan for future trajectories, which can be
unreliable when the sensors are malfunctioning or occluded. To address this
problem, cooperative perception methods via vehicle-to-vehicle (V2V)
communication have been proposed, but they have tended to focus on detection
and tracking. How those approaches contribute to overall cooperative planning
performance is still under-explored. Inspired by recent progress using Large
Language Models (LLMs) to build autonomous driving systems, we propose a novel
problem setting that integrates an LLM into cooperative autonomous driving,
with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and
benchmark. We also propose our baseline method Vehicle-to-Vehicle Large
Language Model (V2V-LLM), which uses an LLM to fuse perception information from
multiple connected autonomous vehicles (CAVs) and answer driving-related
questions: grounding, notable object identification, and planning. Experimental
results show that our proposed V2V-LLM can be a promising unified model
architecture for performing various tasks in cooperative autonomous driving,
and outperforms other baseline methods that use different fusion approaches.
Our work also creates a new research direction that can improve the safety of
future autonomous driving systems. Our project website:
https://eddyhkchiu.github.io/v2vllm.github.io/ .
67b2d7e86a002d59a415fc99
Hsu-kuang Chiu
67b2d7e86a002d59a415fc9a
Ryo Hachiuma
67b2d7e86a002d59a415fc9b
Chien-Yi Wang
67b2d7e86a002d59a415fc9c
Stephen F. Smith
67b2d7e86a002d59a415fc9d
Yu-Chiang Frank Wang
67b2d7e86a002d59a415fc9e
Min-Hung Chen
claimed_verified
2025-02-18T09:32:44.929Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with
  Multi-Modal Large Language Models
Current autonomous driving vehicles rely mainly on their individual sensors
to understand surrounding scenes and plan for future trajectories, which can be
unreliable when the sensors are malfunctioning or occluded. To address this
problem, cooperative perception methods via vehicle-to-vehicle (V2V)
communication have been proposed, but they have tended to focus on detection
and tracking. How those approaches contribute to overall cooperative planning
performance is still under-explored. Inspired by recent progress using Large
Language Models (LLMs) to build autonomous driving systems, we propose a novel
problem setting that integrates an LLM into cooperative autonomous driving,
with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and
benchmark. We also propose our baseline method Vehicle-to-Vehicle Large
Language Model (V2V-LLM), which uses an LLM to fuse perception information from
multiple connected autonomous vehicles (CAVs) and answer driving-related
questions: grounding, notable object identification, and planning. Experimental
results show that our proposed V2V-LLM can be a promising unified model
architecture for performing various tasks in cooperative autonomous driving,
and outperforms other baseline methods that use different fusion approaches.
Our work also creates a new research direction that can improve the safety of
future autonomous driving systems. Our project website:
https://eddyhkchiu.github.io/v2vllm.github.io/ .
A Vehicle-to-Vehicle Question-Answering (V2V-QA) framework integrates Large Language Models (LLMs) for cooperative autonomous driving, improving safety and performance through information fusion and task handling.
Large Language Models LLMs cooperative autonomous driving Vehicle-to-Vehicle communication V2V V2V-QA Vehicle-to-Vehicle Large Language Model V2V-LLM cooperative planning autonomous vehicles connected autonomous vehicles CAVs perception information fusion driving-related questions grounding notable object identification planning