Dolphins: Multimodal Language Model for Driving
The quest for fully autonomous vehicles (AVs) capable of navigating complex
real-world scenarios with human-like understanding and responsiveness. In this
paper, we introduce Dolphins, a novel vision-language model architected to
imbibe human-like abilities as a conversational driving assistant. Dolphins is
adept at processing multimodal inputs comprising video (or image) data, text
instructions, and historical control signals to generate informed outputs
corresponding to the provided instructions. Building upon the open-sourced
pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's
reasoning capabilities through an innovative Grounded Chain of Thought (GCoT)
process. Then we tailored Dolphins to the driving domain by constructing
driving-specific instruction data and conducting instruction tuning. Through
the utilization of the BDD-X dataset, we designed and consolidated four
distinct AV tasks into Dolphins to foster a holistic understanding of intricate
driving scenarios. As a result, the distinctive features of Dolphins are
characterized into two dimensions: (1) the ability to provide a comprehensive
understanding of complex and long-tailed open-world driving scenarios and solve
a spectrum of AV tasks, and (2) the emergence of human-like capabilities
including gradient-free instant adaptation via in-context learning and error
recovery via reflection.
656d333c9496f21be8a012ae
Yingzi Ma
656d333c9496f21be8a012af
Yulong Cao
admin_assigned
2023-12-04T09:54:57.743Z
6553968b5f3ee7fac18e53d4
/avatars/b7ade6719c2c6ed1bfb38aaa14f98654.svg
Yulong Cao
user
yulongc
656d333c9496f21be8a012b0
Jiachen Sun
656d333c9496f21be8a012b1
Marco Pavone
656d333c9496f21be8a012b2
Chaowei Xiao
Dolphins: Multimodal Language Model for Driving
The quest for fully autonomous vehicles (AVs) capable of navigating complex
real-world scenarios with human-like understanding and responsiveness. In this
paper, we introduce Dolphins, a novel vision-language model architected to
imbibe human-like abilities as a conversational driving assistant. Dolphins is
adept at processing multimodal inputs comprising video (or image) data, text
instructions, and historical control signals to generate informed outputs
corresponding to the provided instructions. Building upon the open-sourced
pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's
reasoning capabilities through an innovative Grounded Chain of Thought (GCoT)
process. Then we tailored Dolphins to the driving domain by constructing
driving-specific instruction data and conducting instruction tuning. Through
the utilization of the BDD-X dataset, we designed and consolidated four
distinct AV tasks into Dolphins to foster a holistic understanding of intricate
driving scenarios. As a result, the distinctive features of Dolphins are
characterized into two dimensions: (1) the ability to provide a comprehensive
understanding of complex and long-tailed open-world driving scenarios and solve
a spectrum of AV tasks, and (2) the emergence of human-like capabilities
including gradient-free instant adaptation via in-context learning and error
recovery via reflection.
Dolphins, a vision-language model enhanced with Grounded Chain of Thought, provides human-like capabilities as a conversational driving assistant by processing multimodal inputs and tailoring to specific driving tasks.
Vision-Language Model Grounded Chain of Thought multimodal inputs vision-language in-context learning error recovery