title: Dolphins: Multimodal Language Model for Driving
summary: The quest for fully autonomous vehicles (AVs) capable of navigating complex
  real-world scenarios with human-like understanding and responsiveness. In this
  paper, we introduce Dolphins, a novel vision-language model architected to
  imbibe human-like abilities as a conversational driving assistant. Dolphins is
  adept at processing multimodal inputs comprising video (or image) data, text
  instructions, and historical control signals to generate informed outputs
  corresponding to the provided instructions. Building upon the open-sourced
  pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's
  reasoning capabilities through an innovative Grounded Chain of Thought (GCoT)
  process. Then we tailored Dolphins to the driving domain by constructing
  driving-specific instruction data and conducting instruction tuning. Through
  the utilization of the BDD-X dataset, we designed and consolidated four
  distinct AV tasks into Dolphins to foster a holistic understanding of intricate
  driving scenarios. As a result, the distinctive features of Dolphins are
  characterized into two dimensions: (1) the ability to provide a comprehensive
  understanding of complex and long-tailed open-world driving scenarios and solve
  a spectrum of AV tasks, and (2) the emergence of human-like capabilities
  including gradient-free instant adaptation via in-context learning and error
  recovery via reflection.
paper_authors._id: 656d333c9496f21be8a012ae
paper_authors.name: Yingzi Ma
paper_authors._id: 656d333c9496f21be8a012af
paper_authors.name: Yulong Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:54:57.743Z
paper_authors._id: 6553968b5f3ee7fac18e53d4
paper_authors.avatarUrl: /avatars/b7ade6719c2c6ed1bfb38aaa14f98654.svg
paper_authors.fullname: Yulong Cao
paper_authors.type: user
paper_authors.user: yulongc
paper_authors._id: 656d333c9496f21be8a012b0
paper_authors.name: Jiachen Sun
paper_authors._id: 656d333c9496f21be8a012b1
paper_authors.name: Marco Pavone
paper_authors._id: 656d333c9496f21be8a012b2
paper_authors.name: Chaowei Xiao
paper_title: Dolphins: Multimodal Language Model for Driving
paper_summary: The quest for fully autonomous vehicles (AVs) capable of navigating complex
  real-world scenarios with human-like understanding and responsiveness. In this
  paper, we introduce Dolphins, a novel vision-language model architected to
  imbibe human-like abilities as a conversational driving assistant. Dolphins is
  adept at processing multimodal inputs comprising video (or image) data, text
  instructions, and historical control signals to generate informed outputs
  corresponding to the provided instructions. Building upon the open-sourced
  pretrained Vision-Language Model, OpenFlamingo, we first enhance Dolphins's
  reasoning capabilities through an innovative Grounded Chain of Thought (GCoT)
  process. Then we tailored Dolphins to the driving domain by constructing
  driving-specific instruction data and conducting instruction tuning. Through
  the utilization of the BDD-X dataset, we designed and consolidated four
  distinct AV tasks into Dolphins to foster a holistic understanding of intricate
  driving scenarios. As a result, the distinctive features of Dolphins are
  characterized into two dimensions: (1) the ability to provide a comprehensive
  understanding of complex and long-tailed open-world driving scenarios and solve
  a spectrum of AV tasks, and (2) the emergence of human-like capabilities
  including gradient-free instant adaptation via in-context learning and error
  recovery via reflection.
paper_ai_keywords: vision-language model Grounded Chain of Thought (GCoT) OpenFlamingo multimodal inputs video image data text instructions historical control signals driving-specific instruction data instruction tuning BDD-X dataset AV tasks in-context learning error recovery gradient-free instant adaptation