title: PandaGPT: One Model To Instruction-Follow Them All
summary: We present PandaGPT, an approach to emPower large lANguage moDels with visual
  and Auditory instruction-following capabilities. Our pilot experiments show
  that PandaGPT can perform complex tasks such as detailed image description
  generation, writing stories inspired by videos, and answering questions about
  audios. More interestingly, PandaGPT can take multimodal inputs simultaneously
  and compose their semantics naturally. For example, PandaGPT can connect how
  objects look in an image/video and how they sound in an audio. To do so,
  PandaGPT combines the multimodal encoders from ImageBind and the large language
  models from Vicuna. Notably, only aligned image-text pairs are required for the
  training of PandaGPT. Thanks to the strong capability of ImageBind in embedding
  data from different modalities into the same space, PandaGPT displays emergent,
  i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,
  video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an
  initial step toward building AGI that can perceive and understand inputs in
  different modalities holistically, as we humans do. Our project page is at
  https://panda-gpt.github.io/.
paper_authors._id: 6474192abf9b32c6bbcd081a
paper_authors.name: Yixuan Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:00.810Z
paper_authors._id: 6144c40da8f4f7baca305243
paper_authors.avatarUrl: /avatars/1f9bbb70f859f34563fdd0a7de4cd254.svg
paper_authors.fullname: Yixuan Su
paper_authors.type: user
paper_authors.user: pangpang666
paper_authors._id: 6474192abf9b32c6bbcd081b
paper_authors.name: Tian Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T10:11:57.273Z
paper_authors._id: 627ca439a09edfe62239c671
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652335660508-noauth.jpeg
paper_authors.fullname: Tian Lan
paper_authors.type: user
paper_authors.user: GMFTBY
paper_authors._id: 6474192abf9b32c6bbcd081c
paper_authors.name: Huayang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:43.811Z
paper_authors._id: 64672e286192d3914221d633
paper_authors.avatarUrl: /avatars/a26908ebcbb74feaf44fe83a44fef8ac.svg
paper_authors.fullname: Huayang
paper_authors.type: user
paper_authors.user: huayangli
paper_authors._id: 6474192abf9b32c6bbcd081d
paper_authors.name: Jialu Xu
paper_authors._id: 6474192abf9b32c6bbcd081e
paper_authors.name: Yan Wang
paper_authors._id: 6474192abf9b32c6bbcd081f
paper_authors.name: Deng Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:40.790Z
paper_authors._id: 646735a1946476c5d214e5b2
paper_authors.avatarUrl: /avatars/23392fda23fa6288fcb309f7eaa73472.svg
paper_authors.fullname: Deng Cai
paper_authors.type: user
paper_authors.user: jcyk
paper_title: PandaGPT: One Model To Instruction-Follow Them All
paper_summary: We present PandaGPT, an approach to emPower large lANguage moDels with visual
  and Auditory instruction-following capabilities. Our pilot experiments show
  that PandaGPT can perform complex tasks such as detailed image description
  generation, writing stories inspired by videos, and answering questions about
  audios. More interestingly, PandaGPT can take multimodal inputs simultaneously
  and compose their semantics naturally. For example, PandaGPT can connect how
  objects look in an image/video and how they sound in an audio. To do so,
  PandaGPT combines the multimodal encoders from ImageBind and the large language
  models from Vicuna. Notably, only aligned image-text pairs are required for the
  training of PandaGPT. Thanks to the strong capability of ImageBind in embedding
  data from different modalities into the same space, PandaGPT displays emergent,
  i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g.,
  video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an
  initial step toward building AGI that can perceive and understand inputs in
  different modalities holistically, as we humans do. Our project page is at
  https://panda-gpt.github.io/.
paper_ai_keywords: ImageBind Vicuna multimodal encoders aligned image-text pairs cross-modal behaviors AGI multimodal inputs zero-shot perceptual encodings