title: ImageBind-LLM: Multi-modality Instruction Tuning
summary: We present ImageBind-LLM, a multi-modality instruction tuning method of large
  language models (LLMs) via ImageBind. Existing works mainly focus on language
  and image instruction tuning, different from which, our ImageBind-LLM can
  respond to multi-modality conditions, including audio, 3D point clouds, video,
  and their embedding-space arithmetic by only image-text alignment training.
  During training, we adopt a learnable bind network to align the embedding space
  between LLaMA and ImageBind's image encoder. Then, the image features
  transformed by the bind network are added to word tokens of all layers in
  LLaMA, which progressively injects visual instructions via an attention-free
  and zero-initialized gating mechanism. Aided by the joint embedding of
  ImageBind, the simple image-text training enables our model to exhibit superior
  multi-modality instruction-following capabilities. During inference, the
  multi-modality inputs are fed into the corresponding ImageBind encoders, and
  processed by a proposed visual cache model for further cross-modal embedding
  enhancement. The training-free cache model retrieves from three million image
  features extracted by ImageBind, which effectively mitigates the
  training-inference modality discrepancy. Notably, with our approach,
  ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
  significant language generation quality. Code is released at
  https://github.com/OpenGVLab/LLaMA-Adapter.
paper_authors._id: 64fa84f4b8d50cebd618e52f
paper_authors.name: Jiaming Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-08T15:51:41.647Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 64fa84f4b8d50cebd618e530
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:40:41.530Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 64fa84f4b8d50cebd618e531
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:15:46.591Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 64fa84f4b8d50cebd618e532
paper_authors.name: Peng Gao
paper_authors._id: 64fa84f4b8d50cebd618e533
paper_authors.name: Peng Xu
paper_authors._id: 64fa84f4b8d50cebd618e534
paper_authors.name: Han Xiao
paper_authors._id: 64fa84f4b8d50cebd618e535
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:18:24.403Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 64fa84f4b8d50cebd618e536
paper_authors.name: Chris Liu
paper_authors._id: 64fa84f4b8d50cebd618e537
paper_authors.name: Song Wen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:52.087Z
paper_authors._id: 638d79bf5869d78eb7e8ccec
paper_authors.avatarUrl: /avatars/c047dc15cd650f91a276115d8d1ad10a.svg
paper_authors.fullname: Song Wen
paper_authors.type: user
paper_authors.user: swen9
paper_authors._id: 64fa84f4b8d50cebd618e538
paper_authors.name: Ziyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:21:38.093Z
paper_authors._id: 647d9ab61a1fcad2fdbf2d3d
paper_authors.avatarUrl: /avatars/48c8aeae8979d2c87df8bde922437d62.svg
paper_authors.fullname: Ziyu Guo
paper_authors.type: user
paper_authors.user: ZiyuG
paper_authors._id: 64fa84f4b8d50cebd618e539
paper_authors.name: Xudong Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:22:11.890Z
paper_authors._id: 642e686bbe01b88c9446db8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e686bbe01b88c9446db8b/tb1DKe5xt50ykOeXiUuTE.jpeg
paper_authors.fullname: Lu Xudong
paper_authors.type: user
paper_authors.user: lucky-lance
paper_authors._id: 64fa84f4b8d50cebd618e53a
paper_authors.name: Shuai Ren
paper_authors._id: 64fa84f4b8d50cebd618e53b
paper_authors.name: Yafei Wen
paper_authors._id: 64fa84f4b8d50cebd618e53c
paper_authors.name: Xiaoxin Chen
paper_authors._id: 64fa84f4b8d50cebd618e53d
paper_authors.name: Xiangyu Yue
paper_authors._id: 64fa84f4b8d50cebd618e53e
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:21.127Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 64fa84f4b8d50cebd618e53f
paper_authors.name: Yu Qiao
paper_title: ImageBind-LLM: Multi-modality Instruction Tuning
paper_summary: We present ImageBind-LLM, a multi-modality instruction tuning method of large
  language models (LLMs) via ImageBind. Existing works mainly focus on language
  and image instruction tuning, different from which, our ImageBind-LLM can
  respond to multi-modality conditions, including audio, 3D point clouds, video,
  and their embedding-space arithmetic by only image-text alignment training.
  During training, we adopt a learnable bind network to align the embedding space
  between LLaMA and ImageBind's image encoder. Then, the image features
  transformed by the bind network are added to word tokens of all layers in
  LLaMA, which progressively injects visual instructions via an attention-free
  and zero-initialized gating mechanism. Aided by the joint embedding of
  ImageBind, the simple image-text training enables our model to exhibit superior
  multi-modality instruction-following capabilities. During inference, the
  multi-modality inputs are fed into the corresponding ImageBind encoders, and
  processed by a proposed visual cache model for further cross-modal embedding
  enhancement. The training-free cache model retrieves from three million image
  features extracted by ImageBind, which effectively mitigates the
  training-inference modality discrepancy. Notably, with our approach,
  ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
  significant language generation quality. Code is released at
  https://github.com/OpenGVLab/LLaMA-Adapter.
paper_ai_keywords: ImageBind-LLM multi-modality instruction tuning image-text alignment learnable bind network embedding space image features word tokens attention-free zero-initialized gating mechanism joint embedding multi-modality instruction-following multi-modality inputs ImageBind encoders visual cache model cross-modal embedding enhancement training-free cache model image features training-inference modality discrepancy