ImageBind-LLM: Multi-modality Instruction Tuning
We present ImageBind-LLM, a multi-modality instruction tuning method of large
language models (LLMs) via ImageBind. Existing works mainly focus on language
and image instruction tuning, different from which, our ImageBind-LLM can
respond to multi-modality conditions, including audio, 3D point clouds, video,
and their embedding-space arithmetic by only image-text alignment training.
During training, we adopt a learnable bind network to align the embedding space
between LLaMA and ImageBind's image encoder. Then, the image features
transformed by the bind network are added to word tokens of all layers in
LLaMA, which progressively injects visual instructions via an attention-free
and zero-initialized gating mechanism. Aided by the joint embedding of
ImageBind, the simple image-text training enables our model to exhibit superior
multi-modality instruction-following capabilities. During inference, the
multi-modality inputs are fed into the corresponding ImageBind encoders, and
processed by a proposed visual cache model for further cross-modal embedding
enhancement. The training-free cache model retrieves from three million image
features extracted by ImageBind, which effectively mitigates the
training-inference modality discrepancy. Notably, with our approach,
ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
significant language generation quality. Code is released at
https://github.com/OpenGVLab/LLaMA-Adapter.
64fa84f4b8d50cebd618e52f
Jiaming Han
claimed_verified
2023-09-08T15:51:41.647Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
64fa84f4b8d50cebd618e530
Renrui Zhang
claimed_verified
2024-03-22T08:40:41.530Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
64fa84f4b8d50cebd618e531
Wenqi Shao
admin_assigned
2023-09-08T11:15:46.591Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
64fa84f4b8d50cebd618e532
Peng Gao
64fa84f4b8d50cebd618e533
Peng Xu
64fa84f4b8d50cebd618e534
Han Xiao
claimed_verified
2025-05-28T09:03:15.745Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
64fa84f4b8d50cebd618e535
Kaipeng Zhang
admin_assigned
2023-09-08T11:18:24.403Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
64fa84f4b8d50cebd618e536
Chris Liu
64fa84f4b8d50cebd618e537
Song Wen
claimed_verified
2024-12-04T09:47:52.087Z
638d79bf5869d78eb7e8ccec
/avatars/c047dc15cd650f91a276115d8d1ad10a.svg
Song Wen
user
swen9
64fa84f4b8d50cebd618e538
Ziyu Guo
admin_assigned
2023-09-08T11:21:38.093Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
64fa84f4b8d50cebd618e539
Xudong Lu
admin_assigned
2023-09-08T11:22:11.890Z
642e686bbe01b88c9446db8b
https://cdn-avatars.huggingface.co/v1/production/uploads/642e686bbe01b88c9446db8b/tb1DKe5xt50ykOeXiUuTE.jpeg
Lu Xudong
user
lucky-lance
64fa84f4b8d50cebd618e53a
Shuai Ren
64fa84f4b8d50cebd618e53b
Yafei Wen
64fa84f4b8d50cebd618e53c
Xiaoxin Chen
64fa84f4b8d50cebd618e53d
Xiangyu Yue
64fa84f4b8d50cebd618e53e
Hongsheng Li
claimed_verified
2024-02-05T10:50:21.127Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
64fa84f4b8d50cebd618e53f
Yu Qiao
ImageBind-LLM: Multi-modality Instruction Tuning
We present ImageBind-LLM, a multi-modality instruction tuning method of large
language models (LLMs) via ImageBind. Existing works mainly focus on language
and image instruction tuning, different from which, our ImageBind-LLM can
respond to multi-modality conditions, including audio, 3D point clouds, video,
and their embedding-space arithmetic by only image-text alignment training.
During training, we adopt a learnable bind network to align the embedding space
between LLaMA and ImageBind's image encoder. Then, the image features
transformed by the bind network are added to word tokens of all layers in
LLaMA, which progressively injects visual instructions via an attention-free
and zero-initialized gating mechanism. Aided by the joint embedding of
ImageBind, the simple image-text training enables our model to exhibit superior
multi-modality instruction-following capabilities. During inference, the
multi-modality inputs are fed into the corresponding ImageBind encoders, and
processed by a proposed visual cache model for further cross-modal embedding
enhancement. The training-free cache model retrieves from three million image
features extracted by ImageBind, which effectively mitigates the
training-inference modality discrepancy. Notably, with our approach,
ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
significant language generation quality. Code is released at
https://github.com/OpenGVLab/LLaMA-Adapter.
ImageBind-LLM uses a learnable bind network to enable large language models to follow multi-modal instructions through image-text alignment and a visual cache model.
multi-modality instruction tuning ImageBind image-text alignment learnable bind network embedding space LLaMA visual instructions attention-free zero-initialized gating mechanism visual cache model cross-modal embedding enhancement training-free cache model multi-modality inputs ImageBind encoders image features language generation quality