TextBind: Multi-turn Interleaved Multimodal Instruction-following
Large language models with instruction-following abilities have
revolutionized the field of artificial intelligence. These models show
exceptional generalizability to tackle various real-world tasks through their
natural language interfaces. However, their performance heavily relies on
high-quality exemplar data, which is often difficult to obtain. This challenge
is further exacerbated when it comes to multimodal instruction following. We
introduce TextBind, an almost annotation-free framework for empowering larger
language models with the multi-turn interleaved multimodal
instruction-following capabilities. Our approach requires only image-caption
pairs and generates multi-turn multimodal instruction-response conversations
from a language model. We release our dataset, model, and demo to foster future
research in the area of multimodal instruction following.
65090190634e02df56a3cbeb
Huayang Li
claimed_verified
2023-09-19T11:29:58.987Z
64672e286192d3914221d633
/avatars/a26908ebcbb74feaf44fe83a44fef8ac.svg
Huayang
user
huayangli
65090190634e02df56a3cbec
Siheng Li
admin_assigned
2023-09-20T14:38:00.566Z
62f71b23d3bdacb7eec43b1f
https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
SihengLi
user
SihengLi
65090190634e02df56a3cbed
Deng Cai
claimed_verified
2023-09-20T09:49:54.132Z
646735a1946476c5d214e5b2
/avatars/23392fda23fa6288fcb309f7eaa73472.svg
Deng Cai
user
jcyk
65090190634e02df56a3cbee
Longyue Wang
admin_assigned
2023-09-20T14:38:07.968Z
636b030c328133bdb3a523bc
/avatars/15d5d5403fef2f1368bb4185b199061d.svg
Longyue Wang
user
longyuewang
65090190634e02df56a3cbef
Lemao Liu
admin_assigned
2023-09-20T14:38:14.924Z
64462d77efa0723f3946356b
/avatars/f9757030d82c69aef933309e0c83ccd0.svg
Lemao Liu
user
lemaoliu
65090190634e02df56a3cbf0
Taro Watanabe
65090190634e02df56a3cbf1
Yujiu Yang
admin_assigned
2023-09-20T14:38:30.623Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
65090190634e02df56a3cbf2
Shuming Shi
admin_assigned
2023-09-20T14:38:37.725Z
62eb8e9eaf9bed70f5ad5a7e
/avatars/971763f3b724d63bc64b7d3599cfc753.svg
Shuming Shi
user
Shuming
TextBind: Multi-turn Interleaved Multimodal Instruction-following
Large language models with instruction-following abilities have
revolutionized the field of artificial intelligence. These models show
exceptional generalizability to tackle various real-world tasks through their
natural language interfaces. However, their performance heavily relies on
high-quality exemplar data, which is often difficult to obtain. This challenge
is further exacerbated when it comes to multimodal instruction following. We
introduce TextBind, an almost annotation-free framework for empowering larger
language models with the multi-turn interleaved multimodal
instruction-following capabilities. Our approach requires only image-caption
pairs and generates multi-turn multimodal instruction-response conversations
from a language model. We release our dataset, model, and demo to foster future
research in the area of multimodal instruction following.
TextBind is an annotation-free framework that enhances large language models for multimodal instruction following by generating conversations from image-caption pairs.
large language models instruction-following abilities generalizability real-world tasks natural language interfaces multimodal instruction following TextBind multi-turn interleaved multimodal instruction-following image-caption pairs conversations dataset demo