title: TextBind: Multi-turn Interleaved Multimodal Instruction-following
summary: Large language models with instruction-following abilities have
  revolutionized the field of artificial intelligence. These models show
  exceptional generalizability to tackle various real-world tasks through their
  natural language interfaces. However, their performance heavily relies on
  high-quality exemplar data, which is often difficult to obtain. This challenge
  is further exacerbated when it comes to multimodal instruction following. We
  introduce TextBind, an almost annotation-free framework for empowering larger
  language models with the multi-turn interleaved multimodal
  instruction-following capabilities. Our approach requires only image-caption
  pairs and generates multi-turn multimodal instruction-response conversations
  from a language model. We release our dataset, model, and demo to foster future
  research in the area of multimodal instruction following.
paper_authors._id: 65090190634e02df56a3cbeb
paper_authors.name: Huayang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-19T11:29:58.987Z
paper_authors._id: 64672e286192d3914221d633
paper_authors.avatarUrl: /avatars/a26908ebcbb74feaf44fe83a44fef8ac.svg
paper_authors.fullname: Huayang
paper_authors.type: user
paper_authors.user: huayangli
paper_authors._id: 65090190634e02df56a3cbec
paper_authors.name: Siheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:38:00.566Z
paper_authors._id: 62f71b23d3bdacb7eec43b1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f71b23d3bdacb7eec43b1f/6RNVG9zug9tutjo5i8T4C.jpeg
paper_authors.fullname: SihengLi
paper_authors.type: user
paper_authors.user: SihengLi
paper_authors._id: 65090190634e02df56a3cbed
paper_authors.name: Deng Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-20T09:49:54.132Z
paper_authors._id: 646735a1946476c5d214e5b2
paper_authors.avatarUrl: /avatars/23392fda23fa6288fcb309f7eaa73472.svg
paper_authors.fullname: Deng Cai
paper_authors.type: user
paper_authors.user: jcyk
paper_authors._id: 65090190634e02df56a3cbee
paper_authors.name: Longyue Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:38:07.968Z
paper_authors._id: 636b030c328133bdb3a523bc
paper_authors.avatarUrl: /avatars/15d5d5403fef2f1368bb4185b199061d.svg
paper_authors.fullname: Longyue Wang
paper_authors.type: user
paper_authors.user: longyuewang
paper_authors._id: 65090190634e02df56a3cbef
paper_authors.name: Lemao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:38:14.924Z
paper_authors._id: 64462d77efa0723f3946356b
paper_authors.avatarUrl: /avatars/f9757030d82c69aef933309e0c83ccd0.svg
paper_authors.fullname: Lemao Liu
paper_authors.type: user
paper_authors.user: lemaoliu
paper_authors._id: 65090190634e02df56a3cbf0
paper_authors.name: Taro Watanabe
paper_authors._id: 65090190634e02df56a3cbf1
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:38:30.623Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_authors._id: 65090190634e02df56a3cbf2
paper_authors.name: Shuming Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:38:37.725Z
paper_authors._id: 62eb8e9eaf9bed70f5ad5a7e
paper_authors.avatarUrl: /avatars/971763f3b724d63bc64b7d3599cfc753.svg
paper_authors.fullname: Shuming Shi
paper_authors.type: user
paper_authors.user: Shuming
paper_title: TextBind: Multi-turn Interleaved Multimodal Instruction-following
paper_summary: Large language models with instruction-following abilities have
  revolutionized the field of artificial intelligence. These models show
  exceptional generalizability to tackle various real-world tasks through their
  natural language interfaces. However, their performance heavily relies on
  high-quality exemplar data, which is often difficult to obtain. This challenge
  is further exacerbated when it comes to multimodal instruction following. We
  introduce TextBind, an almost annotation-free framework for empowering larger
  language models with the multi-turn interleaved multimodal
  instruction-following capabilities. Our approach requires only image-caption
  pairs and generates multi-turn multimodal instruction-response conversations
  from a language model. We release our dataset, model, and demo to foster future
  research in the area of multimodal instruction following.
paper_ai_summary: TextBind is an annotation-free framework that enhances large language models for multimodal instruction following by generating conversations from image-caption pairs.
paper_ai_keywords: large language models instruction-following abilities generalizability real-world tasks natural language interfaces multimodal instruction following TextBind multi-turn interleaved multimodal instruction-following image-caption pairs conversations dataset demo