title: MIMIC-IT: Multi-Modal In-Context Instruction Tuning
summary: High-quality instructions and responses are essential for the zero-shot
  performance of large language models on interactive natural language tasks. For
  interactive vision-language tasks involving intricate visual scenes, a large
  quantity of diverse and creative instruction-response pairs should be
  imperative to tune vision-language models (VLMs). Nevertheless, the current
  availability of vision-language instruction-response pairs in terms of
  quantity, diversity, and creativity remains limited, posing challenges to the
  generalization of interactive VLMs. Here we present MultI-Modal In-Context
  Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal
  instruction-response pairs, with 2.2 million unique instructions derived from
  images and videos. Each pair is accompanied by multi-modal in-context
  information, forming conversational contexts aimed at empowering VLMs in
  perception, reasoning, and planning. The instruction-response collection
  process, dubbed as Syphus, is scaled using an automatic annotation pipeline
  that combines human expertise with GPT's capabilities. Using the MIMIC-IT
  dataset, we train a large VLM named Otter. Based on extensive evaluations
  conducted on vision-language benchmarks, it has been observed that Otter
  demonstrates remarkable proficiency in multi-modal perception, reasoning, and
  in-context learning. Human evaluation reveals it effectively aligns with the
  user's intentions. We release the MIMIC-IT dataset, instruction-response
  collection pipeline, benchmarks, and the Otter model.
paper_authors._id: 6482a1fa93362a0d1216f274
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:49:28.279Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 6482a1fa93362a0d1216f275
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:32:49.378Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 6482a1fa93362a0d1216f276
paper_authors.name: Liangyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T09:12:26.078Z
paper_authors._id: 62b67da0f56de4396ca9e44b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658586059273-62b67da0f56de4396ca9e44b.jpeg
paper_authors.fullname: Liangyu Chen
paper_authors.type: user
paper_authors.user: liangyuch
paper_authors._id: 6482a1fa93362a0d1216f277
paper_authors.name: Jinghao Wang
paper_authors._id: 6482a1fa93362a0d1216f278
paper_authors.name: Fanyi Pu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:33:45.435Z
paper_authors._id: 646e1ef5075bbcc48ddf21e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_vJC0zeVOIvaNV2R6toqg.jpeg
paper_authors.fullname: Pu Fanyi
paper_authors.type: user
paper_authors.user: pufanyi
paper_authors._id: 6482a1fa93362a0d1216f279
paper_authors.name: Jingkang Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:34:01.256Z
paper_authors._id: 62b5777f593a2c49da69dc02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
paper_authors.fullname: Jingkang Yang
paper_authors.type: user
paper_authors.user: Jingkang
paper_authors._id: 6482a1fa93362a0d1216f27a
paper_authors.name: Chunyuan Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-11T05:13:22.469Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 6482a1fa93362a0d1216f27b
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:34:25.660Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: MIMIC-IT: Multi-Modal In-Context Instruction Tuning
paper_summary: High-quality instructions and responses are essential for the zero-shot
  performance of large language models on interactive natural language tasks. For
  interactive vision-language tasks involving intricate visual scenes, a large
  quantity of diverse and creative instruction-response pairs should be
  imperative to tune vision-language models (VLMs). Nevertheless, the current
  availability of vision-language instruction-response pairs in terms of
  quantity, diversity, and creativity remains limited, posing challenges to the
  generalization of interactive VLMs. Here we present MultI-Modal In-Context
  Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal
  instruction-response pairs, with 2.2 million unique instructions derived from
  images and videos. Each pair is accompanied by multi-modal in-context
  information, forming conversational contexts aimed at empowering VLMs in
  perception, reasoning, and planning. The instruction-response collection
  process, dubbed as Syphus, is scaled using an automatic annotation pipeline
  that combines human expertise with GPT's capabilities. Using the MIMIC-IT
  dataset, we train a large VLM named Otter. Based on extensive evaluations
  conducted on vision-language benchmarks, it has been observed that Otter
  demonstrates remarkable proficiency in multi-modal perception, reasoning, and
  in-context learning. Human evaluation reveals it effectively aligns with the
  user's intentions. We release the MIMIC-IT dataset, instruction-response
  collection pipeline, benchmarks, and the Otter model.
paper_ai_keywords: vision-language models (VLMs) instruction-response pairs multimodal instruction-response pairs Discrimin MultI-Modal In-Context Instruction Tuning (MIMIC-IT) Conversational contexts Syphus automatic annotation pipeline perception reasoning planning in-context learning Otter vision-language benchmarks multi-modal perception multi-modal reasoning multi-modal learning human evaluation intended outcomes