title: Otter: A Multi-Modal Model with In-Context Instruction Tuning
summary: Large language models (LLMs) have demonstrated significant universal
  capabilities as few/zero-shot learners in various tasks due to their
  pre-training on vast amounts of text data, as exemplified by GPT-3, which
  boosted to InstrctGPT and ChatGPT, effectively following natural language
  instructions to accomplish real-world tasks. In this paper, we propose to
  introduce instruction tuning into multi-modal models, motivated by the Flamingo
  model's upstream interleaved format pretraining dataset. We adopt a similar
  approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT)
  dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo
  (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and
  showcasing improved instruction-following ability and in-context learning. We
  also optimize OpenFlamingo's implementation for researchers, democratizing the
  required training resources from 1times A100 GPU to 4times RTX-3090 GPUs,
  and integrate both OpenFlamingo and Otter into Huggingface Transformers for
  more researchers to incorporate the models into their customized training and
  inference pipelines.
paper_authors._id: 64584aa5bb7bdc9655aefbb6
paper_authors.name: Bo Li
paper_authors._id: 64584aa5bb7bdc9655aefbb7
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:53:29.635Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 64584aa5bb7bdc9655aefbb8
paper_authors.name: Liangyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:52:44.728Z
paper_authors._id: 62b67da0f56de4396ca9e44b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658586059273-62b67da0f56de4396ca9e44b.jpeg
paper_authors.fullname: Liangyu Chen
paper_authors.type: user
paper_authors.user: liangyuch
paper_authors._id: 64584aa5bb7bdc9655aefbb9
paper_authors.name: Jinghao Wang
paper_authors._id: 64584aa5bb7bdc9655aefbba
paper_authors.name: Jingkang Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:54:09.479Z
paper_authors._id: 62b5777f593a2c49da69dc02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
paper_authors.fullname: Jingkang Yang
paper_authors.type: user
paper_authors.user: Jingkang
paper_authors._id: 64584aa5bb7bdc9655aefbbb
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:54:47.227Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Otter: A Multi-Modal Model with In-Context Instruction Tuning
paper_summary: Large language models (LLMs) have demonstrated significant universal
  capabilities as few/zero-shot learners in various tasks due to their
  pre-training on vast amounts of text data, as exemplified by GPT-3, which
  boosted to InstrctGPT and ChatGPT, effectively following natural language
  instructions to accomplish real-world tasks. In this paper, we propose to
  introduce instruction tuning into multi-modal models, motivated by the Flamingo
  model's upstream interleaved format pretraining dataset. We adopt a similar
  approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT)
  dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo
  (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and
  showcasing improved instruction-following ability and in-context learning. We
  also optimize OpenFlamingo's implementation for researchers, democratizing the
  required training resources from 1times A100 GPU to 4times RTX-3090 GPUs,
  and integrate both OpenFlamingo and Otter into Huggingface Transformers for
  more researchers to incorporate the models into their customized training and
  inference pipelines.
paper_ai_keywords: Large language models (LLMs) few/zero-shot learners pre-training InstrctGPT ChatGPT natural language instructions multi-modal models Flamingo model upstream interleaved format pretraining dataset MultI-Modal In-Context Instruction Tuning (MIMIC-IT) Otter OpenFlamingo instruction-following ability in-context learning OpenFlamingo's implementation Huggingface Transformers