title: MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
summary: In this work, we discuss building performant Multimodal Large Language Models
  (MLLMs). In particular, we study the importance of various architecture
  components and data choices. Through careful and comprehensive ablations of the
  image encoder, the vision language connector, and various pre-training data
  choices, we identified several crucial design lessons. For example, we
  demonstrate that for large-scale multimodal pre-training using a careful mix of
  image-caption, interleaved image-text, and text-only data is crucial for
  achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks,
  compared to other published pre-training results. Further, we show that the
  image encoder together with image resolution and the image token count has
  substantial impact, while the vision-language connector design is of
  comparatively negligible importance. By scaling up the presented recipe, we
  build MM1, a family of multimodal models up to 30B parameters, consisting of
  both dense models and mixture-of-experts (MoE) variants, that are SOTA in
  pre-training metrics and achieve competitive performance after supervised
  fine-tuning on a range of established multimodal benchmarks. Thanks to
  large-scale pre-training, MM1 enjoys appealing properties such as enhanced
  in-context learning, and multi-image reasoning, enabling few-shot
  chain-of-thought prompting.
paper_authors._id: 65f3b1fb7c44e9e26df10330
paper_authors.name: Brandon McKinzie
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-15T14:48:40.375Z
paper_authors._id: 65a16b5bfb3898fe81c63f51
paper_authors.avatarUrl: /avatars/828e8f3a8efe8c5eb07eb2c0796809d2.svg
paper_authors.fullname: Brandon McKinzie
paper_authors.type: user
paper_authors.user: bmckinz
paper_authors._id: 65f3b1fb7c44e9e26df10331
paper_authors.name: Zhe Gan
paper_authors._id: 65f3b1fb7c44e9e26df10332
paper_authors.name: Jean-Philippe Fauconnier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T21:09:15.876Z
paper_authors._id: 62cd5c43299c0c2e0e437842
paper_authors.avatarUrl: /avatars/8fa8511baf2d9bd95d3ba4535a5b3d69.svg
paper_authors.fullname: Jean-Philippe Fauconnier
paper_authors.type: user
paper_authors.user: j4kn
paper_authors._id: 65f3b1fb7c44e9e26df10333
paper_authors.name: Sam Dodge
paper_authors._id: 65f3b1fb7c44e9e26df10334
paper_authors.name: Bowen Zhang
paper_authors._id: 65f3b1fb7c44e9e26df10335
paper_authors.name: Philipp Dufter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:13:01.977Z
paper_authors._id: 6079980b88160e14e4e2e49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
paper_authors.fullname: Philipp Dufter
paper_authors.type: user
paper_authors.user: pdufter
paper_authors._id: 65f3b1fb7c44e9e26df10336
paper_authors.name: Dhruti Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:13:14.745Z
paper_authors._id: 65a686f7f319aa03481ed105
paper_authors.avatarUrl: /avatars/0e344f4ff0714932340b6225eadfcd6d.svg
paper_authors.fullname: Dhruti Shah
paper_authors.type: user
paper_authors.user: dhruti96shah
paper_authors._id: 65f3b1fb7c44e9e26df10337
paper_authors.name: Xianzhi Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:13:23.713Z
paper_authors._id: 65cc30b80390fce6291d03cf
paper_authors.avatarUrl: /avatars/7e774270d1cca48f43f0b379af87003e.svg
paper_authors.fullname: Xianzhi Du
paper_authors.type: user
paper_authors.user: xianzhi-du
paper_authors._id: 65f3b1fb7c44e9e26df10338
paper_authors.name: Futang Peng
paper_authors._id: 65f3b1fb7c44e9e26df10339
paper_authors.name: Floris Weers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T11:02:56.703Z
paper_authors._id: 65f41f1d6724ea19d7bc9052
paper_authors.avatarUrl: /avatars/4ccd6279aac3a18e1b23642a1a549e2f.svg
paper_authors.fullname: Floris Weers
paper_authors.type: user
paper_authors.user: fweers
paper_authors._id: 65f3b1fb7c44e9e26df1033a
paper_authors.name: Anton Belyi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:13:45.933Z
paper_authors._id: 656b630a9c8778992f7966a4
paper_authors.avatarUrl: /avatars/6438033af2f1abc3b6761cd5cac92cb8.svg
paper_authors.fullname: Anton Belyi
paper_authors.type: user
paper_authors.user: antonis-white
paper_authors._id: 65f3b1fb7c44e9e26df1033b
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T08:03:10.698Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 65f3b1fb7c44e9e26df1033c
paper_authors.name: Karanjeet Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:13:55.064Z
paper_authors._id: 649b97d86104e208ef64bb6f
paper_authors.avatarUrl: /avatars/f93682073b4a2dd67ca223439294ed56.svg
paper_authors.fullname: karanjeet singh
paper_authors.type: user
paper_authors.user: sorryihaveagf
paper_authors._id: 65f3b1fb7c44e9e26df1033d
paper_authors.name: Doug Kang
paper_authors._id: 65f3b1fb7c44e9e26df1033e
paper_authors.name: Hongyu Hè
paper_authors._id: 65f3b1fb7c44e9e26df1033f
paper_authors.name: Max Schwarzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:14:21.670Z
paper_authors._id: 6306f2de435ec751b730087e
paper_authors.avatarUrl: /avatars/7e2858cbc1526f6060a730f8ca3af290.svg
paper_authors.fullname: Max Schwarzer
paper_authors.type: user
paper_authors.user: MaxSchwarzer
paper_authors._id: 65f3b1fb7c44e9e26df10340
paper_authors.name: Tom Gunter
paper_authors._id: 65f3b1fb7c44e9e26df10341
paper_authors.name: Xiang Kong
paper_authors._id: 65f3b1fb7c44e9e26df10342
paper_authors.name: Aonan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-16T10:37:28.641Z
paper_authors._id: 65f3e68a138c6ab771434e2d
paper_authors.avatarUrl: /avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
paper_authors.fullname: Aonan Zhang
paper_authors.type: user
paper_authors.user: AonanZhang
paper_authors._id: 65f3b1fb7c44e9e26df10343
paper_authors.name: Jianyu Wang
paper_authors._id: 65f3b1fb7c44e9e26df10344
paper_authors.name: Chong Wang
paper_authors._id: 65f3b1fb7c44e9e26df10345
paper_authors.name: Nan Du
paper_authors._id: 65f3b1fb7c44e9e26df10346
paper_authors.name: Tao Lei
paper_authors._id: 65f3b1fb7c44e9e26df10347
paper_authors.name: Sam Wiseman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:16:00.960Z
paper_authors._id: 64a435475db6c484bc9a9bd4
paper_authors.avatarUrl: /avatars/f5ecc5affd2d5064fd21b50e0ea0fa0e.svg
paper_authors.fullname: Sam Wiseman
paper_authors.type: user
paper_authors.user: swiseman
paper_authors._id: 65f3b1fb7c44e9e26df10357
paper_authors.name: Guoli Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T09:51:05.703Z
paper_authors._id: 605b7f42935268bc086131ba
paper_authors.avatarUrl: /avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
paper_authors.fullname: Guoli Yin
paper_authors.type: user
paper_authors.user: gyin94
paper_authors._id: 65f3b1fb7c44e9e26df10348
paper_authors.name: Mark Lee
paper_authors._id: 65f3b1fb7c44e9e26df10349
paper_authors.name: Zirui Wang
paper_authors._id: 65f3b1fb7c44e9e26df1034a
paper_authors.name: Ruoming Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:16:56.862Z
paper_authors._id: 654ef8ad3fe6c0b1f871942f
paper_authors.avatarUrl: /avatars/8d3689b9bf57c7c8060ac510a1839158.svg
paper_authors.fullname: Ruoming Pang
paper_authors.type: user
paper_authors.user: ruoming
paper_authors._id: 65f3b1fb7c44e9e26df1034b
paper_authors.name: Peter Grasch
paper_authors._id: 65f3b1fb7c44e9e26df1034c
paper_authors.name: Alexander Toshev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:16:44.937Z
paper_authors._id: 6541636c4f939214d3a0c742
paper_authors.avatarUrl: /avatars/c9507d6ec2b81681461837c8261703c7.svg
paper_authors.fullname: Alexander Toshev
paper_authors.type: user
paper_authors.user: toshev
paper_authors._id: 65f3b1fb7c44e9e26df1034d
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:16:36.271Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_title: MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
paper_summary: In this work, we discuss building performant Multimodal Large Language Models
  (MLLMs). In particular, we study the importance of various architecture
  components and data choices. Through careful and comprehensive ablations of the
  image encoder, the vision language connector, and various pre-training data
  choices, we identified several crucial design lessons. For example, we
  demonstrate that for large-scale multimodal pre-training using a careful mix of
  image-caption, interleaved image-text, and text-only data is crucial for
  achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks,
  compared to other published pre-training results. Further, we show that the
  image encoder together with image resolution and the image token count has
  substantial impact, while the vision-language connector design is of
  comparatively negligible importance. By scaling up the presented recipe, we
  build MM1, a family of multimodal models up to 30B parameters, consisting of
  both dense models and mixture-of-experts (MoE) variants, that are SOTA in
  pre-training metrics and achieve competitive performance after supervised
  fine-tuning on a range of established multimodal benchmarks. Thanks to
  large-scale pre-training, MM1 enjoys appealing properties such as enhanced
  in-context learning, and multi-image reasoning, enabling few-shot
  chain-of-thought prompting.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) image encoder vision language connector pre-training data choices image-caption interleaved image-text text-only data state-of-the-art (SOTA) few-shot results image resolution image token count mixture-of-experts (MoE) pre-training metrics supervised fine-tuning competitive performance established multimodal benchmarks in-context learning multi-image reasoning few-shot chain-of-thought prompting