title: MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
summary: We present MM1.5, a new family of multimodal large language models (MLLMs)
  designed to enhance capabilities in text-rich image understanding, visual
  referring and grounding, and multi-image reasoning. Building upon the MM1
  architecture, MM1.5 adopts a data-centric approach to model training,
  systematically exploring the impact of diverse data mixtures across the entire
  model training lifecycle. This includes high-quality OCR data and synthetic
  captions for continual pre-training, as well as an optimized visual
  instruction-tuning data mixture for supervised fine-tuning. Our models range
  from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
  variants, and demonstrate that careful data curation and training strategies
  can yield strong performance even at small scales (1B and 3B). Additionally, we
  introduce two specialized variants: MM1.5-Video, designed for video
  understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
  extensive empirical studies and ablations, we provide detailed insights into
  the training processes and decisions that inform our final designs, offering
  valuable guidance for future research in MLLM development.
paper_authors._id: 66fb8eda0c0759cfa0920c46
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:16:34.961Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 66fb8eda0c0759cfa0920c47
paper_authors.name: Mingfei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:34:18.731Z
paper_authors._id: 64388d5f1efe72ba48071c52
paper_authors.avatarUrl: /avatars/2653fba13e71f471e9d55c134fe25efc.svg
paper_authors.fullname: Mingfei Gao
paper_authors.type: user
paper_authors.user: fly6464
paper_authors._id: 66fb8eda0c0759cfa0920c48
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:34:38.598Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 66fb8eda0c0759cfa0920c49
paper_authors.name: Philipp Dufter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:34:44.643Z
paper_authors._id: 6079980b88160e14e4e2e49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
paper_authors.fullname: Philipp Dufter
paper_authors.type: user
paper_authors.user: pdufter
paper_authors._id: 66fb8eda0c0759cfa0920c4a
paper_authors.name: Nina Wenzel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:34:50.560Z
paper_authors._id: 65f9a3ebf8c8ab9966fccffc
paper_authors.avatarUrl: /avatars/07e466225e1142d2fd7ebc2f23fe5062.svg
paper_authors.fullname: Nina Wenzel
paper_authors.type: user
paper_authors.user: nm-w
paper_authors._id: 66fb8eda0c0759cfa0920c4b
paper_authors.name: Forrest Huang
paper_authors._id: 66fb8eda0c0759cfa0920c4c
paper_authors.name: Dhruti Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:35:11.193Z
paper_authors._id: 65a686f7f319aa03481ed105
paper_authors.avatarUrl: /avatars/0e344f4ff0714932340b6225eadfcd6d.svg
paper_authors.fullname: Dhruti Shah
paper_authors.type: user
paper_authors.user: dhruti96shah
paper_authors._id: 66fb8eda0c0759cfa0920c4d
paper_authors.name: Xianzhi Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:35:17.781Z
paper_authors._id: 65cc30b80390fce6291d03cf
paper_authors.avatarUrl: /avatars/7e774270d1cca48f43f0b379af87003e.svg
paper_authors.fullname: Xianzhi Du
paper_authors.type: user
paper_authors.user: xianzhi-du
paper_authors._id: 66fb8eda0c0759cfa0920c4e
paper_authors.name: Bowen Zhang
paper_authors._id: 66fb8eda0c0759cfa0920c4f
paper_authors.name: Yanghao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:35:45.486Z
paper_authors._id: 65dad3870af7e21ba473439f
paper_authors.avatarUrl: /avatars/da542e7d68ae937bbdb791f17096bb1c.svg
paper_authors.fullname: Yanghao Li
paper_authors.type: user
paper_authors.user: FrozzZen
paper_authors._id: 66fb8eda0c0759cfa0920c50
paper_authors.name: Sam Dodge
paper_authors._id: 66fb8eda0c0759cfa0920c51
paper_authors.name: Keen You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:35:59.099Z
paper_authors._id: 6150a95100a52e55be2994eb
paper_authors.avatarUrl: /avatars/c8e2f85a00ca54e1984a692f29f3096f.svg
paper_authors.fullname: Keen You
paper_authors.type: user
paper_authors.user: keenyou
paper_authors._id: 66fb8eda0c0759cfa0920c52
paper_authors.name: Zhen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:24.739Z
paper_authors._id: 66047f6f0e127d15dcfc262f
paper_authors.avatarUrl: /avatars/a2355f30a358fed4ae6bf600ec4be9c7.svg
paper_authors.fullname: Zhen Yang
paper_authors.type: user
paper_authors.user: zhen-yang
paper_authors._id: 66fb8eda0c0759cfa0920c53
paper_authors.name: Aleksei Timofeev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:36:26.873Z
paper_authors._id: 6473ae08485a7c8e1bd34e0d
paper_authors.avatarUrl: /avatars/575faf4b3e1dc7b657ed18fac9b9501a.svg
paper_authors.fullname: Aleksei Timofeev
paper_authors.type: user
paper_authors.user: Ymaril
paper_authors._id: 66fb8eda0c0759cfa0920c54
paper_authors.name: Mingze Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:36:51.729Z
paper_authors._id: 65fc655930aee594d74dd7b8
paper_authors.avatarUrl: /avatars/e8df7170fa1cf99bb3a2b9a083ee99a1.svg
paper_authors.fullname: Mingze Xu
paper_authors.type: user
paper_authors.user: xumingze0308
paper_authors._id: 66fb8eda0c0759cfa0920c55
paper_authors.name: Hong-You Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:36:57.955Z
paper_authors._id: 64b720891a4d97b5d7f055b8
paper_authors.avatarUrl: /avatars/7247a4f3a433a58fe934f245d9a757b2.svg
paper_authors.fullname: Hong-You Chen
paper_authors.type: user
paper_authors.user: Cbbjames
paper_authors._id: 66fb8eda0c0759cfa0920c56
paper_authors.name: Jean-Philippe Fauconnier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:37:04.189Z
paper_authors._id: 62cd5c43299c0c2e0e437842
paper_authors.avatarUrl: /avatars/8fa8511baf2d9bd95d3ba4535a5b3d69.svg
paper_authors.fullname: Jean-Philippe Fauconnier
paper_authors.type: user
paper_authors.user: j4kn
paper_authors._id: 66fb8eda0c0759cfa0920c57
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:34:23.314Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 66fb8eda0c0759cfa0920c58
paper_authors.name: Haoxuan You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:37:20.593Z
paper_authors._id: 65284ba3e3b3844f5e487a0b
paper_authors.avatarUrl: /avatars/fce12a233bddb611d441f25a2ab17725.svg
paper_authors.fullname: Haoxuan You
paper_authors.type: user
paper_authors.user: HaoxuanYou
paper_authors._id: 66fb8eda0c0759cfa0920c59
paper_authors.name: Zirui Wang
paper_authors._id: 66fb8eda0c0759cfa0920c5a
paper_authors.name: Afshin Dehghan
paper_authors._id: 66fb8eda0c0759cfa0920c5b
paper_authors.name: Peter Grasch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:38:00.608Z
paper_authors._id: 6649e0b6f50d471119285ab8
paper_authors.avatarUrl: /avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
paper_authors.fullname: Peter Grasch
paper_authors.type: user
paper_authors.user: petergrasch
paper_authors._id: 66fb8eda0c0759cfa0920c5c
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T07:38:07.212Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_title: MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
paper_summary: We present MM1.5, a new family of multimodal large language models (MLLMs)
  designed to enhance capabilities in text-rich image understanding, visual
  referring and grounding, and multi-image reasoning. Building upon the MM1
  architecture, MM1.5 adopts a data-centric approach to model training,
  systematically exploring the impact of diverse data mixtures across the entire
  model training lifecycle. This includes high-quality OCR data and synthetic
  captions for continual pre-training, as well as an optimized visual
  instruction-tuning data mixture for supervised fine-tuning. Our models range
  from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
  variants, and demonstrate that careful data curation and training strategies
  can yield strong performance even at small scales (1B and 3B). Additionally, we
  introduce two specialized variants: MM1.5-Video, designed for video
  understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
  extensive empirical studies and ablations, we provide detailed insights into
  the training processes and decisions that inform our final designs, offering
  valuable guidance for future research in MLLM development.
paper_ai_summary: MM1.5 enhances multimodal language models with a data-centric approach, supporting diverse tasks like text-rich image understanding and video comprehension through careful data curation and training strategies.
paper_ai_keywords: multimodal large language models MLLMs data-centric approach high-quality OCR data synthetic captions visual instruction-tuning dense mixture-of-experts MoE MM1.5-Video MM1.5-UI video understanding mobile UI understanding empirical studies ablations