title: xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
summary: This report introduces xGen-MM (also known as BLIP-3), a framework for
  developing Large Multimodal Models (LMMs). The framework comprises meticulously
  curated datasets, a training recipe, model architectures, and a resulting suite
  of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
  initiative on foundation AI models. Our models undergo rigorous evaluation
  across a range of tasks, including both single and multi-image benchmarks. Our
  pre-trained base model exhibits strong in-context learning capabilities and the
  instruction-tuned model demonstrates competitive performance among open-source
  LMMs with similar model sizes. In addition, we introduce a safety-tuned model
  with DPO, aiming to mitigate harmful behaviors such as hallucinations and
  improve safety. We open-source our models, curated large-scale datasets, and
  our fine-tuning codebase to facilitate further advancements in LMM research.
  Associated resources will be available on our project page above.
paper_authors._id: 66c2a7d20836dd7a55716f8e
paper_authors.name: Le Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:52:05.876Z
paper_authors._id: 63dd73e7422ca8d7f7e3698c
paper_authors.avatarUrl: /avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
paper_authors.fullname: Le Xue
paper_authors.type: user
paper_authors.user: SFXX
paper_authors._id: 66c2a7d20836dd7a55716f8f
paper_authors.name: Manli Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:15.619Z
paper_authors._id: 63bdcb129a15a3e94198e7d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
paper_authors.fullname: Manli Shu
paper_authors.type: user
paper_authors.user: Manli
paper_authors._id: 66c2a7d20836dd7a55716f90
paper_authors.name: Anas Awadalla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:07.818Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 66c2a7d20836dd7a55716f91
paper_authors.name: Jun Wang
paper_authors._id: 66c2a7d20836dd7a55716f92
paper_authors.name: An Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:13.154Z
paper_authors._id: 634dfb9bbe5a827d48749f39
paper_authors.avatarUrl: /avatars/a8df0eeb1df39f3935dd686a94768f30.svg
paper_authors.fullname: An Yan
paper_authors.type: user
paper_authors.user: zzxslp
paper_authors._id: 66c2a7d20836dd7a55716f93
paper_authors.name: Senthil Purushwalkam
paper_authors._id: 66c2a7d20836dd7a55716f94
paper_authors.name: Honglu Zhou
paper_authors._id: 66c2a7d20836dd7a55716f95
paper_authors.name: Viraj Prabhu
paper_authors._id: 66c2a7d20836dd7a55716f96
paper_authors.name: Yutong Dai
paper_authors._id: 66c2a7d20836dd7a55716f97
paper_authors.name: Michael S Ryoo
paper_authors._id: 66c2a7d20836dd7a55716f98
paper_authors.name: Shrikant Kendre
paper_authors._id: 66c2a7d20836dd7a55716f99
paper_authors.name: Jieyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:52:03.251Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 66c2a7d20836dd7a55716f9a
paper_authors.name: Can Qin
paper_authors._id: 66c2a7d20836dd7a55716f9b
paper_authors.name: Shu Zhang
paper_authors._id: 66c2a7d20836dd7a55716f9c
paper_authors.name: Chia-Chih Chen
paper_authors._id: 66c2a7d20836dd7a55716f9d
paper_authors.name: Ning Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:22:11.542Z
paper_authors._id: 6362bcbe8f43a912fc722969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
paper_authors.fullname: Ning Yu
paper_authors.type: user
paper_authors.user: ningyu1991
paper_authors._id: 66c2a7d20836dd7a55716f9e
paper_authors.name: Juntao Tan
paper_authors._id: 66c2a7d20836dd7a55716f9f
paper_authors.name: Tulika Manoj Awalgaonkar
paper_authors._id: 66c2a7d20836dd7a55716fa0
paper_authors.name: Shelby Heinecke
paper_authors._id: 66c2a7d20836dd7a55716fa1
paper_authors.name: Huan Wang
paper_authors._id: 66c2a7d20836dd7a55716fa2
paper_authors.name: Yejin Choi
paper_authors._id: 66c2a7d20836dd7a55716fa3
paper_authors.name: Ludwig Schmidt
paper_authors._id: 66c2a7d20836dd7a55716fa4
paper_authors.name: Zeyuan Chen
paper_authors._id: 66c2a7d20836dd7a55716fa5
paper_authors.name: Silvio Savarese
paper_authors._id: 66c2a7d20836dd7a55716fa6
paper_authors.name: Juan Carlos Niebles
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:05.476Z
paper_authors._id: 65bb26b5d61b51a50805aba2
paper_authors.avatarUrl: /avatars/966c8469f55776948a7cf42d98accff3.svg
paper_authors.fullname: Juan Carlos Niebles
paper_authors.type: user
paper_authors.user: niebles
paper_authors._id: 66c2a7d20836dd7a55716fa7
paper_authors.name: Caiming Xiong
paper_authors._id: 66c2a7d20836dd7a55716fa8
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:10.424Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_title: xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
paper_summary: This report introduces xGen-MM (also known as BLIP-3), a framework for
  developing Large Multimodal Models (LMMs). The framework comprises meticulously
  curated datasets, a training recipe, model architectures, and a resulting suite
  of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
  initiative on foundation AI models. Our models undergo rigorous evaluation
  across a range of tasks, including both single and multi-image benchmarks. Our
  pre-trained base model exhibits strong in-context learning capabilities and the
  instruction-tuned model demonstrates competitive performance among open-source
  LMMs with similar model sizes. In addition, we introduce a safety-tuned model
  with DPO, aiming to mitigate harmful behaviors such as hallucinations and
  improve safety. We open-source our models, curated large-scale datasets, and
  our fine-tuning codebase to facilitate further advancements in LMM research.
  Associated resources will be available on our project page above.