xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
This report introduces xGen-MM (also known as BLIP-3), a framework for
developing Large Multimodal Models (LMMs). The framework comprises meticulously
curated datasets, a training recipe, model architectures, and a resulting suite
of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
initiative on foundation AI models. Our models undergo rigorous evaluation
across a range of tasks, including both single and multi-image benchmarks. Our
pre-trained base model exhibits strong in-context learning capabilities and the
instruction-tuned model demonstrates competitive performance among open-source
LMMs with similar model sizes. In addition, we introduce a safety-tuned model
with DPO, aiming to mitigate harmful behaviors such as hallucinations and
improve safety. We open-source our models, curated large-scale datasets, and
our fine-tuning codebase to facilitate further advancements in LMM research.
Associated resources will be available on our project page above.
66c2a7d20836dd7a55716f8e
Le Xue
claimed_verified
2024-08-22T19:52:05.876Z
63dd73e7422ca8d7f7e3698c
/avatars/7b0f8419f6941230b81dbbbb4f273edf.svg
Le Xue
user
SFXX
66c2a7d20836dd7a55716f8f
Manli Shu
claimed_verified
2024-08-19T22:13:15.619Z
63bdcb129a15a3e94198e7d3
https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
Manli Shu
user
Manli
66c2a7d20836dd7a55716f90
Anas Awadalla
claimed_verified
2024-08-19T22:13:07.818Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
66c2a7d20836dd7a55716f91
Jun Wang
66c2a7d20836dd7a55716f92
An Yan
claimed_verified
2024-08-19T22:13:13.154Z
634dfb9bbe5a827d48749f39
/avatars/a8df0eeb1df39f3935dd686a94768f30.svg
An Yan
user
zzxslp
66c2a7d20836dd7a55716f93
Senthil Purushwalkam
66c2a7d20836dd7a55716f94
Honglu Zhou
66c2a7d20836dd7a55716f95
Viraj Prabhu
66c2a7d20836dd7a55716f96
Yutong Dai
66c2a7d20836dd7a55716f97
Michael S Ryoo
66c2a7d20836dd7a55716f98
Shrikant Kendre
66c2a7d20836dd7a55716f99
Jieyu Zhang
claimed_verified
2024-08-22T19:52:03.251Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
66c2a7d20836dd7a55716f9a
Can Qin
66c2a7d20836dd7a55716f9b
Shu Zhang
66c2a7d20836dd7a55716f9c
Chia-Chih Chen
66c2a7d20836dd7a55716f9d
Ning Yu
claimed_verified
2025-01-23T09:22:11.542Z
6362bcbe8f43a912fc722969
https://cdn-avatars.huggingface.co/v1/production/uploads/6362bcbe8f43a912fc722969/ktl2ePfpOIseqlIbuldNa.png
Ning Yu
user
ningyu1991
66c2a7d20836dd7a55716f9e
Juntao Tan
66c2a7d20836dd7a55716f9f
Tulika Manoj Awalgaonkar
66c2a7d20836dd7a55716fa0
Shelby Heinecke
66c2a7d20836dd7a55716fa1
Huan Wang
66c2a7d20836dd7a55716fa2
Yejin Choi
66c2a7d20836dd7a55716fa3
Ludwig Schmidt
66c2a7d20836dd7a55716fa4
Zeyuan Chen
66c2a7d20836dd7a55716fa5
Silvio Savarese
66c2a7d20836dd7a55716fa6
Juan Carlos Niebles
claimed_verified
2024-08-19T22:13:05.476Z
65bb26b5d61b51a50805aba2
/avatars/966c8469f55776948a7cf42d98accff3.svg
Juan Carlos Niebles
user
niebles
66c2a7d20836dd7a55716fa7
Caiming Xiong
66c2a7d20836dd7a55716fa8
Ran Xu
claimed_verified
2024-08-19T22:13:10.424Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
This report introduces xGen-MM (also known as BLIP-3), a framework for
developing Large Multimodal Models (LMMs). The framework comprises meticulously
curated datasets, a training recipe, model architectures, and a resulting suite
of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
initiative on foundation AI models. Our models undergo rigorous evaluation
across a range of tasks, including both single and multi-image benchmarks. Our
pre-trained base model exhibits strong in-context learning capabilities and the
instruction-tuned model demonstrates competitive performance among open-source
LMMs with similar model sizes. In addition, we introduce a safety-tuned model
with DPO, aiming to mitigate harmful behaviors such as hallucinations and
improve safety. We open-source our models, curated large-scale datasets, and
our fine-tuning codebase to facilitate further advancements in LMM research.
Associated resources will be available on our project page above.
xGen-MM, an extension of Salesforce's xGen initiative, provides a framework for developing Large Multimodal Models with pre-trained, instruction-tuned, and safety-tuned variants.
Large Multimodal Models LMMs in-context learning instruction-tuned safety-tuned DPO hallucinations