xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
<|reference_start|>xGen-MM (BLIP-3): A Family of Open Large Multimodal Models: This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our pre-trained base model exhibits strong in-context learning capabilities and the instruction-tuned model demonstrates competitive performance among open-source LMMs with similar model sizes. In addition, we introduce a safety-tuned model with DPO, aiming to mitigate harmful behaviors such as hallucinations and improve safety. We open-source our models, curated large-scale datasets, and our fine-tuning codebase to facilitate further advancements in LMM research. Associated resources will be available on our project page above.<|reference_end|>
@article{xue2024xgen-mm,
    title={xGen-MM (BLIP-3): A Family of Open Large Multimodal Models},
    author={Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil
  Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant
  Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan,
  Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig
  Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong,
  Ran Xu},
    journal={arXiv preprint arXiv:2408.08872},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.08872},
    primaryClass={cs.CV cs.AI cs.CL}
}