title: SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
    Language Models
summary: We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
  series developed upon SPHINX. To improve the architecture and training
  efficiency, we modify the SPHINX framework by removing redundant visual
  encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
  multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
  potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
  dataset covering publicly available resources in language, vision, and
  vision-language tasks. We further enrich this collection with our curated OCR
  intensive and Set-of-Mark datasets, extending the diversity and generality. By
  training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
  LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
  parameter size and multilingual capabilities. Comprehensive benchmarking
  reveals a strong correlation between the multi-modal performance with the data
  and parameter scales. Code and models are released at
  https://github.com/Alpha-VLLM/LLaMA2-Accessory
paper_authors._id: 65c59a8fc83b7f1e35ed1a2e
paper_authors.name: Peng Gao
paper_authors._id: 65c59a8fc83b7f1e35ed1a2f
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:39:39.087Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 65c59a8fc83b7f1e35ed1a30
paper_authors.name: Chris Liu
paper_authors._id: 65c59a8fc83b7f1e35ed1a31
paper_authors.name: Longtian Qiu
paper_authors._id: 65c59a8fc83b7f1e35ed1a32
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T07:14:08.354Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 65c59a8fc83b7f1e35ed1a33
paper_authors.name: Weifeng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:38:08.676Z
paper_authors._id: 63483f3450fe0799927b9b0f
paper_authors.avatarUrl: /avatars/66c2be2d03dafcb91cec0239ca7b8b44.svg
paper_authors.fullname: Weifeng Lin
paper_authors.type: user
paper_authors.user: Singaless
paper_authors._id: 65c59a8fc83b7f1e35ed1a34
paper_authors.name: Shitian Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:38:16.388Z
paper_authors._id: 62c66504031996c36c86976a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
paper_authors.fullname: steve z
paper_authors.type: user
paper_authors.user: stzhao
paper_authors._id: 65c59a8fc83b7f1e35ed1a35
paper_authors.name: Shijie Geng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:04:27.952Z
paper_authors._id: 62130cc0581b98bb1ad00a9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672648250385-62130cc0581b98bb1ad00a9d.jpeg
paper_authors.fullname: Shijie Geng
paper_authors.type: user
paper_authors.user: makitanikaze
paper_authors._id: 65c59a8fc83b7f1e35ed1a36
paper_authors.name: Ziyi Lin
paper_authors._id: 65c59a8fc83b7f1e35ed1a37
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:31:15.753Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 65c59a8fc83b7f1e35ed1a38
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:41:43.454Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 65c59a8fc83b7f1e35ed1a39
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:41:50.941Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 65c59a8fc83b7f1e35ed1a3a
paper_authors.name: Chao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:42:06.755Z
paper_authors._id: 6468a48cff18750165a5aad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
paper_authors.fullname: Chao Xu
paper_authors.type: user
paper_authors.user: chaoxu
paper_authors._id: 65c59a8fc83b7f1e35ed1a3b
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:42:13.507Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 65c59a8fc83b7f1e35ed1a3c
paper_authors.name: Junjun He
paper_authors._id: 65c59a8fc83b7f1e35ed1a3d
paper_authors.name: Hao Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:42:41.197Z
paper_authors._id: 651bda4337fecec1fe8bb09f
paper_authors.avatarUrl: /avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
paper_authors.fullname: Hao Shao
paper_authors.type: user
paper_authors.user: deepcs233
paper_authors._id: 65c59a8fc83b7f1e35ed1a3e
paper_authors.name: Pan Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:42:59.342Z
paper_authors._id: 60f5f68fa7fd83d025749234
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
paper_authors.fullname: Pan Lu
paper_authors.type: user
paper_authors.user: lupantech
paper_authors._id: 65c59a8fc83b7f1e35ed1a3f
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T09:43:07.274Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 65c59a8fc83b7f1e35ed1a40
paper_authors.name: Yu Qiao
paper_title: SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
    Language Models
paper_summary: We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
  series developed upon SPHINX. To improve the architecture and training
  efficiency, we modify the SPHINX framework by removing redundant visual
  encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
  multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
  potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
  dataset covering publicly available resources in language, vision, and
  vision-language tasks. We further enrich this collection with our curated OCR
  intensive and Set-of-Mark datasets, extending the diversity and generality. By
  training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
  LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
  parameter size and multilingual capabilities. Comprehensive benchmarking
  reveals a strong correlation between the multi-modal performance with the data
  and parameter scales. Code and models are released at
  https://github.com/Alpha-VLLM/LLaMA2-Accessory
paper_ai_keywords: Multimodality Large Language Model (MLLM) SPHINX framework visual encoders skip tokens one-stage all-in-one paradigm multi-domain multimodal dataset publicly available resources language vision vision-language tasks OCR intensive Set-of-Mark datasets TinyLlama1.1B InternLM2-7B LLaMA2-13B Mixtral8x7B multilingual capabilities comprehensive benchmarking multi-modal performance