title: SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for
    Multi-modal Large Language Models
summary: We present SPHINX, a versatile multi-modal large language model (MLLM) with a
  joint mixing of model weights, tuning tasks, and visual embeddings. First, for
  stronger vision-language alignment, we unfreeze the large language model (LLM)
  during pre-training, and introduce a weight mix strategy between LLMs trained
  by real-world and synthetic data. By directly integrating the weights from two
  domains, the mixed LLM can efficiently incorporate diverse semantics with
  favorable robustness. Then, to enable multi-purpose capabilities, we mix a
  variety of tasks for joint visual instruction tuning, and design task-specific
  instructions to avoid inter-task conflict. In addition to the basic visual
  question answering, we include more challenging tasks such as region-level
  understanding, caption grounding, document layout detection, and human pose
  estimation, contributing to mutual enhancement over different scenarios.
  Additionally, we propose to extract comprehensive visual embeddings from
  various network architectures, pre-training paradigms, and information
  granularity, providing language models with more robust image representations.
  Based on our proposed joint mixing, SPHINX exhibits superior multi-modal
  understanding capabilities on a wide range of applications. On top of this, we
  further propose an efficient strategy aiming to better capture fine-grained
  appearances of high-resolution images. With a mixing of different scales and
  high-resolution sub-images, SPHINX attains exceptional visual parsing and
  reasoning performance on existing evaluation benchmarks. We hope our work may
  cast a light on the exploration of joint mixing in future MLLM research. Code
  is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
paper_authors._id: 6552f0561d1a7cefd8e843e4
paper_authors.name: Ziyi Lin
paper_authors._id: 6552f0561d1a7cefd8e843e5
paper_authors.name: Chris Liu
paper_authors._id: 6552f0561d1a7cefd8e843e6
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:40:09.742Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 6552f0561d1a7cefd8e843e7
paper_authors.name: Peng Gao
paper_authors._id: 6552f0561d1a7cefd8e843e8
paper_authors.name: Longtian Qiu
paper_authors._id: 6552f0561d1a7cefd8e843e9
paper_authors.name: Han Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:03:29.725Z
paper_authors._id: 666aa99cd1652853e4f9a8b9
paper_authors.avatarUrl: /avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
paper_authors.fullname: HanXiao
paper_authors.type: user
paper_authors.user: HanXiao1999
paper_authors._id: 6552f0561d1a7cefd8e843ea
paper_authors.name: Han Qiu
paper_authors._id: 6552f0561d1a7cefd8e843eb
paper_authors.name: Chen Lin
paper_authors._id: 6552f0561d1a7cefd8e843ec
paper_authors.name: Wenqi Shao
paper_authors._id: 6552f0561d1a7cefd8e843ed
paper_authors.name: Keqin Chen
paper_authors._id: 6552f0561d1a7cefd8e843ee
paper_authors.name: Jiaming Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:36:45.193Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 6552f0561d1a7cefd8e843ef
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T12:33:04.806Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_authors._id: 6552f0561d1a7cefd8e843f0
paper_authors.name: Yichi Zhang
paper_authors._id: 6552f0561d1a7cefd8e843f1
paper_authors.name: Xuming He
paper_authors._id: 6552f0561d1a7cefd8e843f2
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:49:47.542Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 6552f0561d1a7cefd8e843f3
paper_authors.name: Yu Qiao
paper_title: SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for
    Multi-modal Large Language Models
paper_summary: We present SPHINX, a versatile multi-modal large language model (MLLM) with a
  joint mixing of model weights, tuning tasks, and visual embeddings. First, for
  stronger vision-language alignment, we unfreeze the large language model (LLM)
  during pre-training, and introduce a weight mix strategy between LLMs trained
  by real-world and synthetic data. By directly integrating the weights from two
  domains, the mixed LLM can efficiently incorporate diverse semantics with
  favorable robustness. Then, to enable multi-purpose capabilities, we mix a
  variety of tasks for joint visual instruction tuning, and design task-specific
  instructions to avoid inter-task conflict. In addition to the basic visual
  question answering, we include more challenging tasks such as region-level
  understanding, caption grounding, document layout detection, and human pose
  estimation, contributing to mutual enhancement over different scenarios.
  Additionally, we propose to extract comprehensive visual embeddings from
  various network architectures, pre-training paradigms, and information
  granularity, providing language models with more robust image representations.
  Based on our proposed joint mixing, SPHINX exhibits superior multi-modal
  understanding capabilities on a wide range of applications. On top of this, we
  further propose an efficient strategy aiming to better capture fine-grained
  appearances of high-resolution images. With a mixing of different scales and
  high-resolution sub-images, SPHINX attains exceptional visual parsing and
  reasoning performance on existing evaluation benchmarks. We hope our work may
  cast a light on the exploration of joint mixing in future MLLM research. Code
  is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
paper_ai_summary: SPHINX, a multi-modal large language model, enhances vision-language alignment and multi-purpose capabilities through joint weight and task mixing, resulting in superior performance across various applications and image parsing tasks.
paper_ai_keywords: multi-modal large language model weight mix strategy large language model real-world data synthetic data visual instruction tuning task-specific instructions region-level understanding caption grounding document layout detection human pose estimation visual embeddings network architectures pre-training paradigms information granularity efficient strategy fine-grained appearances high-resolution images visual parsing reasoning performance evaluation benchmarks multi-modal understanding capabilities