SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for
  Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a
joint mixing of model weights, tuning tasks, and visual embeddings. First, for
stronger vision-language alignment, we unfreeze the large language model (LLM)
during pre-training, and introduce a weight mix strategy between LLMs trained
by real-world and synthetic data. By directly integrating the weights from two
domains, the mixed LLM can efficiently incorporate diverse semantics with
favorable robustness. Then, to enable multi-purpose capabilities, we mix a
variety of tasks for joint visual instruction tuning, and design task-specific
instructions to avoid inter-task conflict. In addition to the basic visual
question answering, we include more challenging tasks such as region-level
understanding, caption grounding, document layout detection, and human pose
estimation, contributing to mutual enhancement over different scenarios.
Additionally, we propose to extract comprehensive visual embeddings from
various network architectures, pre-training paradigms, and information
granularity, providing language models with more robust image representations.
Based on our proposed joint mixing, SPHINX exhibits superior multi-modal
understanding capabilities on a wide range of applications. On top of this, we
further propose an efficient strategy aiming to better capture fine-grained
appearances of high-resolution images. With a mixing of different scales and
high-resolution sub-images, SPHINX attains exceptional visual parsing and
reasoning performance on existing evaluation benchmarks. We hope our work may
cast a light on the exploration of joint mixing in future MLLM research. Code
is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
6552f0561d1a7cefd8e843e4
Ziyi Lin
6552f0561d1a7cefd8e843e5
Chris Liu
6552f0561d1a7cefd8e843e6
Renrui Zhang
claimed_verified
2024-03-22T08:40:09.742Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
6552f0561d1a7cefd8e843e7
Peng Gao
6552f0561d1a7cefd8e843e8
Longtian Qiu
6552f0561d1a7cefd8e843e9
Han Xiao
claimed_verified
2025-05-28T09:03:29.725Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
6552f0561d1a7cefd8e843ea
Han Qiu
6552f0561d1a7cefd8e843eb
Chen Lin
6552f0561d1a7cefd8e843ec
Wenqi Shao
6552f0561d1a7cefd8e843ed
Keqin Chen
6552f0561d1a7cefd8e843ee
Jiaming Han
claimed_verified
2023-11-14T09:36:45.193Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
6552f0561d1a7cefd8e843ef
Siyuan Huang
admin_assigned
2023-11-14T12:33:04.806Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
6552f0561d1a7cefd8e843f0
Yichi Zhang
6552f0561d1a7cefd8e843f1
Xuming He
6552f0561d1a7cefd8e843f2
Hongsheng Li
claimed_verified
2024-02-05T10:49:47.542Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
6552f0561d1a7cefd8e843f3
Yu Qiao
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for
  Multi-modal Large Language Models
We present SPHINX, a versatile multi-modal large language model (MLLM) with a
joint mixing of model weights, tuning tasks, and visual embeddings. First, for
stronger vision-language alignment, we unfreeze the large language model (LLM)
during pre-training, and introduce a weight mix strategy between LLMs trained
by real-world and synthetic data. By directly integrating the weights from two
domains, the mixed LLM can efficiently incorporate diverse semantics with
favorable robustness. Then, to enable multi-purpose capabilities, we mix a
variety of tasks for joint visual instruction tuning, and design task-specific
instructions to avoid inter-task conflict. In addition to the basic visual
question answering, we include more challenging tasks such as region-level
understanding, caption grounding, document layout detection, and human pose
estimation, contributing to mutual enhancement over different scenarios.
Additionally, we propose to extract comprehensive visual embeddings from
various network architectures, pre-training paradigms, and information
granularity, providing language models with more robust image representations.
Based on our proposed joint mixing, SPHINX exhibits superior multi-modal
understanding capabilities on a wide range of applications. On top of this, we
further propose an efficient strategy aiming to better capture fine-grained
appearances of high-resolution images. With a mixing of different scales and
high-resolution sub-images, SPHINX attains exceptional visual parsing and
reasoning performance on existing evaluation benchmarks. We hope our work may
cast a light on the exploration of joint mixing in future MLLM research. Code
is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.
SPHINX, a multi-modal large language model, enhances vision-language alignment and multi-purpose capabilities through joint weight and task mixing, resulting in superior performance across various applications and image parsing tasks.
multi-modal large language model weight mix strategy large language model real-world data synthetic data visual instruction tuning task-specific instructions region-level understanding caption grounding document layout detection human pose estimation visual embeddings network architectures pre-training paradigms information granularity efficient strategy fine-grained appearances high-resolution images visual parsing reasoning performance evaluation benchmarks multi-modal understanding capabilities