Bring Reason to Vision: Understanding Perception and Reasoning through
  Model Merging
Vision-Language Models (VLMs) combine visual perception with the general
capabilities, such as reasoning, of Large Language Models (LLMs). However, the
mechanisms by which these two abilities can be combined and contribute remain
poorly understood. In this work, we explore to compose perception and reasoning
through model merging that connects parameters of different models. Unlike
previous works that often focus on merging models of the same kind, we propose
merging models across modalities, enabling the incorporation of the reasoning
capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate
that model merging offers a successful pathway to transfer reasoning abilities
from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged
models to understand the internal mechanism of perception and reasoning and how
merging affects it. We find that perception capabilities are predominantly
encoded in the early layers of the model, whereas reasoning is largely
facilitated by the middle-to-late layers. After merging, we observe that all
layers begin to contribute to reasoning, whereas the distribution of perception
abilities across layers remains largely unchanged. These observations shed
light on the potential of model merging as a tool for multimodal integration
and interpretation.
6824c5964fc74fcd88891f6f
Shiqi Chen
claimed_verified
2025-05-18T19:39:29.258Z
62bf99cae140faaa85335ab8
/avatars/bcd2aa1823995b385096e2a68ce3e071.svg
Shiqi Chen
user
ShiqiChen
6824c5964fc74fcd88891f70
Jinghan Zhang
claimed_verified
2025-05-15T10:32:09.768Z
642c5631ab0cc792e4379099
/avatars/c660650950d4804f48329970226e63d1.svg
Jinghan ZHANG
user
jinghan23
6824c5964fc74fcd88891f71
Tongyao Zhu
6824c5964fc74fcd88891f72
Wei Liu
claimed_verified
2025-05-22T07:18:22.397Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
user
PeterV09
6824c5964fc74fcd88891f73
Siyang Gao
6824c5964fc74fcd88891f74
Miao Xiong
claimed_verified
2025-05-27T10:03:41.689Z
6530cf34e7535baecd9620a7
/avatars/e6058a932d88e42b4957734f653cbcfd.svg
Miao Xiong
user
happymio
6824c5964fc74fcd88891f75
Manling Li
6824c5964fc74fcd88891f76
Junxian He
Bring Reason to Vision: Understanding Perception and Reasoning through
  Model Merging
Vision-Language Models (VLMs) combine visual perception with the general
capabilities, such as reasoning, of Large Language Models (LLMs). However, the
mechanisms by which these two abilities can be combined and contribute remain
poorly understood. In this work, we explore to compose perception and reasoning
through model merging that connects parameters of different models. Unlike
previous works that often focus on merging models of the same kind, we propose
merging models across modalities, enabling the incorporation of the reasoning
capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate
that model merging offers a successful pathway to transfer reasoning abilities
from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged
models to understand the internal mechanism of perception and reasoning and how
merging affects it. We find that perception capabilities are predominantly
encoded in the early layers of the model, whereas reasoning is largely
facilitated by the middle-to-late layers. After merging, we observe that all
layers begin to contribute to reasoning, whereas the distribution of perception
abilities across layers remains largely unchanged. These observations shed
light on the potential of model merging as a tool for multimodal integration
and interpretation.
Merging models across modalities effectively transfers reasoning abilities from LLMs to VLMs without additional training, revealing layer-specific contributions to perception and reasoning.
Vision-Language Models Large Language Models model merging multimodal integration perception reasoning