title: Bring Reason to Vision: Understanding Perception and Reasoning through
    Model Merging
summary: Vision-Language Models (VLMs) combine visual perception with the general
  capabilities, such as reasoning, of Large Language Models (LLMs). However, the
  mechanisms by which these two abilities can be combined and contribute remain
  poorly understood. In this work, we explore to compose perception and reasoning
  through model merging that connects parameters of different models. Unlike
  previous works that often focus on merging models of the same kind, we propose
  merging models across modalities, enabling the incorporation of the reasoning
  capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate
  that model merging offers a successful pathway to transfer reasoning abilities
  from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged
  models to understand the internal mechanism of perception and reasoning and how
  merging affects it. We find that perception capabilities are predominantly
  encoded in the early layers of the model, whereas reasoning is largely
  facilitated by the middle-to-late layers. After merging, we observe that all
  layers begin to contribute to reasoning, whereas the distribution of perception
  abilities across layers remains largely unchanged. These observations shed
  light on the potential of model merging as a tool for multimodal integration
  and interpretation.
paper_authors._id: 6824c5964fc74fcd88891f6f
paper_authors.name: Shiqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T19:39:29.258Z
paper_authors._id: 62bf99cae140faaa85335ab8
paper_authors.avatarUrl: /avatars/bcd2aa1823995b385096e2a68ce3e071.svg
paper_authors.fullname: Shiqi Chen
paper_authors.type: user
paper_authors.user: ShiqiChen
paper_authors._id: 6824c5964fc74fcd88891f70
paper_authors.name: Jinghan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:32:09.768Z
paper_authors._id: 642c5631ab0cc792e4379099
paper_authors.avatarUrl: /avatars/c660650950d4804f48329970226e63d1.svg
paper_authors.fullname: Jinghan ZHANG
paper_authors.type: user
paper_authors.user: jinghan23
paper_authors._id: 6824c5964fc74fcd88891f71
paper_authors.name: Tongyao Zhu
paper_authors._id: 6824c5964fc74fcd88891f72
paper_authors.name: Wei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:18:22.397Z
paper_authors._id: 6458af46f4d212d780bd7c68
paper_authors.avatarUrl: /avatars/832fd34bcc041b0b7b551873a459fc3c.svg
paper_authors.fullname: Wei Liu
paper_authors.type: user
paper_authors.user: PeterV09
paper_authors._id: 6824c5964fc74fcd88891f73
paper_authors.name: Siyang Gao
paper_authors._id: 6824c5964fc74fcd88891f74
paper_authors.name: Miao Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T10:03:41.689Z
paper_authors._id: 6530cf34e7535baecd9620a7
paper_authors.avatarUrl: /avatars/e6058a932d88e42b4957734f653cbcfd.svg
paper_authors.fullname: Miao Xiong
paper_authors.type: user
paper_authors.user: happymio
paper_authors._id: 6824c5964fc74fcd88891f75
paper_authors.name: Manling Li
paper_authors._id: 6824c5964fc74fcd88891f76
paper_authors.name: Junxian He
paper_title: Bring Reason to Vision: Understanding Perception and Reasoning through
    Model Merging
paper_summary: Vision-Language Models (VLMs) combine visual perception with the general
  capabilities, such as reasoning, of Large Language Models (LLMs). However, the
  mechanisms by which these two abilities can be combined and contribute remain
  poorly understood. In this work, we explore to compose perception and reasoning
  through model merging that connects parameters of different models. Unlike
  previous works that often focus on merging models of the same kind, we propose
  merging models across modalities, enabling the incorporation of the reasoning
  capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate
  that model merging offers a successful pathway to transfer reasoning abilities
  from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged
  models to understand the internal mechanism of perception and reasoning and how
  merging affects it. We find that perception capabilities are predominantly
  encoded in the early layers of the model, whereas reasoning is largely
  facilitated by the middle-to-late layers. After merging, we observe that all
  layers begin to contribute to reasoning, whereas the distribution of perception
  abilities across layers remains largely unchanged. These observations shed
  light on the potential of model merging as a tool for multimodal integration
  and interpretation.
paper_ai_summary: Merging models across modalities effectively transfers reasoning abilities from LLMs to VLMs without additional training, revealing layer-specific contributions to perception and reasoning.
paper_ai_keywords: Vision-Language Models Large Language Models model merging multimodal integration perception reasoning