DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs
  for Knowledge-Intensive Visual Grounding
Human experts excel at fine-grained visual discrimination by leveraging
domain knowledge to refine perceptual features, a capability that remains
underdeveloped in current Multimodal Large Language Models (MLLMs). Despite
possessing vast expert-level knowledge, MLLMs struggle to integrate reasoning
into visual perception, often generating direct responses without deeper
analysis. To bridge this gap, we introduce knowledge-intensive visual grounding
(KVG), a novel visual grounding task that requires both fine-grained perception
and domain-specific knowledge integration. To address the challenges of KVG, we
propose DeepPerception, an MLLM enhanced with cognitive visual perception
capabilities. Our approach consists of (1) an automated data synthesis pipeline
that generates high-quality, knowledge-aligned training samples, and (2) a
two-stage training framework combining supervised fine-tuning for cognitive
reasoning scaffolding and reinforcement learning to optimize
perception-cognition synergy. To benchmark performance, we introduce KVG-Bench
a comprehensive dataset spanning 10 domains with 1.3K manually curated test
cases. Experimental results demonstrate that DeepPerception significantly
outperforms direct fine-tuning, achieving +8.08\% accuracy improvements on
KVG-Bench and exhibiting +4.60\% superior cross-domain generalization over
baseline approaches. Our findings highlight the importance of integrating
cognitive processes into MLLMs for human-like visual perception and open new
directions for multimodal reasoning research. The data, codes, and models are
released at https://github.com/thunlp/DeepPerception.
67da2c83aa2c34f7d95e46ff
Xinyu Ma
claimed_verified
2025-03-19T09:44:31.865Z
63c3b67ec7d7f4c63a4eea3a
/avatars/4a5f98cb6b0c1e37a2c09af72f7a9946.svg
Xinyu Ma
user
MaxyLee
67da2c83aa2c34f7d95e4700
Ziyang Ding
claimed_verified
2025-03-19T09:44:35.757Z
65903c4aa78a277803bde77b
/avatars/061388320ccf1a66e1e99519dd426a60.svg
Ziyang Ding
user
sdudzy
67da2c83aa2c34f7d95e4701
Zhicong Luo
67da2c83aa2c34f7d95e4702
Chi Chen
claimed_verified
2025-03-19T09:44:39.356Z
642086ed290342c5df85662d
/avatars/915a4d7b89455ae97b8544c79286ddf8.svg
Chi Chen
user
carboncoo
67da2c83aa2c34f7d95e4703
Zonghao Guo
admin_assigned
2025-03-19T13:23:57.046Z
6491af36c1741666238f3bff
/avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
Zonghao Guo
user
guozonghao96
67da2c83aa2c34f7d95e4704
Derek F. Wong
admin_assigned
2025-03-19T13:23:48.590Z
648bd523805e4bcc541ec320
/avatars/443ca0c7cbeda3a08eb4af6a0e2da8bc.svg
Derek Wong
user
derekfw
67da2c83aa2c34f7d95e4705
Xiaoyi Feng
admin_assigned
2025-03-19T13:23:24.786Z
6502c59b8670cc10e6b6b039
/avatars/ae96d1fa2bed83cd427d18ca25d69efa.svg
Xiaoyi Feng
user
LecterF
67da2c83aa2c34f7d95e4706
Maosong Sun
DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs
  for Knowledge-Intensive Visual Grounding
Human experts excel at fine-grained visual discrimination by leveraging
domain knowledge to refine perceptual features, a capability that remains
underdeveloped in current Multimodal Large Language Models (MLLMs). Despite
possessing vast expert-level knowledge, MLLMs struggle to integrate reasoning
into visual perception, often generating direct responses without deeper
analysis. To bridge this gap, we introduce knowledge-intensive visual grounding
(KVG), a novel visual grounding task that requires both fine-grained perception
and domain-specific knowledge integration. To address the challenges of KVG, we
propose DeepPerception, an MLLM enhanced with cognitive visual perception
capabilities. Our approach consists of (1) an automated data synthesis pipeline
that generates high-quality, knowledge-aligned training samples, and (2) a
two-stage training framework combining supervised fine-tuning for cognitive
reasoning scaffolding and reinforcement learning to optimize
perception-cognition synergy. To benchmark performance, we introduce KVG-Bench
a comprehensive dataset spanning 10 domains with 1.3K manually curated test
cases. Experimental results demonstrate that DeepPerception significantly
outperforms direct fine-tuning, achieving +8.08\% accuracy improvements on
KVG-Bench and exhibiting +4.60\% superior cross-domain generalization over
baseline approaches. Our findings highlight the importance of integrating
cognitive processes into MLLMs for human-like visual perception and open new
directions for multimodal reasoning research. The data, codes, and models are
released at https://github.com/thunlp/DeepPerception.
DeepPerception enhances Multimodal Large Language Models with cognitive visual perception capabilities, improving accuracy and cross-domain generalization through a novel visual grounding task and a two-stage training framework.
knowledge-intensive visual grounding DeepPerception cognitive visual perception automated data synthesis pipeline supervised fine-tuning reinforcement learning perception-cognition synergy KVG-Bench