title: DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs
    for Knowledge-Intensive Visual Grounding
summary: Human experts excel at fine-grained visual discrimination by leveraging
  domain knowledge to refine perceptual features, a capability that remains
  underdeveloped in current Multimodal Large Language Models (MLLMs). Despite
  possessing vast expert-level knowledge, MLLMs struggle to integrate reasoning
  into visual perception, often generating direct responses without deeper
  analysis. To bridge this gap, we introduce knowledge-intensive visual grounding
  (KVG), a novel visual grounding task that requires both fine-grained perception
  and domain-specific knowledge integration. To address the challenges of KVG, we
  propose DeepPerception, an MLLM enhanced with cognitive visual perception
  capabilities. Our approach consists of (1) an automated data synthesis pipeline
  that generates high-quality, knowledge-aligned training samples, and (2) a
  two-stage training framework combining supervised fine-tuning for cognitive
  reasoning scaffolding and reinforcement learning to optimize
  perception-cognition synergy. To benchmark performance, we introduce KVG-Bench
  a comprehensive dataset spanning 10 domains with 1.3K manually curated test
  cases. Experimental results demonstrate that DeepPerception significantly
  outperforms direct fine-tuning, achieving +8.08\% accuracy improvements on
  KVG-Bench and exhibiting +4.60\% superior cross-domain generalization over
  baseline approaches. Our findings highlight the importance of integrating
  cognitive processes into MLLMs for human-like visual perception and open new
  directions for multimodal reasoning research. The data, codes, and models are
  released at https://github.com/thunlp/DeepPerception.
paper_authors._id: 67da2c83aa2c34f7d95e46ff
paper_authors.name: Xinyu Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:31.865Z
paper_authors._id: 63c3b67ec7d7f4c63a4eea3a
paper_authors.avatarUrl: /avatars/4a5f98cb6b0c1e37a2c09af72f7a9946.svg
paper_authors.fullname: Xinyu Ma
paper_authors.type: user
paper_authors.user: MaxyLee
paper_authors._id: 67da2c83aa2c34f7d95e4700
paper_authors.name: Ziyang Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:35.757Z
paper_authors._id: 65903c4aa78a277803bde77b
paper_authors.avatarUrl: /avatars/061388320ccf1a66e1e99519dd426a60.svg
paper_authors.fullname: Ziyang Ding
paper_authors.type: user
paper_authors.user: sdudzy
paper_authors._id: 67da2c83aa2c34f7d95e4701
paper_authors.name: Zhicong Luo
paper_authors._id: 67da2c83aa2c34f7d95e4702
paper_authors.name: Chi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:44:39.356Z
paper_authors._id: 642086ed290342c5df85662d
paper_authors.avatarUrl: /avatars/915a4d7b89455ae97b8544c79286ddf8.svg
paper_authors.fullname: Chi Chen
paper_authors.type: user
paper_authors.user: carboncoo
paper_authors._id: 67da2c83aa2c34f7d95e4703
paper_authors.name: Zonghao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:23:57.046Z
paper_authors._id: 6491af36c1741666238f3bff
paper_authors.avatarUrl: /avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
paper_authors.fullname: Zonghao Guo
paper_authors.type: user
paper_authors.user: guozonghao96
paper_authors._id: 67da2c83aa2c34f7d95e4704
paper_authors.name: Derek F. Wong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:23:48.590Z
paper_authors._id: 648bd523805e4bcc541ec320
paper_authors.avatarUrl: /avatars/443ca0c7cbeda3a08eb4af6a0e2da8bc.svg
paper_authors.fullname: Derek Wong
paper_authors.type: user
paper_authors.user: derekfw
paper_authors._id: 67da2c83aa2c34f7d95e4705
paper_authors.name: Xiaoyi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:23:24.786Z
paper_authors._id: 6502c59b8670cc10e6b6b039
paper_authors.avatarUrl: /avatars/ae96d1fa2bed83cd427d18ca25d69efa.svg
paper_authors.fullname: Xiaoyi Feng
paper_authors.type: user
paper_authors.user: LecterF
paper_authors._id: 67da2c83aa2c34f7d95e4706
paper_authors.name: Maosong Sun
paper_title: DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs
    for Knowledge-Intensive Visual Grounding
paper_summary: Human experts excel at fine-grained visual discrimination by leveraging
  domain knowledge to refine perceptual features, a capability that remains
  underdeveloped in current Multimodal Large Language Models (MLLMs). Despite
  possessing vast expert-level knowledge, MLLMs struggle to integrate reasoning
  into visual perception, often generating direct responses without deeper
  analysis. To bridge this gap, we introduce knowledge-intensive visual grounding
  (KVG), a novel visual grounding task that requires both fine-grained perception
  and domain-specific knowledge integration. To address the challenges of KVG, we
  propose DeepPerception, an MLLM enhanced with cognitive visual perception
  capabilities. Our approach consists of (1) an automated data synthesis pipeline
  that generates high-quality, knowledge-aligned training samples, and (2) a
  two-stage training framework combining supervised fine-tuning for cognitive
  reasoning scaffolding and reinforcement learning to optimize
  perception-cognition synergy. To benchmark performance, we introduce KVG-Bench
  a comprehensive dataset spanning 10 domains with 1.3K manually curated test
  cases. Experimental results demonstrate that DeepPerception significantly
  outperforms direct fine-tuning, achieving +8.08\% accuracy improvements on
  KVG-Bench and exhibiting +4.60\% superior cross-domain generalization over
  baseline approaches. Our findings highlight the importance of integrating
  cognitive processes into MLLMs for human-like visual perception and open new
  directions for multimodal reasoning research. The data, codes, and models are
  released at https://github.com/thunlp/DeepPerception.
paper_ai_summary: DeepPerception enhances Multimodal Large Language Models with cognitive visual perception capabilities, improving accuracy and cross-domain generalization through a novel visual grounding task and a two-stage training framework.
paper_ai_keywords: knowledge-intensive visual grounding DeepPerception cognitive visual perception automated data synthesis pipeline supervised fine-tuning reinforcement learning perception-cognition synergy KVG-Bench