title: Introducing Visual Perception Token into Multimodal Large Language Model
summary: To utilize visual information, Multimodal Large Language Model (MLLM) relies
  on the perception process of its vision encoder. The completeness and accuracy
  of visual perception significantly influence the precision of spatial
  reasoning, fine-grained understanding, and other tasks. However, MLLM still
  lacks the autonomous capability to control its own visual perception processes,
  for example, selectively reviewing specific regions of an image or focusing on
  information related to specific object categories. In this work, we propose the
  concept of Visual Perception Token, aiming to empower MLLM with a mechanism to
  control its visual perception processes. We design two types of Visual
  Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding
  Token. MLLMs autonomously generate these tokens, just as they generate text,
  and use them to trigger additional visual perception actions. The Region
  Selection Token explicitly identifies specific regions in an image that require
  further perception, while the Vision Re-Encoding Token uses its hidden states
  as control signals to guide additional visual perception processes. Extensive
  experiments demonstrate the advantages of these tokens in handling spatial
  reasoning, improving fine-grained understanding, and other tasks. On average,
  the introduction of Visual Perception Tokens improves the performance of a 2B
  model by 23.6\%, increasing its score from 0.572 to 0.708, and even outperforms
  a 7B parameter model by 13.4\% (from 0.624). Please check out our repo
  https://github.com/yu-rp/VisualPerceptionToken
paper_authors._id: 67bddd63c7d8b835b82ced9a
paper_authors.name: Runpeng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T09:14:07.580Z
paper_authors._id: 635364b3c41f548fe39db945
paper_authors.avatarUrl: /avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg
paper_authors.fullname: Runpeng Yu
paper_authors.type: user
paper_authors.user: rp-yu
paper_authors._id: 67bddd63c7d8b835b82ced9b
paper_authors.name: Xinyin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T09:14:13.670Z
paper_authors._id: 64396ebc21221ac7411852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
paper_authors.fullname: Xinyin Ma
paper_authors.type: user
paper_authors.user: horseee
paper_authors._id: 67bddd63c7d8b835b82ced9c
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T09:14:53.838Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: Introducing Visual Perception Token into Multimodal Large Language Model
paper_summary: To utilize visual information, Multimodal Large Language Model (MLLM) relies
  on the perception process of its vision encoder. The completeness and accuracy
  of visual perception significantly influence the precision of spatial
  reasoning, fine-grained understanding, and other tasks. However, MLLM still
  lacks the autonomous capability to control its own visual perception processes,
  for example, selectively reviewing specific regions of an image or focusing on
  information related to specific object categories. In this work, we propose the
  concept of Visual Perception Token, aiming to empower MLLM with a mechanism to
  control its visual perception processes. We design two types of Visual
  Perception Tokens, termed the Region Selection Token and the Vision Re-Encoding
  Token. MLLMs autonomously generate these tokens, just as they generate text,
  and use them to trigger additional visual perception actions. The Region
  Selection Token explicitly identifies specific regions in an image that require
  further perception, while the Vision Re-Encoding Token uses its hidden states
  as control signals to guide additional visual perception processes. Extensive
  experiments demonstrate the advantages of these tokens in handling spatial
  reasoning, improving fine-grained understanding, and other tasks. On average,
  the introduction of Visual Perception Tokens improves the performance of a 2B
  model by 23.6\%, increasing its score from 0.572 to 0.708, and even outperforms
  a 7B parameter model by 13.4\% (from 0.624). Please check out our repo
  https://github.com/yu-rp/VisualPerceptionToken
paper_ai_summary: Visual Perception Tokens improve Multimodal Large Language Models' control over visual perception processes, enhancing tasks like spatial reasoning and fine-grained understanding.
paper_ai_keywords: Multimodal Large Language Model vision encoder Visual Perception Token Region Selection Token Vision Re-Encoding Token