VisionThink: Smart and Efficient Vision Language Model via Reinforcement
  Learning
Recent advancements in vision-language models (VLMs) have improved
performance by increasing the number of visual tokens, which are often
significantly longer than text tokens. However, we observe that most real-world
scenarios do not require such an extensive number of visual tokens. While the
performance drops significantly in a small subset of OCR-related tasks, models
still perform accurately in most other general VQA tasks with only 1/4
resolution. Therefore, we propose to dynamically process distinct samples with
different resolutions, and present a new paradigm for visual token compression,
namely, VisionThink. It starts with a downsampled image and smartly decides
whether it is sufficient for problem solving. Otherwise, the model could output
a special token to request the higher-resolution image. Compared to existing
Efficient VLM methods that compress tokens using fixed pruning ratios or
thresholds, VisionThink autonomously decides whether to compress tokens case by
case. As a result, it demonstrates strong fine-grained visual understanding
capability on OCR-related tasks, and meanwhile saves substantial visual tokens
on simpler tasks. We adopt reinforcement learning and propose the LLM-as-Judge
strategy to successfully apply RL to general VQA tasks. Moreover, we carefully
design a reward function and penalty mechanism to achieve a stable and
reasonable image resize call ratio. Extensive experiments demonstrate the
superiority, efficiency, and effectiveness of our method. Our code is available
at https://github.com/dvlab-research/VisionThink.
6879ba2021b37e676c8e40c9
Senqiao Yang
admin_assigned
2025-07-18T12:43:44.095Z
6527b7280ae663e384eb8499
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
Senqiao Yang
user
Senqiao
6879ba2021b37e676c8e40ca
Junyi Li
admin_assigned
2025-07-18T12:43:50.898Z
6614a9a9ef2015c1b33d3095
https://cdn-avatars.huggingface.co/v1/production/uploads/6614a9a9ef2015c1b33d3095/_rxcYbY47yvHZrB1EpmrU.jpeg
JunyiLi
user
junyili
6879ba2021b37e676c8e40cb
Xin Lai
6879ba2021b37e676c8e40cc
Bei Yu
6879ba2021b37e676c8e40cd
Hengshuang Zhao
6879ba2021b37e676c8e40ce
Jiaya Jia
admin_assigned
2025-07-18T12:44:13.346Z
6848d10fb95a61a03a3f9ee4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/fUwUJjGLkEDtILRefxkMZ.jpeg
Jia
user
Jiaya1
VisionThink: Smart and Efficient Vision Language Model via Reinforcement
  Learning
Recent advancements in vision-language models (VLMs) have improved
performance by increasing the number of visual tokens, which are often
significantly longer than text tokens. However, we observe that most real-world
scenarios do not require such an extensive number of visual tokens. While the
performance drops significantly in a small subset of OCR-related tasks, models
still perform accurately in most other general VQA tasks with only 1/4
resolution. Therefore, we propose to dynamically process distinct samples with
different resolutions, and present a new paradigm for visual token compression,
namely, VisionThink. It starts with a downsampled image and smartly decides
whether it is sufficient for problem solving. Otherwise, the model could output
a special token to request the higher-resolution image. Compared to existing
Efficient VLM methods that compress tokens using fixed pruning ratios or
thresholds, VisionThink autonomously decides whether to compress tokens case by
case. As a result, it demonstrates strong fine-grained visual understanding
capability on OCR-related tasks, and meanwhile saves substantial visual tokens
on simpler tasks. We adopt reinforcement learning and propose the LLM-as-Judge
strategy to successfully apply RL to general VQA tasks. Moreover, we carefully
design a reward function and penalty mechanism to achieve a stable and
reasonable image resize call ratio. Extensive experiments demonstrate the
superiority, efficiency, and effectiveness of our method. Our code is available
at https://github.com/dvlab-research/VisionThink.
VisionThink dynamically adjusts image resolution and visual token processing for efficient and effective vision-language tasks, improving performance and reducing computational cost.
vision-language models visual tokens text tokens downsampled image smart decision-making special token token compression Efficient VLM reinforcement learning LLM-as-Judge reward function penalty mechanism image resize call ratio