OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning
  via Iterative Self-Improvement
Recent advancements demonstrated by DeepSeek-R1 have shown that complex
reasoning abilities in large language models (LLMs), including sophisticated
behaviors such as self-verification and self-correction, can be achieved by RL
with verifiable rewards and significantly improves model performance on
challenging tasks such as AIME. Motivated by these findings, our study
investigates whether similar reasoning capabilities can be successfully
integrated into large vision-language models (LVLMs) and assesses their impact
on challenging multimodal reasoning tasks. We consider an approach that
iteratively leverages supervised fine-tuning (SFT) on lightweight training data
and Reinforcement Learning (RL) to further improve model generalization.
Initially, reasoning capabilities were distilled from pure-text R1 models by
generating reasoning steps using high-quality captions of the images sourced
from diverse visual datasets. Subsequently, iterative RL training further
enhance reasoning skills, with each iteration's RL-improved model generating
refined SFT datasets for the next round. This iterative process yielded
OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on
challenging benchmarks such as MathVista, MathVerse, and MathVision,
demonstrating the potential of our strategy for robust vision-language
reasoning. The code, model and data are held at
https://github.com/yihedeng9/OpenVLThinker.
67e0bcc9e5fa0da84e121032
Yihe Deng
admin_assigned
2025-03-24T12:41:41.202Z
642f4c789b2484d7d8551a93
https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
Yihe Deng
user
ydeng9
67e0bcc9e5fa0da84e121033
Hritik Bansal
admin_assigned
2025-03-24T12:41:47.419Z
61c5c25705aa54027c52f7b3
/avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
Hritik Bansal
user
hbXNov
67e0bcc9e5fa0da84e121034
Fan Yin
admin_assigned
2025-03-24T12:42:02.800Z
639bd19e445b133a4e94c3ee
/avatars/63ab9919b5e67261cce9007192a70deb.svg
Fan Yin
user
fanyin3639
67e0bcc9e5fa0da84e121035
Nanyun Peng
67e0bcc9e5fa0da84e121036
Wei Wang
admin_assigned
2025-03-24T12:42:09.903Z
62fa0ffe0697d224219a0cb7
/avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
Wei Wang
user
WeiWang
67e0bcc9e5fa0da84e121037
Kai-Wei Chang
admin_assigned
2025-03-24T12:42:16.596Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning
  via Iterative Self-Improvement
Recent advancements demonstrated by DeepSeek-R1 have shown that complex
reasoning abilities in large language models (LLMs), including sophisticated
behaviors such as self-verification and self-correction, can be achieved by RL
with verifiable rewards and significantly improves model performance on
challenging tasks such as AIME. Motivated by these findings, our study
investigates whether similar reasoning capabilities can be successfully
integrated into large vision-language models (LVLMs) and assesses their impact
on challenging multimodal reasoning tasks. We consider an approach that
iteratively leverages supervised fine-tuning (SFT) on lightweight training data
and Reinforcement Learning (RL) to further improve model generalization.
Initially, reasoning capabilities were distilled from pure-text R1 models by
generating reasoning steps using high-quality captions of the images sourced
from diverse visual datasets. Subsequently, iterative RL training further
enhance reasoning skills, with each iteration's RL-improved model generating
refined SFT datasets for the next round. This iterative process yielded
OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on
challenging benchmarks such as MathVista, MathVerse, and MathVision,
demonstrating the potential of our strategy for robust vision-language
reasoning. The code, model and data are held at
https://github.com/yihedeng9/OpenVLThinker.
Iterative supervised fine-tuning and reinforcement learning improve reasoning capabilities in large vision-language models, enhancing performance on multimodal reasoning tasks like MathVista, MathVerse, and MathVision.
DeepSeek-R1 reinforcement learning verifiable rewards large language models large vision-language models supervised fine-tuning reasoning steps high-quality captions visual datasets MathVista MathVerse MathVision