title: OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning
    via Iterative Self-Improvement
summary: Recent advancements demonstrated by DeepSeek-R1 have shown that complex
  reasoning abilities in large language models (LLMs), including sophisticated
  behaviors such as self-verification and self-correction, can be achieved by RL
  with verifiable rewards and significantly improves model performance on
  challenging tasks such as AIME. Motivated by these findings, our study
  investigates whether similar reasoning capabilities can be successfully
  integrated into large vision-language models (LVLMs) and assesses their impact
  on challenging multimodal reasoning tasks. We consider an approach that
  iteratively leverages supervised fine-tuning (SFT) on lightweight training data
  and Reinforcement Learning (RL) to further improve model generalization.
  Initially, reasoning capabilities were distilled from pure-text R1 models by
  generating reasoning steps using high-quality captions of the images sourced
  from diverse visual datasets. Subsequently, iterative RL training further
  enhance reasoning skills, with each iteration's RL-improved model generating
  refined SFT datasets for the next round. This iterative process yielded
  OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on
  challenging benchmarks such as MathVista, MathVerse, and MathVision,
  demonstrating the potential of our strategy for robust vision-language
  reasoning. The code, model and data are held at
  https://github.com/yihedeng9/OpenVLThinker.
paper_authors._id: 67e0bcc9e5fa0da84e121032
paper_authors.name: Yihe Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:41:41.202Z
paper_authors._id: 642f4c789b2484d7d8551a93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
paper_authors.fullname: Yihe Deng
paper_authors.type: user
paper_authors.user: ydeng9
paper_authors._id: 67e0bcc9e5fa0da84e121033
paper_authors.name: Hritik Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:41:47.419Z
paper_authors._id: 61c5c25705aa54027c52f7b3
paper_authors.avatarUrl: /avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
paper_authors.fullname: Hritik Bansal
paper_authors.type: user
paper_authors.user: hbXNov
paper_authors._id: 67e0bcc9e5fa0da84e121034
paper_authors.name: Fan Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:02.800Z
paper_authors._id: 639bd19e445b133a4e94c3ee
paper_authors.avatarUrl: /avatars/63ab9919b5e67261cce9007192a70deb.svg
paper_authors.fullname: Fan Yin
paper_authors.type: user
paper_authors.user: fanyin3639
paper_authors._id: 67e0bcc9e5fa0da84e121035
paper_authors.name: Nanyun Peng
paper_authors._id: 67e0bcc9e5fa0da84e121036
paper_authors.name: Wei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:09.903Z
paper_authors._id: 62fa0ffe0697d224219a0cb7
paper_authors.avatarUrl: /avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
paper_authors.fullname: Wei Wang
paper_authors.type: user
paper_authors.user: WeiWang
paper_authors._id: 67e0bcc9e5fa0da84e121037
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:16.596Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_title: OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning
    via Iterative Self-Improvement
paper_summary: Recent advancements demonstrated by DeepSeek-R1 have shown that complex
  reasoning abilities in large language models (LLMs), including sophisticated
  behaviors such as self-verification and self-correction, can be achieved by RL
  with verifiable rewards and significantly improves model performance on
  challenging tasks such as AIME. Motivated by these findings, our study
  investigates whether similar reasoning capabilities can be successfully
  integrated into large vision-language models (LVLMs) and assesses their impact
  on challenging multimodal reasoning tasks. We consider an approach that
  iteratively leverages supervised fine-tuning (SFT) on lightweight training data
  and Reinforcement Learning (RL) to further improve model generalization.
  Initially, reasoning capabilities were distilled from pure-text R1 models by
  generating reasoning steps using high-quality captions of the images sourced
  from diverse visual datasets. Subsequently, iterative RL training further
  enhance reasoning skills, with each iteration's RL-improved model generating
  refined SFT datasets for the next round. This iterative process yielded
  OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on
  challenging benchmarks such as MathVista, MathVerse, and MathVision,
  demonstrating the potential of our strategy for robust vision-language
  reasoning. The code, model and data are held at
  https://github.com/yihedeng9/OpenVLThinker.
paper_ai_summary: Iterative supervised fine-tuning and reinforcement learning improve reasoning capabilities in large vision-language models, enhancing performance on multimodal reasoning tasks like MathVista, MathVerse, and MathVision.
paper_ai_keywords: DeepSeek-R1 reinforcement learning verifiable rewards large language models large vision-language models supervised fine-tuning reasoning steps high-quality captions visual datasets MathVista MathVerse MathVision