title: SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
summary: Recent advances have shown success in eliciting strong reasoning abilities in
  multimodal large language models (MLLMs) through rule-based reinforcement
  learning (RL) with outcome rewards. However, this paradigm typically lacks
  supervision over the thinking process leading to the final outcome.As a result,
  the model may learn sub-optimal reasoning strategies, which can hinder its
  generalization ability. In light of this, we propose SophiaVL-R1, as an attempt
  to add reward signals for the thinking process in this paradigm. To achieve
  this, we first train a thinking reward model that evaluates the quality of the
  entire thinking process. Given that the thinking reward may be unreliable for
  certain samples due to reward hacking, we propose the Trust-GRPO method, which
  assigns a trustworthiness weight to the thinking reward during training. This
  weight is computed based on the thinking reward comparison of responses leading
  to correct answers versus incorrect answers, helping to mitigate the impact of
  potentially unreliable thinking rewards. Moreover, we design an annealing
  training strategy that gradually reduces the thinking reward over time,
  allowing the model to rely more on the accurate rule-based outcome reward in
  later training stages. Experiments show that our SophiaVL-R1 surpasses a series
  of reasoning MLLMs on various benchmarks (e.g., MathVisita, MMMU),
  demonstrating strong reasoning and generalization capabilities. Notably, our
  SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite
  the latter having 10 times more parameters. All code, models, and datasets are
  made publicly available at https://github.com/kxfan2002/SophiaVL-R1.
paper_authors._id: 682fe2b865bac3ec3556c016
paper_authors.name: Kaixuan Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:24.518Z
paper_authors._id: 6656b9a6ca7f02c3d4817744
paper_authors.avatarUrl: /avatars/01d32967cad6054389c61b5bb9a528b1.svg
paper_authors.fullname: kxbunny
paper_authors.type: user
paper_authors.user: bunny127
paper_authors._id: 682fe2b865bac3ec3556c017
paper_authors.name: Kaituo Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T12:13:41.509Z
paper_authors._id: 67079840a9bcb7459b8d2a46
paper_authors.avatarUrl: /avatars/32466863c5554f20cb2775b138832ac3.svg
paper_authors.fullname: Kaituo Feng
paper_authors.type: user
paper_authors.user: KaituoFeng
paper_authors._id: 682fe2b865bac3ec3556c018
paper_authors.name: Haoming Lyu
paper_authors._id: 682fe2b865bac3ec3556c019
paper_authors.name: Dongzhan Zhou
paper_authors._id: 682fe2b865bac3ec3556c01a
paper_authors.name: Xiangyu Yue
paper_title: SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
paper_summary: Recent advances have shown success in eliciting strong reasoning abilities in
  multimodal large language models (MLLMs) through rule-based reinforcement
  learning (RL) with outcome rewards. However, this paradigm typically lacks
  supervision over the thinking process leading to the final outcome.As a result,
  the model may learn sub-optimal reasoning strategies, which can hinder its
  generalization ability. In light of this, we propose SophiaVL-R1, as an attempt
  to add reward signals for the thinking process in this paradigm. To achieve
  this, we first train a thinking reward model that evaluates the quality of the
  entire thinking process. Given that the thinking reward may be unreliable for
  certain samples due to reward hacking, we propose the Trust-GRPO method, which
  assigns a trustworthiness weight to the thinking reward during training. This
  weight is computed based on the thinking reward comparison of responses leading
  to correct answers versus incorrect answers, helping to mitigate the impact of
  potentially unreliable thinking rewards. Moreover, we design an annealing
  training strategy that gradually reduces the thinking reward over time,
  allowing the model to rely more on the accurate rule-based outcome reward in
  later training stages. Experiments show that our SophiaVL-R1 surpasses a series
  of reasoning MLLMs on various benchmarks (e.g., MathVisita, MMMU),
  demonstrating strong reasoning and generalization capabilities. Notably, our
  SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite
  the latter having 10 times more parameters. All code, models, and datasets are
  made publicly available at https://github.com/kxfan2002/SophiaVL-R1.
paper_ai_summary: An enhanced multimodal language model incorporates thinking process rewards to improve reasoning and generalization, achieving superior performance on benchmarks compared to larger models.
paper_ai_keywords: multimodal large language models rule-based reinforcement learning reward signals thinking reward model Trust-GRPO method thinking reward comparison annealing training strategy reasoning MLLMs MathVisita MMMU