GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal
  Reasoning
Recent reinforcement learning approaches, such as outcome-supervised GRPO,
have advanced Chain-of-Thought reasoning in large language models (LLMs), yet
their adaptation to multimodal LLMs (MLLMs) is unexplored. To address the lack
of rigorous evaluation for MLLM post-training methods, we introduce
SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced
perception and reasoning. It offers a large training set and evaluates
generalization across three escalating challenges: in-distribution,
cross-environment, and cross-environment-task scenarios. Using SEED-Bench-R1,
we find that standard GRPO, while improving answer accuracy, often reduces
logical coherence between reasoning steps and answers, with only a 57.9%
consistency rate. This stems from reward signals focusing solely on final
answers, encouraging shortcuts, and strict KL penalties limiting exploration.To
address this, we propose GRPO-CARE, a consistency-aware RL framework optimizing
both answer correctness and reasoning coherence without explicit supervision.
GRPO-CARE introduces a two-tiered reward: (1) a base reward for answer
correctness, and (2) an adaptive consistency bonus, computed by comparing the
model's reasoning-to-answer likelihood (via a slowly-evolving reference model)
against group peers.This dual mechanism amplifies rewards for reasoning paths
that are both correct and logically consistent. Replacing KL penalties with
this adaptive bonus, GRPO-CARE outperforms standard GRPO on SEED-Bench-R1,
achieving a 6.7% performance gain on the hardest evaluation level and a 24.5%
improvement in consistency. It also shows strong transferability, improving
model performance across diverse video understanding benchmarks. Our work
contributes a systematically designed benchmark and a generalizable
post-training framework, advancing the development of more interpretable and
robust MLLMs.
6858b1fac0c8e29df8ea3c18
Yi Chen
6858b1fac0c8e29df8ea3c19
Yuying Ge
admin_assigned
2025-06-25T13:34:16.831Z
6455cc8f654d8bccae50e4d4
/avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
Yuying Ge
user
tttoaster
6858b1fac0c8e29df8ea3c1a
Rui Wang
6858b1fac0c8e29df8ea3c1b
Yixiao Ge
admin_assigned
2025-06-25T13:34:24.360Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
6858b1fac0c8e29df8ea3c1c
Junhao Cheng
admin_assigned
2025-06-25T13:34:30.450Z
6496f2bb33611fb4330455bd
/avatars/1bc94098010145d302c7bea3c221b39a.svg
Junhao Cheng
user
RefineMe
6858b1fac0c8e29df8ea3c1d
Ying Shan
6858b1fac0c8e29df8ea3c1e
Xihui Liu
GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal
  Reasoning
Recent reinforcement learning approaches, such as outcome-supervised GRPO,
have advanced Chain-of-Thought reasoning in large language models (LLMs), yet
their adaptation to multimodal LLMs (MLLMs) is unexplored. To address the lack
of rigorous evaluation for MLLM post-training methods, we introduce
SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced
perception and reasoning. It offers a large training set and evaluates
generalization across three escalating challenges: in-distribution,
cross-environment, and cross-environment-task scenarios. Using SEED-Bench-R1,
we find that standard GRPO, while improving answer accuracy, often reduces
logical coherence between reasoning steps and answers, with only a 57.9%
consistency rate. This stems from reward signals focusing solely on final
answers, encouraging shortcuts, and strict KL penalties limiting exploration.To
address this, we propose GRPO-CARE, a consistency-aware RL framework optimizing
both answer correctness and reasoning coherence without explicit supervision.
GRPO-CARE introduces a two-tiered reward: (1) a base reward for answer
correctness, and (2) an adaptive consistency bonus, computed by comparing the
model's reasoning-to-answer likelihood (via a slowly-evolving reference model)
against group peers.This dual mechanism amplifies rewards for reasoning paths
that are both correct and logically consistent. Replacing KL penalties with
this adaptive bonus, GRPO-CARE outperforms standard GRPO on SEED-Bench-R1,
achieving a 6.7% performance gain on the hardest evaluation level and a 24.5%
improvement in consistency. It also shows strong transferability, improving
model performance across diverse video understanding benchmarks. Our work
contributes a systematically designed benchmark and a generalizable
post-training framework, advancing the development of more interpretable and
robust MLLMs.
GRPO-CARE, a reinforcement learning framework optimizing for consistency and correctness, outperforms standard GRPO on a new video understanding benchmark, SEED-Bench-R1, improving both performance and logical coherence in multimodal large language models.
reinforcement learning outcome-supervised GRPO Chain-of-Thought reasoning large language models multimodal large language models SEED-Bench-R1 in-distribution cross-environment cross-environment-task logical coherence reasoning steps answer accuracy reward signals shortcuts KL penalties exploration consistency-aware RL framework two-tiered reward reasoning-to-answer likelihood adaptive consistency bonus video understanding benchmarks transferability interpretable models robust models