GRIT: Teaching MLLMs to Think with Images
Recent studies have demonstrated the efficacy of using Reinforcement Learning
(RL) in building reasoning models that articulate chains of thoughts prior to
producing final answers. However, despite ongoing advances that aim at enabling
reasoning for vision-language tasks, existing open-source visual reasoning
models typically generate reasoning content with pure natural language, lacking
explicit integration of visual information. This limits their ability to
produce clearly articulated and visually grounded reasoning chains. To this
end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method
for training MLLMs to think with images. GRIT introduces a grounded reasoning
paradigm, in which models generate reasoning chains that interleave natural
language and explicit bounding box coordinates. These coordinates point to
regions of the input image that the model consults during its reasoning
process. Additionally, GRIT is equipped with a reinforcement learning approach,
GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused
on the final answer accuracy and format of the grounded reasoning output, which
eliminates the need for data with reasoning chain annotations or explicit
bounding box labels. As a result, GRIT achieves exceptional data efficiency,
requiring as few as 20 image-question-answer triplets from existing datasets.
Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to
produce coherent and visually grounded reasoning chains, showing a successful
unification of reasoning and grounding abilities.
682fecc3fd3719dbe6fbb84b
Yue Fan
682fecc3fd3719dbe6fbb84c
Xuehai He
682fecc3fd3719dbe6fbb84d
Diji Yang
claimed_verified
2025-06-11T08:37:20.481Z
62a17ca6d57bf9ab9c697206
/avatars/40cca602d481382b0adb13996328b515.svg
DY
user
dyang39
682fecc3fd3719dbe6fbb84e
Kaizhi Zheng
682fecc3fd3719dbe6fbb84f
Ching-Chen Kuo
682fecc3fd3719dbe6fbb850
Yuting Zheng
682fecc3fd3719dbe6fbb851
Sravana Jyothi Narayanaraju
682fecc3fd3719dbe6fbb852
Xinze Guan
682fecc3fd3719dbe6fbb853
Xin Eric Wang
claimed_verified
2025-07-24T09:15:03.530Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
GRIT: Teaching MLLMs to Think with Images
Recent studies have demonstrated the efficacy of using Reinforcement Learning
(RL) in building reasoning models that articulate chains of thoughts prior to
producing final answers. However, despite ongoing advances that aim at enabling
reasoning for vision-language tasks, existing open-source visual reasoning
models typically generate reasoning content with pure natural language, lacking
explicit integration of visual information. This limits their ability to
produce clearly articulated and visually grounded reasoning chains. To this
end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method
for training MLLMs to think with images. GRIT introduces a grounded reasoning
paradigm, in which models generate reasoning chains that interleave natural
language and explicit bounding box coordinates. These coordinates point to
regions of the input image that the model consults during its reasoning
process. Additionally, GRIT is equipped with a reinforcement learning approach,
GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused
on the final answer accuracy and format of the grounded reasoning output, which
eliminates the need for data with reasoning chain annotations or explicit
bounding box labels. As a result, GRIT achieves exceptional data efficiency,
requiring as few as 20 image-question-answer triplets from existing datasets.
Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to
produce coherent and visually grounded reasoning chains, showing a successful
unification of reasoning and grounding abilities.
A novel method called GRIT enhances visual reasoning in MLLMs by generating reasoning chains that integrate both natural language and bounding box coordinates, guided by a reinforcement learning approach for high data efficiency.
Reinforcement Learning RL MLLMs reasoning chains interleave natural language bounding box coordinates GRPO-GR GRPO grounded reasoning output reasoning and grounding abilities