title: GRIT: Teaching MLLMs to Think with Images
summary: Recent studies have demonstrated the efficacy of using Reinforcement Learning
  (RL) in building reasoning models that articulate chains of thoughts prior to
  producing final answers. However, despite ongoing advances that aim at enabling
  reasoning for vision-language tasks, existing open-source visual reasoning
  models typically generate reasoning content with pure natural language, lacking
  explicit integration of visual information. This limits their ability to
  produce clearly articulated and visually grounded reasoning chains. To this
  end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method
  for training MLLMs to think with images. GRIT introduces a grounded reasoning
  paradigm, in which models generate reasoning chains that interleave natural
  language and explicit bounding box coordinates. These coordinates point to
  regions of the input image that the model consults during its reasoning
  process. Additionally, GRIT is equipped with a reinforcement learning approach,
  GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused
  on the final answer accuracy and format of the grounded reasoning output, which
  eliminates the need for data with reasoning chain annotations or explicit
  bounding box labels. As a result, GRIT achieves exceptional data efficiency,
  requiring as few as 20 image-question-answer triplets from existing datasets.
  Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to
  produce coherent and visually grounded reasoning chains, showing a successful
  unification of reasoning and grounding abilities.
paper_authors._id: 682fecc3fd3719dbe6fbb84b
paper_authors.name: Yue Fan
paper_authors._id: 682fecc3fd3719dbe6fbb84c
paper_authors.name: Xuehai He
paper_authors._id: 682fecc3fd3719dbe6fbb84d
paper_authors.name: Diji Yang
paper_authors._id: 682fecc3fd3719dbe6fbb84e
paper_authors.name: Kaizhi Zheng
paper_authors._id: 682fecc3fd3719dbe6fbb84f
paper_authors.name: Ching-Chen Kuo
paper_authors._id: 682fecc3fd3719dbe6fbb850
paper_authors.name: Yuting Zheng
paper_authors._id: 682fecc3fd3719dbe6fbb851
paper_authors.name: Sravana Jyothi Narayanaraju
paper_authors._id: 682fecc3fd3719dbe6fbb852
paper_authors.name: Xinze Guan
paper_authors._id: 682fecc3fd3719dbe6fbb853
paper_authors.name: Xin Eric Wang
paper_title: GRIT: Teaching MLLMs to Think with Images
paper_summary: Recent studies have demonstrated the efficacy of using Reinforcement Learning
  (RL) in building reasoning models that articulate chains of thoughts prior to
  producing final answers. However, despite ongoing advances that aim at enabling
  reasoning for vision-language tasks, existing open-source visual reasoning
  models typically generate reasoning content with pure natural language, lacking
  explicit integration of visual information. This limits their ability to
  produce clearly articulated and visually grounded reasoning chains. To this
  end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method
  for training MLLMs to think with images. GRIT introduces a grounded reasoning
  paradigm, in which models generate reasoning chains that interleave natural
  language and explicit bounding box coordinates. These coordinates point to
  regions of the input image that the model consults during its reasoning
  process. Additionally, GRIT is equipped with a reinforcement learning approach,
  GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused
  on the final answer accuracy and format of the grounded reasoning output, which
  eliminates the need for data with reasoning chain annotations or explicit
  bounding box labels. As a result, GRIT achieves exceptional data efficiency,
  requiring as few as 20 image-question-answer triplets from existing datasets.
  Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to
  produce coherent and visually grounded reasoning chains, showing a successful
  unification of reasoning and grounding abilities.
paper_ai_summary: A novel method called GRIT enhances visual reasoning in MLLMs by generating reasoning chains that integrate both natural language and bounding box coordinates, guided by a reinforcement learning approach for high data efficiency.
paper_ai_keywords: Reinforcement Learning RL MLLMs reasoning chains interleave natural language bounding box coordinates GRPO-GR GRPO grounded reasoning output reasoning and grounding abilities