title: Text2Grad: Reinforcement Learning from Natural Language Feedback
summary: Traditional RLHF optimizes language models with coarse, scalar rewards that
  mask the fine-grained reasons behind success or failure, leading to slow and
  opaque learning. Recent work augments RL with textual critiques through
  prompting or reflection, improving interpretability but leaving model
  parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm
  that turns free-form textual feedback into span-level gradients. Given human
  (or programmatic) critiques, Text2Grad aligns each feedback phrase with the
  relevant token spans, converts these alignments into differentiable reward
  signals, and performs gradient updates that directly refine the offending
  portions of the model's policy. This yields precise, feedback-conditioned
  adjustments instead of global nudges. Text2Grad is realized through three
  components: (1) a high-quality feedback-annotation pipeline that pairs
  critiques with token spans; (2) a fine-grained reward model that predicts
  span-level reward on answer while generating explanatory critiques; and (3) a
  span-level policy optimizer that back-propagates natural-language gradients.
  Across summarization, code generation, and question answering, Text2Grad
  consistently surpasses scalar-reward RL and prompt-only baselines, providing
  both higher task metrics and richer interpretability. Our results demonstrate
  that natural-language feedback, when converted to gradients, is a powerful
  signal for fine-grained policy optimization. The code for our method is
  available at https://github.com/microsoft/Text2Grad
paper_authors._id: 6837c79576eac3fa930de19b
paper_authors.name: Hanyang Wang
paper_authors._id: 6837c79576eac3fa930de19c
paper_authors.name: Lu Wang
paper_authors._id: 6837c79576eac3fa930de19d
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:42.800Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 6837c79576eac3fa930de19e
paper_authors.name: Tianjun Mao
paper_authors._id: 6837c79576eac3fa930de19f
paper_authors.name: Si Qin
paper_authors._id: 6837c79576eac3fa930de1a0
paper_authors.name: Qingwei Lin
paper_authors._id: 6837c79576eac3fa930de1a1
paper_authors.name: Saravan Rajmohan
paper_authors._id: 6837c79576eac3fa930de1a2
paper_authors.name: Dongmei Zhang
paper_title: Text2Grad: Reinforcement Learning from Natural Language Feedback
paper_summary: Traditional RLHF optimizes language models with coarse, scalar rewards that
  mask the fine-grained reasons behind success or failure, leading to slow and
  opaque learning. Recent work augments RL with textual critiques through
  prompting or reflection, improving interpretability but leaving model
  parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm
  that turns free-form textual feedback into span-level gradients. Given human
  (or programmatic) critiques, Text2Grad aligns each feedback phrase with the
  relevant token spans, converts these alignments into differentiable reward
  signals, and performs gradient updates that directly refine the offending
  portions of the model's policy. This yields precise, feedback-conditioned
  adjustments instead of global nudges. Text2Grad is realized through three
  components: (1) a high-quality feedback-annotation pipeline that pairs
  critiques with token spans; (2) a fine-grained reward model that predicts
  span-level reward on answer while generating explanatory critiques; and (3) a
  span-level policy optimizer that back-propagates natural-language gradients.
  Across summarization, code generation, and question answering, Text2Grad
  consistently surpasses scalar-reward RL and prompt-only baselines, providing
  both higher task metrics and richer interpretability. Our results demonstrate
  that natural-language feedback, when converted to gradients, is a powerful
  signal for fine-grained policy optimization. The code for our method is
  available at https://github.com/microsoft/Text2Grad
paper_ai_summary: Text2Grad converts human textual feedback into span-level gradients to optimize language models precisely and efficiently.
paper_ai_keywords: RLHF reinforcement-learning free-form textual feedback span-level gradients token spans differentiable reward signals gradient updates span-level policy optimizer fine-grained reward model feedback-annotation pipeline