Text2Grad: Reinforcement Learning from Natural Language Feedback
Traditional RLHF optimizes language models with coarse, scalar rewards that
mask the fine-grained reasons behind success or failure, leading to slow and
opaque learning. Recent work augments RL with textual critiques through
prompting or reflection, improving interpretability but leaving model
parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm
that turns free-form textual feedback into span-level gradients. Given human
(or programmatic) critiques, Text2Grad aligns each feedback phrase with the
relevant token spans, converts these alignments into differentiable reward
signals, and performs gradient updates that directly refine the offending
portions of the model's policy. This yields precise, feedback-conditioned
adjustments instead of global nudges. Text2Grad is realized through three
components: (1) a high-quality feedback-annotation pipeline that pairs
critiques with token spans; (2) a fine-grained reward model that predicts
span-level reward on answer while generating explanatory critiques; and (3) a
span-level policy optimizer that back-propagates natural-language gradients.
Across summarization, code generation, and question answering, Text2Grad
consistently surpasses scalar-reward RL and prompt-only baselines, providing
both higher task metrics and richer interpretability. Our results demonstrate
that natural-language feedback, when converted to gradients, is a powerful
signal for fine-grained policy optimization. The code for our method is
available at https://github.com/microsoft/Text2Grad
6837c79576eac3fa930de19b
Hanyang Wang
6837c79576eac3fa930de19c
Lu Wang
6837c79576eac3fa930de19d
Chaoyun Zhang
claimed_verified
2025-06-04T09:03:42.800Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
6837c79576eac3fa930de19e
Tianjun Mao
6837c79576eac3fa930de19f
Si Qin
6837c79576eac3fa930de1a0
Qingwei Lin
6837c79576eac3fa930de1a1
Saravan Rajmohan
6837c79576eac3fa930de1a2
Dongmei Zhang
Text2Grad: Reinforcement Learning from Natural Language Feedback
Traditional RLHF optimizes language models with coarse, scalar rewards that
mask the fine-grained reasons behind success or failure, leading to slow and
opaque learning. Recent work augments RL with textual critiques through
prompting or reflection, improving interpretability but leaving model
parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm
that turns free-form textual feedback into span-level gradients. Given human
(or programmatic) critiques, Text2Grad aligns each feedback phrase with the
relevant token spans, converts these alignments into differentiable reward
signals, and performs gradient updates that directly refine the offending
portions of the model's policy. This yields precise, feedback-conditioned
adjustments instead of global nudges. Text2Grad is realized through three
components: (1) a high-quality feedback-annotation pipeline that pairs
critiques with token spans; (2) a fine-grained reward model that predicts
span-level reward on answer while generating explanatory critiques; and (3) a
span-level policy optimizer that back-propagates natural-language gradients.
Across summarization, code generation, and question answering, Text2Grad
consistently surpasses scalar-reward RL and prompt-only baselines, providing
both higher task metrics and richer interpretability. Our results demonstrate
that natural-language feedback, when converted to gradients, is a powerful
signal for fine-grained policy optimization. The code for our method is
available at https://github.com/microsoft/Text2Grad
Text2Grad converts human textual feedback into span-level gradients to optimize language models precisely and efficiently.
RLHF reinforcement-learning free-form textual feedback span-level gradients token spans differentiable reward signals gradient updates span-level policy optimizer fine-grained reward model feedback-annotation pipeline