ARIA: Training Language Agents with Intention-Driven Reward Aggregation
Large language models (LLMs) have enabled agents to perform complex reasoning
and decision-making through free-form language interactions. However, in
open-ended language action environments (e.g., negotiation or question-asking
games), the action space can be formulated as a joint distribution over tokens,
resulting in an exponentially large action space. Sampling actions in such a
space can lead to extreme reward sparsity, which brings large reward variance,
hindering effective reinforcement learning (RL). To address this, we propose
ARIA, a method that Aggregates Rewards in Intention space to enable efficient
and effective language Agents training. ARIA aims to project natural language
actions from the high-dimensional joint token distribution space into a
low-dimensional intention space, where semantically similar actions are
clustered and assigned shared rewards. This intention-aware reward aggregation
reduces reward variance by densifying reward signals, fostering better policy
optimization. Extensive experiments demonstrate that ARIA not only
significantly reduces policy gradient variance, but also delivers substantial
performance gains of an average of 9.95% across four downstream tasks,
consistently outperforming offline and online RL baselines.
683e76c17a0996f979e72700
Ruihan Yang
claimed_verified
2025-06-03T08:41:32.053Z
671a4abbef737c0abe21b3f8
/avatars/da826af5472a3b9f1969f0c766672731.svg
Ruihan Yang
user
rhyang2021
683e76c17a0996f979e72701
Yikai Zhang
683e76c17a0996f979e72702
Aili Chen
claimed_verified
2025-06-03T08:41:28.422Z
63f86b099f87cc3e645b51d9
/avatars/27ca5ba425640bf67474cee871e8e53a.svg
Ellie Chen
user
sheep33333
683e76c17a0996f979e72703
Xintao Wang
admin_assigned
2025-06-03T10:44:59.705Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
683e76c17a0996f979e72704
Siyu Yuan
683e76c17a0996f979e72705
Jiangjie Chen
683e76c17a0996f979e72706
Deqing Yang
683e76c17a0996f979e72707
Yanghua Xiao
ARIA: Training Language Agents with Intention-Driven Reward Aggregation
Large language models (LLMs) have enabled agents to perform complex reasoning
and decision-making through free-form language interactions. However, in
open-ended language action environments (e.g., negotiation or question-asking
games), the action space can be formulated as a joint distribution over tokens,
resulting in an exponentially large action space. Sampling actions in such a
space can lead to extreme reward sparsity, which brings large reward variance,
hindering effective reinforcement learning (RL). To address this, we propose
ARIA, a method that Aggregates Rewards in Intention space to enable efficient
and effective language Agents training. ARIA aims to project natural language
actions from the high-dimensional joint token distribution space into a
low-dimensional intention space, where semantically similar actions are
clustered and assigned shared rewards. This intention-aware reward aggregation
reduces reward variance by densifying reward signals, fostering better policy
optimization. Extensive experiments demonstrate that ARIA not only
significantly reduces policy gradient variance, but also delivers substantial
performance gains of an average of 9.95% across four downstream tasks,
consistently outperforming offline and online RL baselines.
ARIA aggregates rewards in an intention space to mitigate reward sparsity and improve policy optimization in language-based reinforcement learning tasks.
large language models reinforcement learning action space token distribution extreme reward sparsity reward variance policy optimization intention space semantically similar actions shared rewards policy gradient variance performance gains offline RL online RL