title: ARIA: Training Language Agents with Intention-Driven Reward Aggregation
summary: Large language models (LLMs) have enabled agents to perform complex reasoning
  and decision-making through free-form language interactions. However, in
  open-ended language action environments (e.g., negotiation or question-asking
  games), the action space can be formulated as a joint distribution over tokens,
  resulting in an exponentially large action space. Sampling actions in such a
  space can lead to extreme reward sparsity, which brings large reward variance,
  hindering effective reinforcement learning (RL). To address this, we propose
  ARIA, a method that Aggregates Rewards in Intention space to enable efficient
  and effective language Agents training. ARIA aims to project natural language
  actions from the high-dimensional joint token distribution space into a
  low-dimensional intention space, where semantically similar actions are
  clustered and assigned shared rewards. This intention-aware reward aggregation
  reduces reward variance by densifying reward signals, fostering better policy
  optimization. Extensive experiments demonstrate that ARIA not only
  significantly reduces policy gradient variance, but also delivers substantial
  performance gains of an average of 9.95% across four downstream tasks,
  consistently outperforming offline and online RL baselines.
paper_authors._id: 683e76c17a0996f979e72700
paper_authors.name: Ruihan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:41:32.053Z
paper_authors._id: 671a4abbef737c0abe21b3f8
paper_authors.avatarUrl: /avatars/da826af5472a3b9f1969f0c766672731.svg
paper_authors.fullname: Ruihan Yang
paper_authors.type: user
paper_authors.user: rhyang2021
paper_authors._id: 683e76c17a0996f979e72701
paper_authors.name: Yikai Zhang
paper_authors._id: 683e76c17a0996f979e72702
paper_authors.name: Aili Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:41:28.422Z
paper_authors._id: 63f86b099f87cc3e645b51d9
paper_authors.avatarUrl: /avatars/27ca5ba425640bf67474cee871e8e53a.svg
paper_authors.fullname: Ellie Chen
paper_authors.type: user
paper_authors.user: sheep33333
paper_authors._id: 683e76c17a0996f979e72703
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T10:44:59.705Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 683e76c17a0996f979e72704
paper_authors.name: Siyu Yuan
paper_authors._id: 683e76c17a0996f979e72705
paper_authors.name: Jiangjie Chen
paper_authors._id: 683e76c17a0996f979e72706
paper_authors.name: Deqing Yang
paper_authors._id: 683e76c17a0996f979e72707
paper_authors.name: Yanghua Xiao
paper_title: ARIA: Training Language Agents with Intention-Driven Reward Aggregation
paper_summary: Large language models (LLMs) have enabled agents to perform complex reasoning
  and decision-making through free-form language interactions. However, in
  open-ended language action environments (e.g., negotiation or question-asking
  games), the action space can be formulated as a joint distribution over tokens,
  resulting in an exponentially large action space. Sampling actions in such a
  space can lead to extreme reward sparsity, which brings large reward variance,
  hindering effective reinforcement learning (RL). To address this, we propose
  ARIA, a method that Aggregates Rewards in Intention space to enable efficient
  and effective language Agents training. ARIA aims to project natural language
  actions from the high-dimensional joint token distribution space into a
  low-dimensional intention space, where semantically similar actions are
  clustered and assigned shared rewards. This intention-aware reward aggregation
  reduces reward variance by densifying reward signals, fostering better policy
  optimization. Extensive experiments demonstrate that ARIA not only
  significantly reduces policy gradient variance, but also delivers substantial
  performance gains of an average of 9.95% across four downstream tasks,
  consistently outperforming offline and online RL baselines.
paper_ai_summary: ARIA aggregates rewards in an intention space to mitigate reward sparsity and improve policy optimization in language-based reinforcement learning tasks.
paper_ai_keywords: large language models reinforcement learning action space token distribution extreme reward sparsity reward variance policy optimization intention space semantically similar actions shared rewards policy gradient variance performance gains offline RL online RL