Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
<|reference_start|>Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement: Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. Traditional approaches often depend on meticulously designed prompts, high-quality examples, or additional reward models for in-context learning, supervised fine-tuning, or RLHF. Reinforcement learning (RL) presents a dynamic alternative for LLMs to overcome these dependencies by engaging directly with task-specific environments. Nonetheless, it faces significant hurdles: 1) instability stemming from the exponentially vast action space requiring exploration; 2) challenges in assigning token-level credit based on action-level reward signals, resulting in discord between maximizing rewards and accurately modeling corpus data. In response to these challenges, we introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. At the heart of ETPO is our novel per-token soft Bellman update, designed to harmonize the RL process with the principles of language modeling. This methodology decomposes the Q-function update from a coarse action-level view to a more granular token-level perspective, backed by theoretical proof of optimization consistency. Crucially, this decomposition renders linear time complexity in action exploration. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks; results underline ETPO's potential as a robust method for refining the interactive decision-making capabilities of language agents. For a more detailed preliminary work describing our motivation for token-level decomposition and applying it in PPO methods, please refer to arXiv:2405.15821.<|reference_end|>
@article{wen2024entropy-regularized,
    title={Entropy-Regularized Token-Level Policy Optimization for Language Agent
  Reinforcement},
    author={Muning Wen, Junwei Liao, Cheng Deng, Jun Wang, Weinan Zhang and Ying
  Wen},
    journal={arXiv preprint arXiv:2402.06700},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.06700},
    primaryClass={cs.LG cs.AI}
}