Averaging $n$-step Returns Reduces Variance in Reinforcement Learning
<|reference_start|>Averaging $n$-step Returns Reduces Variance in Reinforcement Learning: Multistep returns, such as $n$-step returns and $\lambda$-returns, are commonly used to improve the sample efficiency of reinforcement learning (RL) methods. The variance of the multistep returns becomes the limiting factor in their length; looking too far into the future increases variance and reverses the benefits of multistep learning. In our work, we demonstrate the ability of compound returns -- weighted averages of $n$-step returns -- to reduce variance. We prove for the first time that any compound return with the same contraction modulus as a given $n$-step return has strictly lower variance. We additionally prove that this variance-reduction property improves the finite-sample complexity of temporal-difference learning under linear function approximation. Because general compound returns can be expensive to implement, we introduce two-bootstrap returns which reduce variance while remaining efficient, even when using minibatched experience replay. We conduct experiments showing that compound returns often increase the sample efficiency of $n$-step deep RL agents like DQN and PPO.<|reference_end|>
@article{daley2024averaging,
    title={Averaging $n$-step Returns Reduces Variance in Reinforcement Learning},
    author={Brett Daley, Martha White, Marlos C. Machado},
    journal={arXiv preprint arXiv:2402.03903},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03903},
    primaryClass={cs.LG}
}