A Tighter Convergence Proof of Reverse Experience Replay
<|reference_start|>A Tighter Convergence Proof of Reverse Experience Replay: In reinforcement learning, Reverse Experience Replay (RER) is a recently proposed algorithm that attains better sample complexity than the classic experience replay method. RER requires the learning algorithm to update the parameters through consecutive state-action-reward tuples in reverse order. However, the most recent theoretical analysis only holds for a minimal learning rate and short consecutive steps, which converge slower than those large learning rate algorithms without RER. In view of this theoretical and empirical gap, we provide a tighter analysis that mitigates the limitation on the learning rate and the length of consecutive steps. Furthermore, we show theoretically that RER converges with a larger learning rate and a longer sequence.<|reference_end|>
@article{jiang2024a,
    title={A Tighter Convergence Proof of Reverse Experience Replay},
    author={Nan Jiang, Jinzhao Li, Yexiang Xue},
    journal={arXiv preprint arXiv:2408.16999},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16999},
    primaryClass={cs.LG stat.ML}
}