Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning
<|reference_start|>Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning: Q-learning is a widely used algorithm in reinforcement learning (RL), but its convergence can be slow, especially when the discount factor is close to one. Successive Over-Relaxation (SOR) Q-learning, which introduces a relaxation factor to speed up convergence, addresses this issue but has two major limitations: In the tabular setting, the relaxation parameter depends on transition probability, making it not entirely model-free, and it suffers from overestimation bias. To overcome these limitations, we propose a sample-based, model-free double SOR Q-learning algorithm. Theoretically and empirically, this algorithm is shown to be less biased than SOR Q-learning. Further, in the tabular setting, the convergence analysis under boundedness assumptions on iterates is discussed. The proposed algorithm is extended to large-scale problems using deep RL. Finally, the tabular version of the proposed algorithm is compared using roulette and grid world environments, while the deep RL version is tested on a maximization bias example and OpenAI Gym environments.<|reference_end|>
@article{r2024double,
    title={Double Successive Over-Relaxation Q-Learning with an Extension to Deep
  Reinforcement Learning},
    author={Shreyas S R},
    journal={arXiv preprint arXiv:2409.06356},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.06356},
    primaryClass={cs.LG cs.AI}
}