Finite-Time Analysis of Simultaneous Double Q-learning
<|reference_start|>Finite-Time Analysis of Simultaneous Double Q-learning: $Q$-learning is one of the most fundamental reinforcement learning (RL) algorithms. Despite its widespread success in various applications, it is prone to overestimation bias in the $Q$-learning update. To address this issue, double $Q$-learning employs two independent $Q$-estimators which are randomly selected and updated during the learning process. This paper proposes a modified double $Q$-learning, called simultaneous double $Q$-learning (SDQ), with its finite-time analysis. SDQ eliminates the need for random selection between the two $Q$-estimators, and this modification allows us to analyze double $Q$-learning through the lens of a novel switching system framework facilitating efficient finite-time analysis. Empirical studies demonstrate that SDQ converges faster than double $Q$-learning while retaining the ability to mitigate the maximization bias. Finally, we derive a finite-time expected error bound for SDQ.<|reference_end|>
@article{na2024finite-time,
    title={Finite-Time Analysis of Simultaneous Double Q-learning},
    author={Hyunjun Na, Donghwan Lee},
    journal={arXiv preprint arXiv:2406.09946},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09946},
    primaryClass={cs.LG cs.SY eess.SY}
}