Regularized Q-learning through Robust Averaging
<|reference_start|>Regularized Q-learning through Robust Averaging: We propose a new Q-learning variant, called 2RA Q-learning, that addresses some weaknesses of existing Q-learning methods in a principled manner. One such weakness is an underlying estimation bias which cannot be controlled and often results in poor performance. We propose a distributionally robust estimator for the maximum expected value term, which allows us to precisely control the level of estimation bias introduced. The distributionally robust estimator admits a closed-form solution such that the proposed algorithm has a computational cost per iteration comparable to Watkins' Q-learning. For the tabular case, we show that 2RA Q-learning converges to the optimal policy and analyze its asymptotic mean-squared error. Lastly, we conduct numerical experiments for various settings, which corroborate our theoretical findings and indicate that 2RA Q-learning often performs better than existing methods.<|reference_end|>
@article{schmitt-förster2024regularized,
    title={Regularized Q-learning through Robust Averaging},
    author={Peter Schmitt-F"orster and Tobias Sutter},
    journal={arXiv preprint arXiv:2405.02201},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.02201},
    primaryClass={math.OC cs.LG}
}