Unified ODE Analysis of Smooth Q-Learning Algorithms
<|reference_start|>Unified ODE Analysis of Smooth Q-Learning Algorithms: Convergence of Q-learning has been the focus of extensive research over the past several decades. Recently, an asymptotic convergence analysis for Q-learning was introduced using a switching system framework. This approach applies the so-called ordinary differential equation (ODE) approach to prove the convergence of the asynchronous Q-learning modeled as a continuous-time switching system, where notions from switching system theory are used to prove its asymptotic stability without using explicit Lyapunov arguments. However, to prove stability, restrictive conditions, such as quasi-monotonicity, must be satisfied for the underlying switching systems, which makes it hard to easily generalize the analysis method to other reinforcement learning algorithms, such as the smooth Q-learning variants. In this paper, we present a more general and unified convergence analysis that improves upon the switching system approach and can analyze Q-learning and its smooth variants. The proposed analysis is motivated by previous work on the convergence of synchronous Q-learning based on $p$-norm serving as a Lyapunov function. However, the proposed analysis addresses more general ODE models that can cover both asynchronous Q-learning and its smooth versions with simpler frameworks.<|reference_end|>
@article{lee2024unified,
    title={Unified ODE Analysis of Smooth Q-Learning Algorithms},
    author={Donghwan Lee},
    journal={arXiv preprint arXiv:2404.14442},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.14442},
    primaryClass={cs.LG cs.AI}
}