Q-learning with Nearest Neighbors
We consider model-free reinforcement learning for infinite-horizon discounted Markov Decision Processes (MDPs) with a continuous state space and unknown transition kernel, when only a single sample path under an arbitrary policy of the system is available. We consider the Nearest Neighbor Q-Learning (NNQL) algorithm to learn the optimal Q function using nearest neighbor regression method. As the main contribution, we provide tight finite sample analysis of the convergence rate. In particular, for MDPs with a d-dimensional state space and the discounted factor 2 (0, 1), given an arbitrary sample path with "covering time" L, we establish that the algorithm is guaranteed to output an "-accurate estimate of the optimal Q-function using e O L/(" 3 (1 ) 7 ) samples. For instance, for a wellbehaved MDP, the covering time of the sample path under the purely random policy scales as e O 1/" d , so the sample complexity scales as e O 1/" d+3 . Indeed, we establish a lower bound that argues that the dependence of e ⌦ 1/" d+2 is necessary.
This work analyzes the sample complexity of a nearest-neighbor Q-learning algorithm. It tackles the case of continuous state space and discrete action space, where discretization of the state space is applied. The paper is well written and relatively easy to follow. The theory and definitions are sound, and the proofs seem to be valid (though I did not verify all of the appendix). I also appreciate the elegancy of the NN Bellman operator in (5). Nonetheless, the work suffers from several weaknesses. Given bellow is a list of remarks regarding these weaknesses and requests for clarifications and updates to the manuscript.  - The algorithm’s O(1/(\esiplon^3 (1-\gamma)^7)) complexity is extremely high. Of course, this is not practical. Notice that as opposed to the nice recovery time O(\epsilon^{-(d+3)}) result, which is almost tight, the above complexity stems from the algorithm’s design. - Part of the intractability of the algorithm comes from the requirement of full coverage of all ball-action pairs, per each iteration. This issue is magnified by the fact that the NN effective distance, h^*, is O(\epsilon (1-\gamma)). This implies a huge discretized state set, which adds up to the above problematic complexity.  The authors mention (though vaguely) that the analysis is probably loose. I wonder how much of the complexity issues originate from the analysis itself, and how much from the algorithm’s design.  - In continuation to the above remark, what do you think can be done (i.e. what minimal assumptions are needed) to relax the need of visiting all ball-action pairs with each iteration? Alternatively, what would happen if you partially cover them? - Table 1 lacks two recent works [1,2] (see below) that analyze the sample complexity of parametrized TD-learning algorithms that has all ‘yes’ values in the columns except for the ‘single sample path’ column. Please update accordingly.  - From personal experience, I believe the Lipschitz assumption is crucial to have any guarantees. Also, this is a non-trivial assumption. Please stress it further in the introduction, and/or perhaps in the abstract itself. - There is another work [3] that should also definitely be cited. Please also explain how your work differs from it. - It is written in the abstract and in at least one more location that your O(\epsilon^{-(d+3)}) complexity is tight, but you mention a lower bound which differs by a factor of \epsilon^{-1}. So this is not really tight, right? If so, please rephrase.  - p.5, l.181: ‘of’ is written twice. p.7, l.267: ‘the’ is written twice.  References: [1] Finite Sample Analyses for TD (0) with Function Approximation, G Dalal, B Szörényi, G Thoppe, S Mannor, AAAI 2018 [2] Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning G Dalal, B Szörényi, G Thoppe, S Mannor, COLT 2018 [3] Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories, Raphael Fonteneau, Susan A. Murphy, Louis Wehenkel, and  Damien Ernst, Annals of Operations Research 2013