ABSTRACT Q-learning suffers from overestimation bias, because it approximates the maximum action value using the maximum estimated action value. Algorithms have been proposed to reduce overestimation bias, but we lack an understanding of how bias interacts with performance, and the extent to which existing algorithms mitigate bias. In this paper, we 1) highlight that the effect of overestimation bias on learning efficiency is environment-dependent; 2) propose a generalization of Qlearning, called Maxmin Q-learning, which provides a parameter to flexibly control bias; 3) show theoretically that there exists a parameter choice for Maxmin Q-learning that leads to unbiased estimation with a lower approximation variance than Q-learning; and 4) prove the convergence of our algorithm in the tabular case, as well as convergence of several previous Q-learning variants, using a novel Generalized Q-learning framework. We empirically verify that our algorithm better controls estimation bias in toy environments, and that it achieves superior performance on several benchmark problems.
This paper proposes a new Q learning algorithm framework: maxmin Q-learning, to address the overestimation bias issue of Q learning. The main contributions of this paper are three folds: 1) It provides an inspiring example on overestimation/underestimation of Q learning. 2) Generalize Q learning by a new maxmin Q-learning by maintaining independent Q estimator and interact them in a max-min way for the update. 3) Provide both theoretical and empirical analyses of their algorithm.
I have two main concerns for this paper:
1) When is your algorithm useful? What's your criterion of picking the hyper-parameters (e.g. number of Q functions you want to learn).
2) Comparison to more intriguing way for jointly update of multiple Q functions, like soft Q learning.
For the first concern, the paper has shown an interesting example in Figure 2. However, it seems that we cannot decide whether overestimation or underestimation will help the exploration, since the reward function is often unknown in real world. And in both cases, maxmin Q learning is not the best algorithm than either Q learning and double q learning. On the other hand, if we use a softmax policy for Q function, e.g. $\pi(a|s) \propto \exp(\alpha Q(s,a))$, a drift for Q learning(e.g. Q(s,a) = Q*(s,a) + c) has no effect on our policy. I believe in this case we should more focusing on the inner difference between different value of Q function, rather than comparing our estimate Q function with the true Q*.
For the second concern, we can view the framework of maxmin q learning as a joint update scheme for different Q function. In experimental part, the comparison is not fair since the paper use multiple Q function to compare with single or double Q function. One reasonable baseline is to update N different Q function, and take the minimum of the final Q function as our decision policy, compare with maxmin Q learning with N different Q function. Another baseline the paper should consider is soft Q learning, where it maintain multiple Q function and jointly update Q different function to maximize the entropy while moving towards an improvement Q.
Overall, I believe the idea of the paper is novel and interesting, but further improvements should be added in order to improve the score the paper.