Differently from the popular Deep Q-Network (DQN) learning, Alternating Qlearning (AltQ) does not fully fit a target Q-function at each iteration, and is generally known to be unstable and inefficient. Limited applications of AltQ mostly rely on substantially altering the algorithm architecture in order to improve its performance. Although Adam appears to be a natural solution, its performance in AltQ has rarely been studied before. In this paper, we first provide a solid exploration on how well AltQ performs with Adam. We then take a further step to improve the implementation by adopting the technique of parameter restart. More specifically, the proposed algorithms are tested on a batch of Atari 2600 games and exhibit superior performance than the DQN learning method. The convergence rate of the slightly modified version of the proposed algorithms is characterized under the linear function approximation. To the best of our knowledge, this is the first theoretical study on the Adam-type algorithms in Q-learning. Q-learning (Watkins & Dayan, 1992) is one of the most important model-free reinforcement learning (RL) problems, which has received considerable research attention in recent years (Bertsekas & Tsitsiklis, 1996;Even-Dar & Mansour, 2003;Hasselt, 2010; Lu et al., 2018;Achiam et al., 2019). When the state-action space is large or continuous, parametric approximation of the Q-function is often necessary. One remarkable success of parametric Q-learning in practice is its combination with deep learning, known as the Deep Q-Network (DQN) learning (Mnih et al., 2013;2015). It has been applied to various applications in computer games (Bhatti et al., 2016), traffic control (Arel et al., 2010), recommendation systems (Zheng et al., 2018;Zhao et al., 2018), chemistry research (Zhou et al., 2017), etc. Its on-policy continuous variant (Silver et al., 2014)  has also led to great achievements in robotics locomotion (Lillicrap et al., 2016).
This paper is well-written and it provides a convergence result for traditional Q-learning, with linear function approximation, when using an Adam-like update (AMSGrad). It does the same for a variation of this algorithm where the momentum-like term is reset every now and then. This second result is not that exciting as it ends up concluding that the “best way” to converge with such an approach is by resetting the momentum-like term rarely. That being said, it is still interesting to have such theoretical result. On the empirical side, this paper evaluates the traditional Q-learning algorithm with non-linear function approximation (through a neural network), using Adam (and AdamR) while not using a target network, in both an LQR problem and a subset of the Atari games. The empirical results are not necessarily that convincing and there are important details missing. I’m willing to increase my score if my concerns w.r.t. the empirical validation are addressed since this paper presents a potentially interesting theoretical result with Adam, which is so prominent in the literature nowadays.
With respect to the empirical analysis, it is not clear to me why only 23 Atari games were used. The traditional answer often revolves around limited computational resources, but for this set of experiments it is particularly concerning. *How were these 23 games chosen?* The reason I ask is that DQN, without a target network, does work in approximately half of the Atari games. In the other games DQN presents instability and it doesn’t succeed. I wonder if the proposed approach would actually be stable throughout the whole set of Atari games. Moreover, it is not clear to me *why such a small buffer size was used*. As acknowledged in the paper: “Considering we use a smaller buffer size than common practice, DQN is not consistently showing improved return over all tested games”. It seems to me that DQN was evaluated in a suboptimal parameter setting, so it is not clear to me how relevant the claims that the evaluated algorithm outperforms DQN are.
The results itself, which are summarized in Figure 2 and in the Appendix, in Figure 3, are not that convincing (I would also like to see a table with the raw numbers at the end, as it is often done). In Figure 2, as acknowledged in the paper, “AltQ-Adam” has a huge variance at the end (raising concerns about its stability, as I previously mentioned). “AltQ-AdamR” does seem to be more stable but the confidence intervals (or standard deviation?) overlap (red and blue). Is there a significant difference? Looking at Figure 3 in the Appendix, the individual learning curves are not that convincing as well. In 12 out of 23 games DQN outperforms the other methods or there is pretty much no difference in performance between them (i.e., Alien, Assault, Asteroids, Bowling, CrazyClimber, FishingDerby, JamesBond, Pitfall, Pong, QBert, Robotank, Tutankham). In 11 out of 23 games there’s a clear advantage of AltQ-Adam or AltQ-AdamR (i.e., Amidar, Asterix, BeamRider, Enduro, DemonAttack(?), DoubleDunk, Gopher, Gravitar, Seaquest, SpaceInvaders, Tennis). Consequently, although AltQ seems better than DQN, the difference is not that big. Importantly, some crucial details are missing (at least I couldn’t find them). *How many seeds were used in each game?* One seed is way too little since the performance difference in Tennis could be explained by “luck” in the random action selection process. Which Atari version was used? The deterministic one or the stochastic one (Machado et al., 2018)? 
With respect to presentation, I don’t understand why Q-learning is being called AltQ. AltQ, with linear function approximation, is exactly what Q-learning does. *Introducing a new name for an old algorithm is very distracting* and it actually might reduce the impact of the paper, since more people can relate to Q-learning than to AltQ, which sounds as a newly introduced algorithm. Some details are not clear in Algorithms 1 and 2. *Is there a minibatch? That’s not represented in these algorithms, it it? Would it be fair to say that AltQ is _online_ Q-learning? I don’t think so since it uses a minibatch uniformly sampled from the experience replay buffer. It would be beneficial to have a paragraph explicitly discussing what are the differences between AltQ and DQN. Is it only dropping the target network?* Finally, there are some typos (e.g., “Nestrov”) and the references could be improved. Specifically, I’d recommend the authors to not use citations as nouns (e.g., “which is justified in (Duchi et al., 2011).”) and to cite Bellemare et al. (2013) instead of (or with) Brockman et al. (2016) when referring to the Atari games.
References:
Marc G. Bellemare, Yavar Naddaf, Joel Veness, Michael Bowling: The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res. 47: 253-279 (2013)
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael Bowling: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61: 523-562 (2018)
--- 
>>> Update after rebuttal: I stand by my score after the rebuttal. 
This paper has several presentation issues that need to be addressed before publication. The theoretical result does make it an interesting contribution, but the empirical section weakens the paper quite a lot. The paper is trying to show that the proposed approach is more stable but the smaller replay buffer size is not satisfying, since it is a setting which no one uses. Moreover, it is not possible to claim that a method presents lower variance when all one has are to 2 seeds to back up those claims. I don't think "limited computational capability and cost" is a good justification in that case.
Finally, the term AltQ is distracting and misleading. The authors didn't demonstrate any intention of changing this name, but if they decide to submit this paper to a next conference, with stronger empirical evidence to back up the paper claims, I recommend them to drop the AltQ learning name as well.