The Boltzmann softmax operator can trade-off well between exploration and exploitation according to current estimation in an exponential weighting scheme, which is a promising way to address the exploration-exploitation dilemma in reinforcement learning. Unfortunately, the Boltzmann softmax operator is not a non-expansion, which may lead to unstable or even divergent learning behavior when used in estimating the value function. The convergence of value iteration is guaranteed in a restricted set of non-expansive operators and how to characterize the effect of such non-expansive operators in value iteration remains an open problem. In this paper, we propose a new technique to analyze the error bound of value iteration with the the Boltzmann softmax operator. We then propose the dynamic Boltzmann softmax(DBS) operator to enable the convergence to the optimal value function in value iteration. We also present convergence rate analysis of the algorithm. Using Q-learning as an application, we show that the DBS operator can be applied in a model-free reinforcement learning algorithm. Finally, we demonstrate the effectiveness of the DBS operator in a toy problem called GridWorld and a suite of Atari games. Experimental results show that outperforms DQN substantially in benchmark games.
Summary: This work demonstrates that, although the Boltzmann softmax operator is not a non-expansion, a proposed dynamic Boltzmann operator (DBS) can be used in conjunction with value iteration and Q-learning to achieve convergence to V* and Q*, respectively. This time-varying operator replaces the traditional max operator. The authors show empirical performance gains of DBS+Q-learning over Q-learning in a gridworld and DBS+DQN over DQN on Atari games.
Novelty: (1) The error bound of value iteration with the Boltzmann softmax operator and convergence & convergence rate results in this setting seem novel. (2) The novelty of the dynamic Boltzmann operator is somewhat thin, as (Singh et al. 2000) show that a dynamic weighting of the Boltzmann operator achieves convergence to the optimal value function in SARSA(0). In that work, the weighting is state-dependent, so the main algorithmic novelty in this paper is removing the dependence on state visitation for the beta parameter by making it solely dependent on time. A question for the authors: How does the proof in this work relate to / differ from the convergence proofs in (Singh et al. 2000)?
Clarity: In the DBS Q-learning algorithm, it is unclear under which policy actions are selected, e.g. using epsilon-greedy/epsilon-Boltzmann versus using the Boltzmann distribution applied to the Q(s, a) values. If the Boltzmann distribution is used then the algorithm that is presented is in fact expected SARSA and not Q-learning. The paper would benefit from making this clear.
Soundness: (1) The proof of Theorem 4 implicitly assumes that all states are visited infinitely often, which is not necessarily true with the given algorithm (if the policy used to select actions is the Boltzmann policy). (2) The proof of Theorem 1 uses the fact that |L(Q) - max(Q)| <= log(|A|) / beta, which is not immediately clear from the result cited in McKay (2003). (3) The paper claims in the introduction that “the non-expansive property is vital to guarantee … the convergence of the learning algorithm.” This is not necessarily the case -- see Bellemare et al., Increasing the Action Gap: New Operators for Reinforcement Learning, 2016. 
Quality: (1) I appreciate that the authors evaluated their method on the suite of 49 Atari games. This said, the increase in median performance is relatively small, the delta being about half that of the increase due to double DQN. The improvement in mean score in great part stems from a large improvement occurs on Atlantis.
There are also a number of experimental details that are missing. Is the only change from DQN the change in update rule, while keeping the epsilon-greedy rule? In this case, I find a disconnect between the stated goal (to trade off exploration and exploitation) and the results. Why would we expect the Boltzmann softmax to work better when combined to epsilon-greedy? If not, can you give more details e.g. how beta was annealed over time, etc.?
Finally, can you briefly compare your algorithm to the temperature scheduling method described in Fox et al., Taming the Noise in Reinforcement Learning via Soft Updates, 2016?
Additional Comments:
(1) It would be helpful to have Atari results provided in raw game scores in addition to the human-normalized scores (Figure 5). (2) The human normalized scores listed in Figure 5 for DQN are different than the ones listed in the Double DQN paper (Van Hasselt et al, 2016). (3) For the DBS-DQN algorithm, the authors set beta_t = ct^2 - how is the value of c determined? (4) Text in legends and axes of Figure 1 and Figure 2 plots is very small. (5) Typo: citation for MacKay - Information Theory, Inference and Learning Algorithms - author name listed twice.
Similarly, if the main contribution is DBS, it would be interesting to have a more in-depth empirical analysis of the method -- how does performance (in Atari or otherwise) vary with the temperature schedule, how exploration is affected, etc.?
After reading the other reviews and responses, I still think the paper needs further improvement before it can published.