Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of γ = 0.999 (instead of γ = 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters.
Summary: This paper proposes three new techniques for improving Atari performance over APE (Horgan 2018).  Two of them are closely linked in that they deal with improving stability.  Another involves integrating the use of expert trajectories from DQfD.  
I will summarize each: 
Transformed Bellman: This applies a rescaling function (it's basically a monotonically increasing version of the sqrt(x) function) to the Q-function and applies the inverse of the function to the max Q-value of the next state (such that the contracting effect h-function is not "applied" multiple times when doing the TD backup).  
Temporal Consistency: This encourages the "next state" after where the TD-update is applied to not change too much.  This addresses a problem discussed in (Durugkar 2018).  I think the intuition here is that the state which follows the state with the TD update may be visually similar, but it does not impact the value in the past states, so its value function should not have a highly correlated change with the previous state's change in value function.    
DQfD: Storing an expert replay buffer and an actor replay buffer.  The expert replay buffer is fixed and the actor replay buffer stores the most recent "actor processes".  Train with both a supervised imitation loss (only for the highest return episode) and the original TD loss.  Additionally, the pre-training phase is removed and the ratio of expert-learned trajectories is fixed (both seem like steps in the right direction).  
Review: This paper proposes a few changes to DQN training, two of which are aimed at reducing instability, and one is aimed at improving exploration (expert trajectories).  Because all of these changes are well justified and the experiments are fairly thorough, I recommend acceptance.  My main reservation is that the ideas presented are not very strongly thematically linked.  The presence of ablation studies compensates for this to some extent.  
Strengths: 
  -The discussion of related work and comparison to baselines is pretty extensive.  For example I appreciated the ablation study removing "transformed Q-learning" and comparison to the pop-art method.  
  -The results, at least for Ape-X DQfD seem impressive to me in that the method works without reward clipping and with a much higher discount factor.  Additionally the results generally outperform DQfD (uses expert trajectories) and Rainbow (no human trajectories).  Additionally evidence was presented that the learned policies often exceed the performance of the human demonstrations (for example in time to achieve rewards).  
Weaknesses: 
  -Two of the techniques: "transformed bellman" and "temporal consistency" seem well-linked thematically, but the expert demonstration idea seems orthogonal.  I would have preferred splitting that idea out into a separate paper, given that the paper is already 20 pages.  
  -The motivation for temporal consistency just references (Durugkar 2018).  The readability of this paper would be improved if it were discussed more here as well.  I also feel like the analysis could be more thorough here, for example a result using the temporal consistency loss on Baird's counter example really should be shown (like figure 2 in Durugkar's paper).  
-It would be nice to see a visualization or a toy problem with the "transformed bellman".  
Questions: 
-Is the "highest return episode" idea (3.4) general or is it exploiting the fact that Atari is deterministic?  It seems like in general we'd want to use many high reward episodes, or the highest reward episodes that go into different parts of state space.  It seems like it could be a very bad idea on certain settings (for example if the reward has a lot of randomness).  
-"Proposition 3.1 shows that in the basic cases when either h is linear or the MDP is deterministic, Th
has the unique fixed point h ◦ Q∗".  From 3.1, it looks if h is linear, then it distributes over r(x,a) + maxh^{-1}(Q) and then it also won't effect which is the max, so it would reduce to h*r(x,a) + max(Q) - which means it's just rescaling the original reward.  So then this result is trivial?  Please correct me if I misunderstood something here.  
-Could an MDP be constructed which causes the transformed bellman operator to perform badly?  I am imagining something where the MDP is just a single step, and there is a stochastic action which behaves like a lottery.  So perhaps there is a 1-in-1-million chance to win 1-billion dollars by taking an action.  If I understand correctly the transformed bellman operator will destroy the large reward here (because in a single step, there is just r(x,a) which h is applied to).  Which would make the action seem bad even though it's actually appealing.  
Notes: 
  -I did not read the proofs in the appendix.