Natural Value Approximators: Learning when to Trust Past Estimates
Neural networks have a smooth initial inductive bias, such that small changes in input do not lead to large changes in output. However, in reinforcement learning domains with sparse rewards, value functions have non-smooth structure with a characteristic asymmetric discontinuity whenever rewards arrive. We propose a mechanism that learns an interpolation between a direct value estimate and a projected value estimate computed from the encountered reward and the previous estimate. This reduces the need to learn about discontinuities, and thus improves the value function approximation. Furthermore, as the interpolation is learned and state-dependent, our method can deal with heterogeneous observability. We demonstrate that this one change leads to significant improvements on multiple Atari games, when applied to the state-of-the-art A3C algorithm.
This paper proposes a novel way to estimate the value function of a game state, by treating its previous reward and value estimations as additional input besides the current state. These additional input is the direct reason why value function depicts a non-smoothness structure (e.g., a sparse immediate rewards lead to a bump in the value function). By taking them into consideration explicitly, the value function can be more easily estimated.
Although the proposed idea is quite interesting, there are a lot of baselines that the proposed method might need to compare. As mentioned in Section 7, eligible trace estimates the cumulative return as a linear combination of k-step *future* returns with geometric weights. Also Generalized Advantage Estimation, as a novel way to estimate the overall return using future estimation, is another example. Comparing these previous approaches with the proposed method that uses past estimation is both interesting and important.  
In addition, there exist many simple approaches that also captures the history (and sparse reward that happened recently), e.g., frame stacking. A comparison will also be interesting. 
The paper starts with the story that the value function is smooth, if the non-smooth part is explained away by previous rewards and values. There are also motivating examples (e.g., Fig. 1 and Fig. 2). However, there is no experiments in Atari games showing that the estimated value function using proposed approach is indeed smoother than the baseline. 
Table. 1 shows that while median score shows a strong boost, mean score of the proposed approach is comparable to the baseline. This might suggests that the proposed approach does a good job in reducing the variance of the trained models, rather than giving higher performance. In the paper, there seems to be no relevant analysis. 
Some detailed questions:
1. According to the paper, the modified A3C algorithm uses N = 20 steps, rather than N = 5 as in the vanilla A3C. Did the baseline also use N = 20? Note that N could be an important factor for performance, since with long horizon the algorithm will see more rewards in one gradient update given the same number of game frames. 
2. Line 154 says "all agents are run with one seed". Does that mean the agents are initialized by the same random seed among different games, or the game environment starts from the same seed? Please clarify.