Learning values across many orders of magnitude
Most learning algorithms are not invariant to the scale of the signal that is being approximated. We propose to adaptively normalize the targets used in the learning updates. This is important in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.
normalization can be used to disentangle the natural magnitude of each component from its relative importance in the loss function. This is particularly useful when the components have different units, such as when we predict signals from sensors with different modalities. Finally, adaptive normalization can help deal with non-stationary. For instance, in reinforcement learning the policy of behavior can change repeatedly during learning, thereby changing the distribution and magnitude of the values.
The authors propose a method for adaptively normalizing the output distribution of the regression targets a neural network is trained to produce. The idea is to be able to elegantly deal with output targets of different scales, especially in the situation where the normalization of the targets cannot be computed a priori, as is generally the case when learning action value functions in reinforcement learning. The authors show on a toy task that their method is able to deal with low frequency targets that are much larger than the average targets. They show on the Atari 2600 reinforcement learning task that their system performs better on average than systems that use a clipping heuristic. Their model also seems to learn qualitatively different behavior because it's able to distinguish between rewards of different magnitudes.
I think the problem that the authors are trying to solve is a very important one that shows up in many gradient-based learning situations. The ideas are straightforward but (to my knowledge) new and apparently effective. For this reason I can see them becoming widely used.  The paper is well-written and at most places clear. The related work section seems to contain enough relevant references, but it would be nice if some of the most related works would be discussed in a bit more detail. The work by Wang et al. (2016), for example, seems related to me in the sense that it seems to make the DQN in that paper less sensitive to the mean of the values as well.  I like the toy example. While it seems to me that a linear model should be able to solve this task (one solution would be to set the input weights equal to powers of two), the gap between the presented data and the infrequent high target makes this impossible to learn from the data and representative of a problem that may even occur when the output targets are normalized but still follow a fat-tailed distribution.   I found it a bit difficult to judge the Atari 2600 results because they represent such a high dimensional comparison of different types of scores. The movies in the supplementary materials (especially Centipede) look convincing and the method sometimes performs much better than the baseline, but it also performs much worse on some games. On average, the method performs better and I agree with the authors that being able to learn most of the games without clipping is already a step forward. It would be interesting to see how well the method would do with at least a bit of hyperparameter tuning. Especially the discount factor. The authors also point out that the success of models that use clipping might be somewhat specific to this task. For this reason, it would have been interesting to see a comparison on another reinforcement learning task as well.  Some small things:  Line 55: The word 'most' is very imprecise for statements of this kind. Does this mean that one of the natural gradient methods is actually invariant to rescaling the targets?  Line 66: I think a more correct interval for the step size would be (0, inf) instead of [0, 1]. You clearly don't want to include 0 and I don't see why a step size couldn't be greater than 1.  Line 88: The actual 'objective' mentioned in this line is never discussed. The rescaling updates seem to be more about satisfying certain constraints like the ones presented in Equation 3 and the bound on the normalized targets.  Algorithm 2: The update of W seems to contain a typo where 'g' should probably be 'h'.