Current Deep Reinforcement Learning (DRL) methods can exhibit both data inefficiency and brittleness, which seem to indicate that they generalize poorly. In this work, we experimentally analyze this issue through the lens of memorization, and show that it can be observed directly during training. More precisely, we find that Deep Neural Networks (DNNs) trained with supervised tasks on trajectories capture temporal structure well, but DNNs trained with TD(0) methods struggle to do so, while using TD(λ) targets leads to better generalization.
Summary:
This paper performs an empirical evaluation of generalization by TD methods with neural nets as function approximators. To quantify generalization, the paper considers the change in the loss function at similar states to the one where the update rule is being applied (where “similar” is usually defined as nearby in time). It comes to a variety of conclusions including that TD(0) does not induce much generalization, that TD(0) does not induce as much generalization as supervised learning, and that the choice of optimizer and objective changes the behavior according to their generalization criteria in various ways.
Decision:
This paper should be rejected because (1) the motivation is unsubstantiated, (2) the main metrics used (“gain” at nearby states and difference since last visit) are of questionable importance, and (3) the conclusions are often vague and not informative.
Main argument:
Motivation:
- The paper motivates the need for an evaluation of “gradient update generalization” by claiming that it is related to sample complexity and brittleness of RL algorithms. While I agree that this is plausible, there is nothing empirical or theoretical to support this claim in the paper or in the references. This is a significant problem since this assumed connection underlies everything in the paper.
- Also, it is this sort of generalization that differentiates the function approximation setting (where there are no convergence guarantees for TD without strong assumptions) from the tabular setting (where there are convergence and sample complexity guarantees). 
Metrics:
- The main metric used is TD gain on temporally nearby states. The TD gain is defined as the reduction in the squared TD error at some state s’ when an update is applied at some other state s. Note this metric does not capture all update generalization, but only update generalization as it effects the TD error.
- It is not evident, nor supported by the paper, that improvements in this metric at nearby states necessarily improve performance of the algorithm. This is especially true when there is a tradeoff between improvement at very nearby vs. somewhat nearby states, it is not clear which behavior is preferable (and this behavior seems to occur in experiments). As a result there is no clear way to use this metric to determine which algorithms are preferable. 
- The other metric used in the paper is the change in value function at a state since the last time that state was sampled from the buffer. It is also not clear whether this measurement is necessarily important for the same reasons as above. 
Conclusions:
- In general the results are presented in plots that do not give clear implications and are difficult to read. I understand that we cannot expect completely clean results on such empirical questions, but the results would be potentially much more convincing and clear if the hypotheses were clearly stated and then one plot could summarize the result with respect to each hypothesis. For example, the gain plots are difficult to compare and interpret clearly and the memorization plots do not give such clear results (eg. 3a and 3d look visually fairly similar). Another example is the comparisons between optimizers, which are fine to have as a specific point in one section, but do not need to be in every plot.
- The main result claimed in the paper is that there is little generalization from TD(0) when compared to supervised learning. This seems to be born out by the difference between figure 1a and 2a, but the difference in scale makes it a bit difficult. It is also not clear how to quantify this result or what the implications are. 
- The plots are averaged across all states over all of training and all environments, while this is somewhat rationalized in figure 11, I am worried that this may be covering some additional complexity/ambiguity in the results. 
- The result about TD(lambda) seems to be born out by figures 3e and f and 5, but is also unsurprising since the objective explicitly averages across temporally nearby states. Again it is not clear how this temporal consistency of updates should be interpreted in terms of the goals of the RL algorithms.
At a higher level, the paper feels like a solid preliminary set of experiments rather than a paper organized around a clear motivating idea with clear hypotheses to test. The results would become more interesting if the metrics used could be connected back (either empirically or theoretically) to an objective. For example, does generalization in the sense defined in the paper give better performance at value estimation or return maximization? Can these results be quantified in a more direct way than the plots presented in the paper? 
Additional feedback:
- The plots all have different scales which makes them difficult to compare
- Using “distance” to refer to the relative tilmestep of samples in the replay buffer is confusing (distance cannot be negative)
- There are strange visual artifacts (horizontal lines) in Figures 3, 5, 10, 12, 13, and 14
- Sections 3.7 and 3.8 very briefly present results that seem to distract from the main thread of the paper
- Using one network architecture across all experiments seems like it may have a significant impact on the results. I understand that the architecture chosen is standard and testing different architectures is time consuming, but making broad claims about the algorithms based on only one architecture is potentially dangerous. 
- I assume that the replay buffer is always being sampled uniformly, but I could not find this detail in the paper.