Deep reinforcement learning (RL) algorithms have shown an impressive ability to learn complex control policies in high-dimensional environments. However, despite the ever-increasing performance on popular benchmarks like the Arcade Learning Environment (ALE), policies learned by deep RL algorithms can struggle to generalize when evaluated in remarkably similar environments. These results are unexpected given the fact that, in supervised learning, deep neural networks often learn robust features that generalize across tasks. In this paper, we study the generalization capabilities of DQN in order to aid in understanding this mismatch between generalization in deep RL and supervised learning methods. We provide evidence suggesting that DQN overspecializes to the domain it is trained on. We then comprehensively evaluate the impact of traditional methods of regularization from supervised learning, ℓ 2 and dropout, and of reusing learned representations to improve the generalization capabilities of DQN. We perform this study using different game modes of Atari 2600 games, a recently introduced modification for the ALE which supports slight variations of the Atari 2600 games used for benchmarking in the field. Despite regularization being largely underutilized in deep RL, we show that it can, in fact, help DQN learn more general features. These features can then be reused and fine-tuned on similar tasks, considerably improving the sample efficiency of DQN.
I totally disagree with the authors that any of their observations are surprising. Indeed the fact that an RL agent does not generalizes to small modifications of the task (either visual or in the dynamics) is well known. If the agent should generalize though is a different question. And I do not mean this in the sense that it is an undesirable property but rather if it is outside of what “learning one task” means. Particularly I feel this is a very pessimistic view of RL and potentially not even in-line with what happens in supervised learning. 
I think one mantra of deep learning (and deep RL needs to obey by it) is that one should test in the same setting as training for things to truly work. For supervised, there is a distribution of data, and the test set are samples from the same distribution. However the testing scenario used here is slightly different. During training, if I do not see car accelerating, I think it makes no sense to expect to generalize to a new game that has this property as it is out-of-distribution. Of course it would be ideal if it could do that. And to clarify, while for us some of these extensions seem very similar and minimal changes, hence it should generalize to rather than transfer to, this is just the effect of imposing our own biases on the learning process. Deep Nets do not learn like we do, and in their universe they have never seen a car accelerating -- it makes sense that it might not to be able to generalize to it. Again, I’m not arguing that we don’t want this, but rather if we should expect it as part of what the system should normally generalize to.
To that end I think this paper enters in that unresolved dispute of what generalization should be versus what is transfer. At what point do we have truly a new task vs a sample from the same task. I don’t think there is an answer.
Going back to the observations in this work. I think the fact that the environment is not stochastic reinforces this overfitting (as in the extreme you end up with a policy that just repeats the optimal sequence of actions). I think in this particular case I can see how finetuning to a variation of the task fails. However true stochasticity in the environment (e.g. having a distribution of variations) like is done in Distral paper (where each episode is a different layout) can behave as a regularizer that will mitigate a bit the overfitting. That is to say that I believe the observed behaviour will be less pronounced in complex stochastic setting. 
Nevertheless the paper seems to highlight an important observation (and back it up with empirical evidence), namely we should use more regularization like L2 or otherwise in practice. Which is mostly absent from publications. And I think this on its own is valuable.