A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).
Review:
This paper considers the problem of overfitting in RL through a specific model of overfitting, namely one where noise is introduced in the observation space, independently of the controllable dynamics.  The paper provides both theoretical and empirical insights into the various manifestations of overfitting in this class of domains.
A strength of this paper is to deepen our understanding of the phenomenon of overfitting in RL.  To get a deep understanding of hard problems it is worthwhile to look at sub-classes of problems, and as such the identification of the observational overfitting setting is interesting.  The paper provides insightful findings such as:
-	Explicit separation of f and g_theta.
-	Theoretical properties of generalization for the specific case of LQR and linear policies.
There are also several empirical results, on three contrasting domains which illustrates various aspects of the interaction between parameterization vs generalization.
But overall, though I read the paper in detail, and am knowledgeable of the topic, I am still struggling with extracting very specific new findings from this work.  Here are a few examples:
p.6:  “We observe empirically that the underlying state dynamics has a significant effect on generalization performance as the policy nontrivially increased test performance” ->  I see where you draw this analysis from the results.  But is this finding new or surprising?  I would have been very comfortable saying (without your results) that the underlying state dynamics can have a significant effect on generalization performance.
p.6: “This suggests that the Rademacher complexity and the weight perturbation bound for rewards vary highly for different environments.”  -> Again, how is this new knowledge?
p.7: “this also suggests that the RL generalization quality of a convolutional architecture is not limited to real world data” -> Same question, what new knowledge have we gained?
Perhaps the key message is that “there are generalization benefits to overparameterization” and that “implicit generalization” plays a key role in controlling this?   If that is the main message, than it is somewhat obfuscated by all the material on LQR, which doesn’t really go in this direction (p.4: “We being with a theorem which implies that a high dimensional observational space directly contributes to overfitting”).  I must say I found most of the material in 3.1 to distrct from the main message – but perhaps it is because I am not very interested in the LQR setting, and did not understand how the findings there supported those in latter sections.
Overall, I would say the paper suffers from some clarity issues.  There are some minor typos, then some key terms of are not sufficiently explained/defined (e.g. g_\theta in Sec.2.2 is said to project “the latent data to unimportant features”, but then there is discussion of “in settings where $g$ does matter” – I’m confused.)   It’s not clear whether Thm 3.1 is new, or adaptation of existing results.  More broadly, I found Sec.3.1 difficult to follow. The last paragraph of Sec.4 seems superfluous.   I would recommend some good editing throughout.
Another concern with the work is the fact that several aspects of RL are ignored (e.g. exploration, entropy, \gamma, noise, stochastic gradients, etc. – as per p.4), yet are known to have an effect on overfitting, perhaps much more than the depth and width of the neural network.  If that’s the case, it is perhaps dangerous to ignore them in the analysis; it is not described in the paper, for the domains studied in the empirical results, whether this is the case or not.  As a result, it’s not clear how far the current analysis carries.  I’m also not clear on the analysis at bottom p.7 “IMPALA-LARGE (…) memorizes less than IMPA due its inherent inductive bias” -> Seems to me IMPALA-LARGE might simply be in underfitting regime.
======
Update post-rebuttal:
Thank you for the thoughtful responses.   Given the other strong reviews, and your careful characterization of various aspects of the work, I am favorable to accepting.  Please try to incorporate elements of this discussion into the paper, which I think will give a richer and more nuanced understanding of the work to future readers.