All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis. We propose that such a search for order amidst chaos might offer a unifying principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing RL (SMiRL). SMiRL trains an agent with the objective of maximizing the probability of observed states under a model trained on all previously seen states. The resulting agents acquire several proactive behaviors to seek and maintain stable states such as balancing and damage avoidance, that are closely tied to the environment's prevailing sources of entropy, such as winds, earthquakes, and other agents. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, and control a humanoid to avoid falls, without any task-specific reward supervision. We further show that SMiRL can be used together with a standard task reward to accelerate reward-driven learning.
Summary
This paper proposes a novel form of surprise-minimizing intrinsic reward signal that leads to interesting behavior in the absence of an external reward signal. The proposed approach encourages an agent to visit states with high probability / density under a parametric marginal state distribution that is learned as the agent interacts with its environment. The method (dubbed SMiRL) is evaluated in visual and proprioceptive high-dimensional "entropic" benchmarks (that progress without the agent doing anything in order to prevent trivial solutions such as standing and never moving), and compared against two surprise-maximizing intrinsic motivation methods (ICM and RND) as well as to a reward-maximizing oracle. The experiments demonstrate that SMiRL can lead to more sensible behavior compared to ICM and RND in the chosen environments, and eventually recover the performance of a purely reward-maximizing agent. Also, SMiRL can be used for imitation learning by pre-training the parametric state distribution with data from a teacher. Finally, SMiRL shows the potential of speeding up reinforcement learning by using intrinsic motivation as an additional reward signal added to the external task-defining reward.
Quality
As a practical paper, this work needs to be judged based on the quality of the experiments. I find the number of benchmarks and baselines sufficient. One major issue, that currently prevents me from voting for acceptance, is that experiments have not been conducted with enough seeds. I couldn't find any information in the paper regarding how many repetitions there are for each experiment. Also, most figures do not indicate any uncertainty measures (standard deviation, percentiles or the like)---some do, e.g. Figure (4), but it is not mentioned what type of uncertainty is depicted. One seed is certainly not enough to support the claims made by the authors---especially not the one that SMiRL can help improve RL in Figure (5). Figure (5a) clearly  does not draw a clear picture under one seed, and (5b) and (5c) require additional expert demonstrations. If experiments are repeated with more seeds and still support the claims, I am happy to increase my score to acceptance (presupposing that the discussion phase does not prevent acceptance for other reasons).
Clarity
The paper is clearly written and easy to follow. However, I do not like one aspect in the way the authors motivate their approach. The problem formulation starts with an MDP formulation. MDPs rely on a stationary reward signal and RL agents aim to optimize for future cumulative rewards based on these stationary reward signals. The authors propose to optimize for a non-stationary signal since the parametric state distribution changes over time. This itself is not uncommon and nothing controversial from a practical perspective. However, the statement that SMiRL agents seek to visit states that will change the parametric state distribution to obtain higher intrinsic reward in the future is controversial (see e.g. at the end of Section 2.1), because optimizing a non-stationary signal is outside the scope of the problem formulation. The reinforcement learning problem of maximizing intrinsic rewards does not know how the intrinsic reward signal is altered in the course of the future, i.e. how the parametric state distribution is updated. These statements should therefore be either adjusted accordingly, or the claims should be backed up theoretically rather than intuitively. On a minor note, I don't think that Figure (1) is necessary and the quote at the beginning of the paper might be better suited for a book chapter (but that is just my personal opinion).
Originality
I find the simple idea presented by the authors to minimize rather than maximize surprise quite original. However, I do not have much experience in the domain of intrinsic motivation and leave the judgement of originality to the other reviewers and the area chair. Also, some references might be missing regarding learning with intrinsic reward (e.g. empowerment).
Significance
The fact that a simple intrinsic reward, as presented by the authors, can lead to interesting behavior, as demonstrated by the experiments, is quite significant. Unfortunately, the experiments are not significant from a statistical perspective which is why I do not recommend acceptance at this stage (as mentioned above, depending on how the authors address this issue and the discussion period, I might change to acceptance).
Update
The authors have addressed my main concern regarding missing seeds. I therefore change to weak accept. But I am not happy how the authors responded to my concern regarding their motivation:
1.) The formulation that tries to justify their reasoning as given in the rebuttal was absent in the first version of this paper, but something along this line would have been necessary since the argument relies on a non-standard MDP formulation.
2.) I don't think the argumentation given in the rebuttal is correct. Imagine hypothetically the agent is in exactly the same augmented state s_t, a_t, \theta_t at two different points in time, e.g. in the first episode and after multiple episodes. In both cases, the collected states seen so far are going to be different, hence the optimization objective for learning a marginal state distribution is different, hence the parameter updates are different, hence the transitions are not stationary.
I do therefore encourage the authors to attenuate their wording.