Reinforcement learning agents need to explore their unknown environments to solve the tasks given to them. The Bayes optimal solution to exploration is intractable for complex environments, and while several exploration methods have been proposed as approximations, it remains unclear what underlying objective is being optimized by existing exploration methods, or how they can be altered to incorporate prior knowledge about the task. Moreover, it is unclear how to acquire a single exploration strategy that will be useful for solving multiple downstream tasks. We address these shortcomings by learning a single exploration policy that can quickly solve a suite of downstream tasks in a multi-task setting, amortizing the cost of learning to explore. We recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution, which can incorporate prior knowledge about the task. We optimize the objective by reducing it to a two-player, zero-sum game between a state density model and a parametric policy. Our theoretical analysis of this approach suggests that prior exploration methods do not learn a policy that does distribution matching, but acquire a replay buffer that performs distribution matching, an observation that potentially explains prior methods' success in single-task settings. On both simulated and real-world tasks, we demonstrate that our algorithm explores faster and adapts more quickly than prior methods.
Summary:
The paper proposes to frame exploration in reinforcement learning as a distribution matching problem. More specifically, the proposed method (SMM) aims to minimize the reverse KL between the state distribution induced by the policy and a desired state distribution. The desired state distribution can be used to guide exploration (e.g. by penalizing bad states) or can be chosen uniformly (resulting in a policy that maximizes state entropy).
SMM iteratively optimizes the policy and a model for its induced state-distribution. The latter is used for approximating the policies state entropy. The algorithm is framed as a two-player zero-sum game between a "policy player" and a "density player". In order to justify that one of the players is assumed fixed while optimizing the other player, SMM optimizes the players against a historical average of the opponent. Such approach is known as fictitious play in game theory and ensures convergence to a Nash equilibrium.
The density player maximizes the likelihood of the states encountered during all previous roll-outs using a VAE. The policy update is framed as a standard reinforcement learning problem where the log-pdf of the target distribution serves as reward and the log-pdf of the model acts as cost. For practical reasons, the policy update does not consider an historical average but only uses the most recent density model. As the learned exploratory policy should correspond to an historical average exploration for the downstream task is achieved by sampling one of the learned policy at the beginning of each roll-out. The historical averaging also benefits prior exploration methods.
Notably, the appendix also describes a modification that learns a mixture of policies in each iteration where a discriminator approximates the responsibilities for each component which is used as additional reward to divide the search space among the components. SMM is compared to prior exploration methods, ICM and Pseudo-Counts and standard SAC, on a simulated robotic box-pushing task and against SAC on simulated point-mass problem and a real-robot valve turning task.
Significance:
Efficient exploration is arguably the main challenge of reinforcement learning as providing shaped reward functions is difficult even for experts and may lead to undesired behavior. I think that maximizing the state-entropy (or matching target state-distributions if available) is a sound objective for learning exploratory policies and the paper could, thus, be of relatively broad interest. 
Novelty:
Maximizing the state entropy for exploration has already been used by (Hazan et al., 2018, reference from manuscript). However, in contrast to this prior work, SSM does not need to learn oracles that predict optimal state-distributions/policies for any given policies/reward functions. While distribution matching is a common approach to imitation learning, it has been little employed for manually specified distributions. Still, a similar objective has been used to replace reward functions by desired distributions in a RL-like setting [1] (not for exploration). However, their approach is quite restricted by assuming Gaussian target distributions.
Soundness:
If I understand correctly, fictitious play assumes optimal responses to understand that the state distribution would often be more important than provably converge to a Nash equilibrium. The paper fails to provide stopping criteria for the optimization steps of the individual players; however, I assume that only few gradient steps are used for practical reasons. Hence, I am not sure whether the actual algorithm can be justified by the theory.
The paper mentions that VAEs are used to model the state distribution. Given that VAEs are not likelihood-based, I do not understand how the reward term log(q(s)) can be computed. 
Clarity:
The paper is well-written and the structure seems fine. 
I think that the density estimation should be better discussed. Models of the state distribution of the policy are often highly desirable, not only for optimizing its entropy, but also, for example, for importance sampling. However, modeling the state distribution is also inherently difficult--especially for large state spaces.
Experiments:
I like that the paper uses a real robot experiment and the ablations with respect to the historical averaging are interesting. Unfortunately, the paper only compares to standard SAC on the hallway task and on the real robot task. I would not consider the entropy regularization of SAC a proper baseline for a proposed exploration technique.
Questions:
Is the same extrinsic reward, i.e. log(p*), also provided to the other exploration methods?
Some imitation learning methods such as GAIL are also able to match desired state-distributions. I think that these methods could be in principle also applied to the setting considered in the manuscript by using samples from the desired distribution as demonstrations. The paper briefly mentions that IRL methods are not applicable because they require access to trajectories, however, the discriminator of GAIL is only trained on individual state samples. I also do not see a problem of providing unachievable demonstrations to such imitation learning methods because, such like SSḾM, they would try to minimize the divergence. I think that GAIL would actually be an important baseline for the proposed method.
How does the method scale with the dimensions of the state? SSM has only be evaluated on relatively low-dimensional problems (compared to some rllab/mujoco tasks with >100 states). I would assume that obtaining meaningful density estimates in such settings might be problematic. May imitation learning methods based on discriminators actually be more promising?
SMM only considers matching state distributions. If I understand correctly, the approach could be easily generalized to state-action distributions, correct? Wouldn't it make sense for some tasks to also explore in the action space?
Decision:
I like the paper in general. The optimization problem seems well-motivated and the algorithm seems reasonable fine. I also like that the paper includes a real robot experiment. However, I am not convinced by the derivation of SMM based on fictitous play and I think that it should be better evaluated with respect to existing exploration methods and also with respect to imitation learning methods. I am therefore slightly leaning towards rejection, currently. 
Typo:
"Note[sic] only does SMM explore a wider range of angles than SAC, but its ability to explore increasing[sic] throughout training, suggesting that the SMM objective is correlated with real-world metrics of exploration."
[1] Arenz, Oleg, Hany Abdulsamad, and Gerhard Neumann. "Optimal control and inverse optimal control by distribution matching." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.