In many reinforcement learning settings, rewards which are extrinsically available to the learning agent are too sparse to train a suitable policy. Beside reward shaping which requires human expertise, utilizing better exploration strategies helps to circumvent the problem of policy training with sparse rewards. In this work, we introduce an exploration approach based on maximizing the entropy of the visited states while learning a goal-conditioned policy. The main contribution of this work is to introduce a novel reward function which combined with a goal proposing scheme, increases the entropy of the visited states faster compared to the prior work. This improves the exploration capability of the agent, and therefore enhances the agent's chance to solve sparse reward problems more efficiently. Our empirical studies demonstrate the superiority of the proposed method to solve different sparse reward problems in comparison to the prior work.
This paper studies the problem of exploration in reinforcement learning. The key idea is to learn a goal-conditioned agent and do exploration by selecting goals at the frontier of previously visited states.  This frontier is estimated using an extension of prior work (Pong 2019). The method is evaluated on two continuous control environments (2D navigation, manipulation), where it seems to outperform baselines.
Overall, I like that the proposed method integrates some notion of novelty with the language of mutual information and density estimation. While this idea has been explored in prior work (as noted in the related work section), the proposed idea seems like a useful contribution to the literature. The use of convolutions to obtain a lower bound on mutual information seems neat. The experimental results are quite strong.
My main concern with the paper is a lack of clarity. I currently have enough reservations and questions (listed below) about the experimental protocol that I am learning towards rejecting this paper. However, if the paper clarified the concerns below, I'd be willing to increase by review.
Questions / Concerns:
* "[Prior work on mutual-information cannot] guarantee that the entire state space can be covered" -- In theory, I think these prior methods should cover the entire state space. Take the DIAYN objective, I(s, z) = H[s] - H[s | z]. This objective is maximized when p(s) is uniform over the entire state space and p(s | z) is a Dirac.
* "It is time consuming to collect enough samples to estimate an accurate state entropy" -- Can you provide a citation/proof for this? Why should we expect the proposed method to require fewer samples?
* "...entropy itself does not provide efficient information to adjust the action at each step." -- Can you provide a citation / proof for this? Also, what does "efficient information" mean?
* It seems like, if S is a finite collection of states in a continuous space, then it has measure zero, so its entropy should be zero. Can you explain why this is not the case?
* If I'm not mistaken, in equation 2, if we take (say) alpha = -0.5, then w_i is proportional to sqrt(p(s_i)), so w_i is an increasing function in p(s_i), not a decreasing function.
* Can you discuss how you might scale a KDE to high dimensions?
* "If the distribution has a larger range, the entropy is larger as well." -- Technically, this is not correct. You can construct distributions with larger ranges but smaller entropies.
* I think that Equation 3 should be the KL divergence between state marginal distributions, not between trajectories. If it were the KL between trajectories, it would include actions and policy terms.
* How are w_int and w_ext chosen? It seems like the method depends critically on the balance between these hyperparameters. Is w_int decayed over time? If not, why does the policy stop exploring once it has found the goal?
* What policy is used at convergence? It seems like the policy is conditioned on Z_t, so how is the Z_t chosen for evaluation?
* Fig 5 -- How are entropy and coverage computed? What are the maximum possible values for both of these quantities? What precisely does the X axis correspond to?
* Table 1 -- How did the baseline algorithms perform on this task?
* Fig 7 -- How did the baseline algorithms perform on this task? If the reward were sparse, shouldn't the Y axis be in the interval [0, 1]?
* "using coverage only during training is not suitable" -- Can you provide a citation/proof for this?
* "As a consequence, the entropy of the distribution of these points is also maximized" -- I believe that a finite number of points in a discrete space have measure zero, so they have zero entropy, regardless of the position of the points.
Other comments
* "What is the difference between *S* (in bold) and S_t?
* I would recommend using some notation other than p(s) to denote the smoothed/convolved density. 
* "history states" -- I was confused about what this meant until it was introduced two sections later.
* "assimilate the definition of curiosity in psychology" -- I think that others (e.g., Oudeyer 2007, Pathak 2017) have noted the similarities between curiosity in humans and RL agents.
* Check for backwards quotes in the related work section.
* "Self-Goal Proposing" -- Some more related works are [Florensa 2017, Savinov 2018]
* "space associated environment" -- I don't know what this means.
* "disc rewards" -- I'd recommend spelling out discounted
* "truncated Gaussian function with a narrow range" -- Can you explain precisely what this is?
* In equation 2, I think it'd be clearer to write p^(1+\alpha).
* For the experiment on the effect of variance, I'd recommend making a plot instead of just listing the values.
* In Section 4.3, it's unclear whether the physical robot was successful at solving the task.
* "We rewrite the equation…" -- This paragraph is repeated.
* Double check that \citet and \citep are used properly
--------------UPDATE AFTER AUTHOR RESPONSE------------------
Thanks for answering many of my questions. This was helpful for clarifying my understanding. However, since a large fraction of my concerns were not addressed, so I am inclined with stick with my original vote to reject the paper. Nonetheless, I should emphasize that I think this paper is on the right track and the empirical results seems strong. With a bit more work on writing, I think it would be a fantastic paper at the next conference.