Efficient exploration is essential to reinforcement learning in huge state space. Recent approaches to address this issue include the intrinsically motivated goal exploration process (IMGEP) and the maximum state entropy exploration (MSEE). In this paper, we disclose that goal-conditioned exploration behaviors in IMGEP can also maximize the state entropy, which bridges the IMGEP and the MSEE. From this connection, we propose a maximum entropy criterion for goal selection in goal-conditioned exploration, which results in the new exploration method novelty-pursuit. Novelty-pursuit performs the exploration in two stages: first, it selects a goal for the goal-conditioned exploration policy to reach the boundary of the explored region; then, it takes random actions to explore the non-explored region. We demonstrate the effectiveness of the proposed method in environments from simple maze environments, Mujoco tasks, to the long-horizon video game of SuperMarioBros. Experiment results show that the proposed method outperforms the state-of-the-art approaches that use curiosity-driven exploration.
The authors study the problem of exploration in deep reinforcement learning. The authors borrow the ideas developed in the intrinsically motivated goal exploration process, and entropy maximization and propose a method on Noverly-Pursuit. The authors then empirically study the proposed method. 
1) The authors investigate an important problem and I would appreciate the authors if they could motivate its importance more in their work.
2) In the second paragraph, the author mentioned that goal-conditioned exploration behaviours can maximize entropy. Later in the same paragraph, they claim that "The exploration policy leads
to maximize the state entropy on the whole state distribution considering tabular MDP". I guess the authors' point was this approach might increase the entropy rather than maximizing it. If the claim is, in fact, maximization, a reference would be helpful. If the authors prove it in this paper, implying it in this paragraph is also helpful. 
3) In the background section, the authors did not specify whether they provide background on tabular MDP or beyond that. By calling the transition kernel the state transition probabilities, it seems they introduced a tabular MDP, but a more concrete introduction and preliminaries would help to follow the paper.
3) The first paragraph of section 2, the author mentioned that
"The target of reinforcement learning is to maximize the expected discounted return". I hope the authors mean"one of the targets in the study of reinforcement learning ..."
4) in the same paragraph, why the \gamma = 0 is excluded? is there any specific reason? Also, when the authors include the gamma = 1, do they make sure the maximization in line 9 of the same paragraph is well defined in regular cases?
5) Regarding the experiment in figure 2. It would be useful to the readers if the authors provide more details about this experimental study.
6) In the few paragraphs below Figure 2, it would be nicer if the authors provide a clear definition of each term. In order to follow the paper, I relied on my imperfect inference to infer the definitions. Also, I find it probably useful to distinguish the random variables and their realizations in the notation.
6) Regarding the theorem1. I would recommend making the statement more transparent and more clear. I also recommend to even not calling it a theorem since it, as mentioned, is as clear as the definitions. Also, arent x_t(i)s non-negative by definition? 
7) In this sentence:
"However, we don’t know what non-visited states are and
where non-visited states locate in practice since we can’t access ..."
I think the authors' point was that "we might not have access to it in general".
8) It would be helpful to me to evaluate this paper if the authors explain more how the following statements go through:
"To deal with this problem, we assume that state density over the whole state space is continuous, thus visited states and non-visited states are close". I am not sure how "thus visited states and non-visited states are close" follows from continuity of density and what is the notion of closeness. 
9-1) Theorem 2: while H seems to be a function of d_\pi1:t(s), I am not sure how to interpret the argmax_{e_t}. A bit of help from the authors would be appreciated. 
9-2) Theorem 2: In the proof, I was not able to justify to my self the transition form g(xi; xj) = Hxi [d1:t+1] 􀀀 Hxj [d1:t+1] to the second line. Also, what is the definition of H_x_i?
10) In equation 2, the authors use a notation d, I guess as distance. It would not only be helpful to define it but also would be helpful to use a different notation for distance and the d used on page 3, presumably for "empirical state distribution". 
11) At the beginning of the paper, the authors motivated the maximum entropy but the final algorithm is based on other approaches. 
12) Despite the fact that I could not find this paper ready enough and well-posed, I also have a concern about the novelty of the approach. I think it is not novel enough for publication at ICLR, but I am open to reading other reviewers', as well as commenters', and more especially the authors' rebuttal response.
13) I also encourage the authors to provide a discussion on the cases where the novelty (whatever that could mean) does not matter, rather the  novelty of state-action pair matters.