Exploration strategy design is one of the challenging problems in reinforcement learning (RL), especially when the environment contains a large state space or sparse rewards. During exploration, the agent tries to discover novel areas or high reward (quality) areas. In most existing methods, the novelty and quality in the neighboring area of the current state are not well utilized to guide the exploration of the agent. To tackle this problem, we propose a novel RL framework, called clustered reinforcement learning (CRL), for efficient exploration in RL. CRL adopts clustering to divide the collected states into several clusters, based on which a bonus reward reflecting both novelty and quality in the neighboring area (cluster) of the current state is given to the agent. Experiments on several continuous control tasks and several Atari-2600 games show that CRL can outperform other state-of-the-art methods to achieve the best performance in most cases.
This paper presents a clear approach to improve the exploration strategy in reinforcement learning, which is named clustered reinforcement learning. The approach tries to push the agent to explore more states with high novelty and quality.  It is done by adding a bonus reward shown in Eq. (3) to the reward function. The author first cluster states into clusters using the k-means algorithm. The bonus reward will return a high value for a state if the corresponding cluster has a high average reward. When the total reward in a cluster is smaller than a certain threshold, the bonus reward will consider the number of states explored. In the experiments, the authors test different models on two MuJoCo tasks and five Atari games. TRPO, TRPO-Hash, VIME are selected as baselines to compare with. Results show that the proposed bonus reward reaches faster convergence and the highest return in both MuJoCo tasks. In those five Atari games, the proposed method achieves the highest or second-highest average returns.
Although the paper is generally easy to follow and the motivations of the equations are clear,  the analysis of the results is missing and thus the paper provides very limited insights on the behavior of the proposed method. As a result, my opinion on this paper leans to a rejection. As ICLR recommends paper length to be 8 pages, the authors can and should use the remaining space to give more details. For example, the authors can show the mean reward and number of states in all clusters to see whether the agent is efficiently exploring different clusters. And as the number of clusters K is an important hyper-parameter, the reader will also be curious about the resultant performance of the method with different K.
Some questions:
1) In Eq. (3), you have two hyper-parameters in the bonus reward, and for MuJoCo and Atari games, you are using different settings for the first coefficient. How do you choose the hyper-parameter settings? Do you perform grid search with another environment or a set of environments to determine the hyper-parameters?
3) In the algorithm, the method has to learn new cluster assignments in each iteration. Does it significantly slow down the training of the agent? 
4) In the experiments, the authors compare with other methods on only five Atari games. However, there should be more environments available.  What is the reason for choosing these five games? Unless it is difficult to gather the scores on more environments, I believe the authors shall provide results with more Atari environments.
5) The conclusion (Section 6) is extremely short and it claims that "CRL can outperform other SOTA methods in most cases". However, for the five Atari games, the only game that CRL achieves the best performance among seven methods is Venture, according to Table 1. 
Such a claim will confuse readers as it is not in line with the results.
All in all, I believe this paper can be significantly improved if more details and analyses are provided.