Representation learning in reinforcement learning (RL) algorithms focuses on extracting useful features for choosing good actions. Expressive representations are essential for learning well-performed policies. In this paper, we study the relationship between the state representation assigned by the state extractor and the performance of the RL agent. We observe that representations assigned by the better state extractor are more scattered than which assigned by the worse one. Moreover, RL agents achieving high performances always have high rank matrices which are composed by their representations. Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations. Therefore, we propose to promote expressiveness so as to improve algorithm performances, and we call it Expressiveness Promoted DRL. We apply our method on both policy gradient and value-based algorithms, and experimental results on 55 Atari games show the superiority of our proposed method.
The paper aims at characterizing and discussing the impact of the expressivity of a state representation on the performance. It then discusses three possible ways of improving performance with a specific regularization on the state representation. Overall the idea 
of studying the impact of expressiveness and comparing different ways of ensuring it is interesting. However, some parts in the paper are not well-supported and many important elements to understand the experiments are lacking.
Some elements are not well supported and probably not true as stated.
"For fixed h, the expressiveness is related to the MDP M: 1) if the transition p(xt+1|xt, at)π(at|xt) of the MDP is not ergodic, i.e., it can only visit a subset of observations, then the matrix {h(X1 ), · · · , h(Xt ), · · · } will more possibly be low rank; 2) if the reward is very sparse, the representation matrix will be low rank."
If you compare two MDPs, where the first one has two states and is ergodic, while the second one has many more states but is not ergodic, do you think your consideration still applies? For a given h, why would the sparsity of the reward function plays any role?
The paper states that the "method can be widely applied to current DRL algorithms, with little extra computational cost." However, it is not explained clearly how the regularization is enforced. If a supplementary loss is used and minimized at each step, the computational cost is not negligeable.
In the experiment section, the protocol is not clear. For instance the coefficient \alpha is apparently set differently for some of the games? How are they chosen? And why are the hyper-parameters chosen not given?
"However, as the observation type and the intrinsic reward mechanism vary considerably for each game, we also use some other regularization term and coefficient for some of games."
Why are there no details related to the NN architecture?
In Table 1, what does it mean "Times Better" and "Times Worse"?
Other comments:
- In Definition 1, what does it mean X_t \sim \mathcal M? \mathcal M is an MDP, not a distribution. Are the X_t taken following a given policy? are they taken on a given trajectory sequentially or i.i.d.?
- (minor) "From the above description, it is easy to see that the performance of an RL agent depends on two parts. First, it depends on whether the state extractor is good. With a good state extractor, the representation which is a depiction of the observation will retain necessary information for taking actions. Second, it depends on the accuracy of the policy: whether the feed-forward model can correctly take the optimal action given the state." Do the two elements in that paragraph mean the same: "A good state extractor provides an abstract representation from which a performant policy can be obtained?"
- Figure 2: The name of the ATARI game is not mentioned.
- In section 2.2, the MDP framework is introduced (with a state space \mathcal S) but there is no mention of the concept of observation x that is used throughout afterwards.
- In the conclusion, it is stated that "Experiments of A3C and DQN on 55 Atari games demonstrate that ExP DRL can promote their performances significantly." However, the algorithm is not tested on 55 games with DQN.
- The related work discusses papers about state representation but even more directly related to this paper, other papers have also discussed the importance of disentangled representation or entropy maximization for deep RL: https://arxiv.org/abs/1707.08475, https://arxiv.org/abs/1809.04506, ... And papers that discuss expressiveness in deep learning such as https://arxiv.org/pdf/1606.05336.pdf should also be discussed.
- There are many typos/english errors.