Expressiveness in Deep Reinforcement Learning
Representation learning in reinforcement learning (RL) algorithms focuses on extracting useful features for choosing good actions. Expressive representations are essential for learning well-performed policies. In this paper, we study the relationship between the state representation assigned by the state extractor and the performance of the RL agent. We observe that representations assigned by the better state extractor are more scattered than which assigned by the worse one. Moreover, RL agents achieving high performances always have high rank matrices which are composed by their representations. Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations. Therefore, we propose to promote expressiveness so as to improve algorithm performances, and we call it Expressiveness Promoted DRL. We apply our method on both policy gradient and value-based algorithms, and experimental results on 55 Atari games show the superiority of our proposed method.
 # Summary This paper proposes a simple regularizer for RL which encourages the state representations learned by neural networks to be more discriminative across different observations. The main idea is to (implicitly) measure the rank of the matrix which is constructed from a sequence of observations and state feature vectors and encourage the rank to be high. This paper introduces three different objectives to implement the same idea (increasing the rank of the matrix). The experimental results on Atari games show that this regularizer improves A3C on most of the games and show that the learned representations with the proposed regularizer has a high rank compared to the baseline. [Pros] - Makes an interesting point about the correlation between RL performance and state representations. - Proposes a simple objective function that gives strong empirical results. [Cons] - Needs more empirical evidences to better support the main hypothesis of the paper. # Novelty and Significance - This paper makes an interesting observation about the correlation between the RL performance and the how discriminative the learned features. - To verify it, this paper proposes a new and simple regularizer which improves the performance across many Atari games. # Quality and Experiment - The main hypothesis of this paper is that the expressiveness of the features (specifically the rank of the matrix that consists of a sequence of features) and its RL performance is highly correlated. Although this paper showed some plots (Figure 2, 6) to verify this, a more extensive statistical test or experiments would be more convincing to show the hypothesis. Examples would be: 1) Measuring the correlation between the two across all Atari games. 2) An ablation study on the hyperparameter (alpha). 3) Learning state representations just from the reconstruction task (without RL) with/without the proposed regularizer and separately learning policies on top of that (with fixed representations). It would be much more convincing if the regularizer helps even in this setup, because this would show the general effect of the expressiveness term (by removing the effect of RL algorithm on the representation). - This paper only reports "relative" performances to the baseline. Though it looks strong, it is also important to report the absolute performances in Atari games (e.g., median human-normalized score, etc) to show how significant the performance gap is. - The results with DQN are not convincing because the agents are trained only for 20M frames (compared to 200M frames in many other papers). It is not much meaningful to compare performances on such a short training regime. I would suggest running longer or removing this result from the paper and focusing on more analysis. # Clarity and Presentation - Figure 1a is not much insightful. It is not surprising that the representations that led to a poor policy (which achieves 0 reward) are much less discriminative given five situations with five distinct optimal actions, because the the policy has no idea which action is better than the other in such situations. It would be more informative to pick N consecutive frames and show how scattered they are in the embedding space.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Expressiveness in Deep Reinforcement Learning
Representation learning in reinforcement learning (RL) algorithms focuses on extracting useful features for choosing good actions. Expressive representations are essential for learning well-performed policies. In this paper, we study the relationship between the state representation assigned by the state extractor and the performance of the RL agent. We observe that representations assigned by the better state extractor are more scattered than which assigned by the worse one. Moreover, RL agents achieving high performances always have high rank matrices which are composed by their representations. Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations. Therefore, we propose to promote expressiveness so as to improve algorithm performances, and we call it Expressiveness Promoted DRL. We apply our method on both policy gradient and value-based algorithms, and experimental results on 55 Atari games show the superiority of our proposed method.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 # Summary This paper proposes a simple regularizer for RL which encourages the state representations learned by neural networks to be more discriminative across different observations. The main idea is to (implicitly) measure the rank of the matrix which is constructed from a sequence of observations and state feature vectors and encourage the rank to be high. This paper introduces three different objectives to implement the same idea (increasing the rank of the matrix). The experimental results on Atari games show that this regularizer improves A3C on most of the games and show that the learned representations with the proposed regularizer has a high rank compared to the baseline. [Pros] - Makes an interesting point about the correlation between RL performance and state representations. - Proposes a simple objective function that gives strong empirical results. [Cons] - Needs more empirical evidences to better support the main hypothesis of the paper. # Novelty and Significance - This paper makes an interesting observation about the correlation between the RL performance and the how discriminative the learned features. - To verify it, this paper proposes a new and simple regularizer which improves the performance across many Atari games. # Quality and Experiment - The main hypothesis of this paper is that the expressiveness of the features (specifically the rank of the matrix that consists of a sequence of features) and its RL performance is highly correlated. Although this paper showed some plots (Figure 2, 6) to verify this, a more extensive statistical test or experiments would be more convincing to show the hypothesis. Examples would be: 1) Measuring the correlation between the two across all Atari games. 2) An ablation study on the hyperparameter (alpha). 3) Learning state representations just from the reconstruction task (without RL) with/without the proposed regularizer and separately learning policies on top of that (with fixed representations). It would be much more convincing if the regularizer helps even in this setup, because this would show the general effect of the expressiveness term (by removing the effect of RL algorithm on the representation). - This paper only reports "relative" performances to the baseline. Though it looks strong, it is also important to report the absolute performances in Atari games (e.g., median human-normalized score, etc) to show how significant the performance gap is. - The results with DQN are not convincing because the agents are trained only for 20M frames (compared to 200M frames in many other papers). It is not much meaningful to compare performances on such a short training regime. I would suggest running longer or removing this result from the paper and focusing on more analysis. # Clarity and Presentation - Figure 1a is not much insightful. It is not surprising that the representations that led to a poor policy (which achieves 0 reward) are much less discriminative given five situations with five distinct optimal actions, because the the policy has no idea which action is better than the other in such situations. It would be more informative to pick N consecutive frames and show how scattered they are in the embedding space.
assistant
 # Summary This paper proposes a simple regularizer for RL which encourages the state representations learned by neural networks to be more discriminative across different observations. The main idea is to (implicitly) measure the rank of the matrix which is constructed from a sequence of observations and state feature vectors and encourage the rank to be high. This paper introduces three different objectives to implement the same idea (increasing the rank of the matrix). The experimental results on Atari games show that this regularizer improves A3C on most of the games and show that the learned representations with the proposed regularizer has a high rank compared to the baseline. [Pros] - Makes an interesting point about the correlation between RL performance and state representations. - Proposes a simple objective function that gives strong empirical results. [Cons] - Needs more empirical evidences to better support the main hypothesis of the paper. # Novelty and Significance - This paper makes an interesting observation about the correlation between the RL performance and the how discriminative the learned features. - To verify it, this paper proposes a new and simple regularizer which improves the performance across many Atari games. # Quality and Experiment - The main hypothesis of this paper is that the expressiveness of the features (specifically the rank of the matrix that consists of a sequence of features) and its RL performance is highly correlated. Although this paper showed some plots (Figure 2, 6) to verify this, a more extensive statistical test or experiments would be more convincing to show the hypothesis. Examples would be: 1) Measuring the correlation between the two across all Atari games. 2) An ablation study on the hyperparameter (alpha). 3) Learning state representations just from the reconstruction task (without RL) with/without the proposed regularizer and separately learning policies on top of that (with fixed representations). It would be much more convincing if the regularizer helps even in this setup, because this would show the general effect of the expressiveness term (by removing the effect of RL algorithm on the representation). - This paper only reports "relative" performances to the baseline. Though it looks strong, it is also important to report the absolute performances in Atari games (e.g., median human-normalized score, etc) to show how significant the performance gap is. - The results with DQN are not convincing because the agents are trained only for 20M frames (compared to 200M frames in many other papers). It is not much meaningful to compare performances on such a short training regime. I would suggest running longer or removing this result from the paper and focusing on more analysis. # Clarity and Presentation - Figure 1a is not much insightful. It is not surprising that the representations that led to a poor policy (which achieves 0 reward) are much less discriminative given five situations with five distinct optimal actions, because the the policy has no idea which action is better than the other in such situations. It would be more informative to pick N consecutive frames and show how scattered they are in the embedding space.