CAPACITY-LIMITED REINFORCEMENT LEARNING: APPLICATIONS IN DEEP ACTOR-CRITIC METHODS FOR CONTINUOUS CONTROL
Biological and artificial agents must learn to act optimally in spite of a limited capacity for processing, storing, and attending to information. We formalize this type of bounded rationality in terms of an information-theoretic constraint on the complexity of policies that agents seek to learn. We present the Capacity-Limited Reinforcement Learning (CLRL) objective which defines an optimal policy subject to an information capacity constraint. This objective is optimized by drawing from methods used in rate distortion theory and information theory, and applied to the reinforcement learning setting. Using this objective we implement a novel Capacity-Limited Actor-Critic (CLAC) algorithm and situate it within a broader family of RL algorithms such as the Soft Actor Critic (SAC) and discuss their similarities and differences. Our experiments show that compared to alternative approaches, CLAC offers improvements in generalization between training and modified test environments. This is achieved in the CLAC model while displaying high sample efficiency and minimal requirements for hyper-parameter tuning.
The paper presents a reinforcement learning method that regularizes the objective using the mutual information term. The idea is simple and the paper is easy to follow. However, the novelty is limited since the difference between the proposed method and Soft Actor Critic (SAC) is just adding the entropy term of pi(a) to the objective function if I understand the method correctly. In addition, the intuition of adding the entropy term of pi(a) to the objective is not clearly described. The proposed method is evaluated on continuous control tasks.The results shown in the paper is mixed, and I cannot conclude that the proposed method outperforms SAC. Thus, the benefit of the proposed method is not clearly supported by the experimental results. For the current form of the paper, I give *weak reject* due to the weak support of the experimental results and the unclear motivation of the method. One of my concerns is that the way of estimating the pi(a) which is the marginal distribution of the action. From the current manuscript, I did not fully understand how it is estimated in the proposed method. I think that the accuracy of the estimation of pi(a) is crucial in the proposed method since it is the difference from SAC. A comment on the paper structure is that the connection to the *capacity-limited* objective should be described more explicitly in Section 2.1. Although the *capacity-limited* reminds me of the objective something like mathcal{L} + | C - I(X;Y) | as in [Dupont, 2018], the objective in the proposed method shown in page 2 is mathcal{L} + _x0008_eta I(X;Y). I did not understand why the proposed method is *capacity-limited* until Section 5. I think authors should explicitly mention in Section 2.1 that _x0008_eta is adjusted so as to limit the information capacity. To improve the manuscript, I request authors the following things: - The proposed method can be interpreted as adding the penalizing the entropy of pi(a) to the entropy-regularized RL. I do not fully understand the intuition of penalizing the entropy of pi(a) in the context of RL. Please explain it. - I think some tasks should be performed with longer training. For example, agent should be trained for 1-2 millions steps on Humanoid and Ant tasks. In addition, evaluation with 10 trials are preferable. - Please cite papers that estimate the marginal distribution pi(a) in the same manner. If there is no previous work, please explain the details of estimating pi(a) and how pi(a) is approximated from samples.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
CAPACITY-LIMITED REINFORCEMENT LEARNING: APPLICATIONS IN DEEP ACTOR-CRITIC METHODS FOR CONTINUOUS CONTROL
Biological and artificial agents must learn to act optimally in spite of a limited capacity for processing, storing, and attending to information. We formalize this type of bounded rationality in terms of an information-theoretic constraint on the complexity of policies that agents seek to learn. We present the Capacity-Limited Reinforcement Learning (CLRL) objective which defines an optimal policy subject to an information capacity constraint. This objective is optimized by drawing from methods used in rate distortion theory and information theory, and applied to the reinforcement learning setting. Using this objective we implement a novel Capacity-Limited Actor-Critic (CLAC) algorithm and situate it within a broader family of RL algorithms such as the Soft Actor Critic (SAC) and discuss their similarities and differences. Our experiments show that compared to alternative approaches, CLAC offers improvements in generalization between training and modified test environments. This is achieved in the CLAC model while displaying high sample efficiency and minimal requirements for hyper-parameter tuning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper presents a reinforcement learning method that regularizes the objective using the mutual information term. The idea is simple and the paper is easy to follow. However, the novelty is limited since the difference between the proposed method and Soft Actor Critic (SAC) is just adding the entropy term of pi(a) to the objective function if I understand the method correctly. In addition, the intuition of adding the entropy term of pi(a) to the objective is not clearly described. The proposed method is evaluated on continuous control tasks.The results shown in the paper is mixed, and I cannot conclude that the proposed method outperforms SAC. Thus, the benefit of the proposed method is not clearly supported by the experimental results. For the current form of the paper, I give *weak reject* due to the weak support of the experimental results and the unclear motivation of the method. One of my concerns is that the way of estimating the pi(a) which is the marginal distribution of the action. From the current manuscript, I did not fully understand how it is estimated in the proposed method. I think that the accuracy of the estimation of pi(a) is crucial in the proposed method since it is the difference from SAC. A comment on the paper structure is that the connection to the *capacity-limited* objective should be described more explicitly in Section 2.1. Although the *capacity-limited* reminds me of the objective something like mathcal{L} + | C - I(X;Y) | as in [Dupont, 2018], the objective in the proposed method shown in page 2 is mathcal{L} + _x0008_eta I(X;Y). I did not understand why the proposed method is *capacity-limited* until Section 5. I think authors should explicitly mention in Section 2.1 that _x0008_eta is adjusted so as to limit the information capacity. To improve the manuscript, I request authors the following things: - The proposed method can be interpreted as adding the penalizing the entropy of pi(a) to the entropy-regularized RL. I do not fully understand the intuition of penalizing the entropy of pi(a) in the context of RL. Please explain it. - I think some tasks should be performed with longer training. For example, agent should be trained for 1-2 millions steps on Humanoid and Ant tasks. In addition, evaluation with 10 trials are preferable. - Please cite papers that estimate the marginal distribution pi(a) in the same manner. If there is no previous work, please explain the details of estimating pi(a) and how pi(a) is approximated from samples.
assistant
The paper presents a reinforcement learning method that regularizes the objective using the mutual information term. The idea is simple and the paper is easy to follow. However, the novelty is limited since the difference between the proposed method and Soft Actor Critic (SAC) is just adding the entropy term of pi(a) to the objective function if I understand the method correctly. In addition, the intuition of adding the entropy term of pi(a) to the objective is not clearly described. The proposed method is evaluated on continuous control tasks.The results shown in the paper is mixed, and I cannot conclude that the proposed method outperforms SAC. Thus, the benefit of the proposed method is not clearly supported by the experimental results. For the current form of the paper, I give *weak reject* due to the weak support of the experimental results and the unclear motivation of the method. One of my concerns is that the way of estimating the pi(a) which is the marginal distribution of the action. From the current manuscript, I did not fully understand how it is estimated in the proposed method. I think that the accuracy of the estimation of pi(a) is crucial in the proposed method since it is the difference from SAC. A comment on the paper structure is that the connection to the *capacity-limited* objective should be described more explicitly in Section 2.1. Although the *capacity-limited* reminds me of the objective something like mathcal{L} + | C - I(X;Y) | as in [Dupont, 2018], the objective in the proposed method shown in page 2 is mathcal{L} + _x0008_eta I(X;Y). I did not understand why the proposed method is *capacity-limited* until Section 5. I think authors should explicitly mention in Section 2.1 that _x0008_eta is adjusted so as to limit the information capacity. To improve the manuscript, I request authors the following things: - The proposed method can be interpreted as adding the penalizing the entropy of pi(a) to the entropy-regularized RL. I do not fully understand the intuition of penalizing the entropy of pi(a) in the context of RL. Please explain it. - I think some tasks should be performed with longer training. For example, agent should be trained for 1-2 millions steps on Humanoid and Ant tasks. In addition, evaluation with 10 trials are preferable. - Please cite papers that estimate the marginal distribution pi(a) in the same manner. If there is no previous work, please explain the details of estimating pi(a) and how pi(a) is approximated from samples.