Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
 Quality and clarity: It seems that the authors can do a better job to improve the readability of the paper and its conciseness. The current structure of paper seems a bit suboptimal to me. The first 6.5 page of the paper is used to explain the idea of RL with entropy reward and how it can be extended to the case of parametrized value function and policy and then the whole experimental results is packed in only 1 page. I think the paper could be organized in a more balanced way by providing a more detailed description and analysis of the numerical results, especially given the fact that in my opinion this is the main strength of the paper. Finally some of the claims made in this paper is not really justified. For instance "Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods" not true, e.g., look at Deepmind AI recent work: https://arxiv.org/abs/1704.04651. Originality and novelty: I think much of the ideas considered in this paper is already explored in previous work as it is acknowledged in the paper. However some of the techniques such as the way the policy is represented and the way the policy gradient formulation is approximated seems to be novel in the context of Deep RL though again these ideas have been explored in the literature of control and RL extensively. Significance: I think the improvement on baseline in control suites is very impressive the problem is that the details of the implementation of algorithm e.g. architecture of neural network size of memory replay, the schedule the target network is implemented is not sufficiently explained in the paper so it is difficult to evaluate these results. Also the paper only compares with the original version of the baseline algorithms. These algorithms are improved since then and the new more efficient algorithms such as distributional policy gradients and NAF have been developed. So it would help to have a better understanding of this result if the paper compares with the state-of-the-art baselines. Minor: for some reason different algorithms have ran for different number of steps which is a bit confusing. would be great if this is fixed in the future version.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Quality and clarity: It seems that the authors can do a better job to improve the readability of the paper and its conciseness. The current structure of paper seems a bit suboptimal to me. The first 6.5 page of the paper is used to explain the idea of RL with entropy reward and how it can be extended to the case of parametrized value function and policy and then the whole experimental results is packed in only 1 page. I think the paper could be organized in a more balanced way by providing a more detailed description and analysis of the numerical results, especially given the fact that in my opinion this is the main strength of the paper. Finally some of the claims made in this paper is not really justified. For instance "Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods" not true, e.g., look at Deepmind AI recent work: https://arxiv.org/abs/1704.04651. Originality and novelty: I think much of the ideas considered in this paper is already explored in previous work as it is acknowledged in the paper. However some of the techniques such as the way the policy is represented and the way the policy gradient formulation is approximated seems to be novel in the context of Deep RL though again these ideas have been explored in the literature of control and RL extensively. Significance: I think the improvement on baseline in control suites is very impressive the problem is that the details of the implementation of algorithm e.g. architecture of neural network size of memory replay, the schedule the target network is implemented is not sufficiently explained in the paper so it is difficult to evaluate these results. Also the paper only compares with the original version of the baseline algorithms. These algorithms are improved since then and the new more efficient algorithms such as distributional policy gradients and NAF have been developed. So it would help to have a better understanding of this result if the paper compares with the state-of-the-art baselines. Minor: for some reason different algorithms have ran for different number of steps which is a bit confusing. would be great if this is fixed in the future version.
assistant
 Quality and clarity: It seems that the authors can do a better job to improve the readability of the paper and its conciseness. The current structure of paper seems a bit suboptimal to me. The first 6.5 page of the paper is used to explain the idea of RL with entropy reward and how it can be extended to the case of parametrized value function and policy and then the whole experimental results is packed in only 1 page. I think the paper could be organized in a more balanced way by providing a more detailed description and analysis of the numerical results, especially given the fact that in my opinion this is the main strength of the paper. Finally some of the claims made in this paper is not really justified. For instance "Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods" not true, e.g., look at Deepmind AI recent work: https://arxiv.org/abs/1704.04651. Originality and novelty: I think much of the ideas considered in this paper is already explored in previous work as it is acknowledged in the paper. However some of the techniques such as the way the policy is represented and the way the policy gradient formulation is approximated seems to be novel in the context of Deep RL though again these ideas have been explored in the literature of control and RL extensively. Significance: I think the improvement on baseline in control suites is very impressive the problem is that the details of the implementation of algorithm e.g. architecture of neural network size of memory replay, the schedule the target network is implemented is not sufficiently explained in the paper so it is difficult to evaluate these results. Also the paper only compares with the original version of the baseline algorithms. These algorithms are improved since then and the new more efficient algorithms such as distributional policy gradients and NAF have been developed. So it would help to have a better understanding of this result if the paper compares with the state-of-the-art baselines. Minor: for some reason different algorithms have ran for different number of steps which is a bit confusing. would be great if this is fixed in the future version.