Answers.Body: <p>First, both SAC and PPO are usable for continuous and discrete action spaces. However, in the case of discrete action spaces, SAC cost functions must be <a href="https://towardsdatascience.com/adapting-soft-actor-critic-for-discrete-action-spaces-a20614d4a50a" rel="nofollow noreferrer">previously adapted</a>. As explained in <a href="https://github.com/hill-a/stable-baselines/issues/422" rel="nofollow noreferrer">this Stable Baselines3 issue</a>, its efficient implementation is not an easy task.</p>
  <p>Contrary to your hypotheses, off-policy algorithms as SAC are generally more sample-efficient than on-policy algorithms (i.e. PPO), which are generally sample-inefficient due to the data loss that occurs when updating its policy. However, if you're looking for faster training, PPO is your option.</p>
  <p>Moreover, as explained by <a href="https://julien-vitay.net/deeprl/EntropyRL.html#sec:soft-actor-critic-sac" rel="nofollow noreferrer">Haarnoja et al. (2018)</a>, SAC allows to have a stochastic actor while being more optimal and sample efficient than on-policy methods such as A3C or PPO. It is also less sensible to hyperparameters than all these methods.</p>
  <p>Depending on the problem you are dealing with, <a href="https://www.reddit.com/r/reinforcementlearning/comments/gl1ov2/why_would_anyone_use_ppo_over_sac_td3_ddpg_and/" rel="nofollow noreferrer">this Reddit thread</a> may provide you further indications. Finally, here you have <a href="https://medium.datadriveninvestor.com/which-reinforcement-learning-rl-algorithm-to-use-where-when-and-in-what-scenario-e3e7617fb0b1" rel="nofollow noreferrer">a post</a> with some additional information about DRL algorithms comparison.</p>
  <p><strong>In conclusion</strong>: for a generic problem I would recommend using SAC as it has shown better performance in most problems. However, if in your case performance is not an aspect to evaluate and you value more the speed of training and lightness of implementation, I recommend PPO. Another option could be to use both and compare their performance on your specific problem, which could also be quite interesting. Finally, if you are open to more alternatives, don't rule out TD3 if none of the above convinces you!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T16:30:24.113Z
Answers.Text: Actually, the SAC cost functions do not need to be adapted as per the linked article. You can obtain differentiator samples from a discrete distribution using the Gumbell Softmax trick.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T00:06:26.163Z
Answers.Text: @Antonio I have read both the Medium post you linked and the article https://arxiv.org/pdf/1910.07207.pdf, explaining how to adapt SAC to discrete action settings and it seems quite straightforward. The only modification to the policy, actor and temperature cost functions is that, for the expectation over policy actions, we can now calculate it exactly as we have a discrete probability distribution (while before we had to approximate it with samples). So I don't know why in the Stable Baselines3 issue says the discrete SAC implementation is not trivial...
Answers.UserDisplayName: Aeryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T00:13:01.930Z
Answers.Text: @DavidIreland It's true that the Gumbell-softmax trick allows the gradients to flow across stochastic nodes (i.e., a node where a discrete probability distribution is sampled) but, related to my previous comment, it seems like there is no point in doing that. Instead, if we simply modify the cost functions (for Discrete SAC) we are now able to exactly calculate the expectation over policy actions. This reduces the variance in the estimation of the policy objective. This is explained in arxiv.org/pdf/1910.07207.pdf (paragraph on top of Equation 10).
Answers.UserDisplayName: Aeryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T10:11:18.460Z
Answers.Text: There is a point. For instance, if you have both continuous and discrete actions then it is natural to sample from your discrete actions using the Gumbell-softmax trick without making any of the aforementioned modifications. If you have only discrete actions then it is not advisable to use SAC when the DQN and it's variants perform so well.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T14:06:25.030Z
Answers.Text: @DavidIreland I understand your point. However, even if only discrete actions are used, I think SAC is still a sensible alternative to DQN. The authors of the Discrete SAC paper (arxiv.org/pdf/1910.07207.pdf) show that the performance of discrete-SAC is very similar to that of Rainbow, even without tuning the SAC hyperparameters. That along with the fact that implementing discrete-SAC seems simpler than implementing Rainbow or other complex DQN variants seems like a good reason for using SAC over DQN (at least in some scenarios).
Answers.UserDisplayName: Aeryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T20:34:42.433Z
Answers.Text: @Aeryan , I agree, as you pointed out it might be the necessary modification of the entropy estimation in the implementation + some details which make the adaptation not straightforward but still quite intuitive?
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-28T07:33:33.470Z
Answers.LastActivityDate: 2022-06-28T07:33:33.470Z
Answers.UserDisplayName: Antonio
Body: <p>I am currently using <strong>Proximal Policy Optimization (PPO)</strong> to solve my RL task. However, after reading about <strong>Soft Actor-Critic (SAC)</strong> now I am unsure whether I should stick to PPO or switch to SAC. Moreover, from this <a href="https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html" rel="nofollow noreferrer">post</a>, it seems that much of the performance in the original PPO paper comes from code optimizations and not the novel clipped objective.</p>
  <p>The main characteristics of my RL task are the following:</p>
  <ul>
  <li>The action space is <strong>discrete</strong>. SAC was originally designed for continuous action spaces but, if I'm not wrong, it can be adapted to discrete action spaces with no problem.</li>
  <li>I am trying to learn a policy for <strong>generating synthetic data</strong> (i.e., generating novel graphs), so diversity is key. For this reason, I want to learn a policy with as much <strong>entropy</strong> as possible (while still solving the task). Both PPO and SAC try to maximize the policy entropy.</li>
  <li>Obtaining trajectories to train the policy is very <strong>expensive</strong>. My algorithm spends much more time obtaining the trajectories than training the deep neural network of the policy. Here, I think SAC is the clear winner, as it is off-policy whereas PPO is on-policy. Still, PPO is <em>supposed</em> to be very sample-efficient.</li>
  </ul>
  <p>Given my current needs, do you think <strong>it is worth it to switch to SAC instead of PPO?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-29T22:29:19.173Z
Comments.Text: Rather than writing a very generic title like "PPO vs SAC for discrete action spaces", please, put your **specific question** in the title! Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-01T14:19:24.143Z
Comments.Text: Thanks for the comment. Do you think something like *Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?* would do or is it too long?
Comments.UserDisplayName: Aeryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-01T16:15:23.000Z
Comments.Text: I don't think it's too long, but I don't know it's accurate because I didn't fully read your post. If it's accurate, it seems good ;)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T14:54:44.643Z
Comments.Text: Okay, just changed the title ^^
Comments.UserDisplayName: Aeryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-30T20:29:37.073Z
Comments.Text: PPO does not solve a maximum entropy objective as far as I know.
Comments.UserDisplayName: Rémy Hosseinkhan Boucher
Tags: <reinforcement-learning><comparison><proximal-policy-optimization><soft-actor-critic>
Title: Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
UserDisplayName: Aeryan
ThreadText: # Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
  
  (Asked by: username_0 on 2022-06-27)
  
  I am currently using **Proximal Policy Optimization (PPO)** to solve my RL task. However, after reading about **Soft Actor-Critic (SAC)** now I am unsure whether I should stick to PPO or switch to SAC. Moreover, from this [post](https://vitalab.github.io/article/2020/01/14/Implementation_Matters.html), it seems that much of the performance in the original PPO paper comes from code optimizations and not the novel clipped objective.
  
  The main characteristics of my RL task are the following:
  
  * The action space is **discrete**. SAC was originally designed for continuous action spaces but, if I'm not wrong, it can be adapted to discrete action spaces with no problem.
  * I am trying to learn a policy for **generating synthetic data** (i.e., generating novel graphs), so diversity is key. For this reason, I want to learn a policy with as much **entropy** as possible (while still solving the task). Both PPO and SAC try to maximize the policy entropy.
  * Obtaining trajectories to train the policy is very **expensive**. My algorithm spends much more time obtaining the trajectories than training the deep neural network of the policy. Here, I think SAC is the clear winner, as it is off-policy whereas PPO is on-policy. Still, PPO is *supposed* to be very sample-efficient.
  
  Given my current needs, do you think **it is worth it to switch to SAC instead of PPO?**
  
  **Tags:** reinforcement-learning, comparison, proximal-policy-optimization, soft-actor-critic
  
  > **Comments:**
  > **username_1**: Rather than writing a very generic title like "PPO vs SAC for discrete action spaces", please, put your **specific question** in the title! Thanks.  
  > **username_0**: Thanks for the comment. Do you think something like *Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?* would do or is it too long?  
  > **username_1**: I don't think it's too long, but I don't know it's accurate because I didn't fully read your post. If it's accurate, it seems good ;)  
  > **username_0**: Okay, just changed the title ^^  
  > **username_2**: PPO does not solve a maximum entropy objective as far as I know.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-06-28. Score: 1 | Accepted answer
  
  First, both SAC and PPO are usable for continuous and discrete action spaces. However, in the case of discrete action spaces, SAC cost functions must be [previously adapted](https://towardsdatascience.com/adapting-soft-actor-critic-for-discrete-action-spaces-a20614d4a50a). As explained in [this Stable Baselines3 issue](https://github.com/hill-a/stable-baselines/issues/422), its efficient implementation is not an easy task.
  
  Contrary to your hypotheses, off-policy algorithms as SAC are generally more sample-efficient than on-policy algorithms (i.e. PPO), which are generally sample-inefficient due to the data loss that occurs when updating its policy. However, if you're looking for faster training, PPO is your option.
  
  Moreover, as explained by [Haarnoja et al. (2018)](https://julien-vitay.net/deeprl/EntropyRL.html#sec:soft-actor-critic-sac), SAC allows to have a stochastic actor while being more optimal and sample efficient than on-policy methods such as A3C or PPO. It is also less sensible to hyperparameters than all these methods.
  
  Depending on the problem you are dealing with, [this Reddit thread](https://www.reddit.com/r/reinforcementlearning/comments/gl1ov2/why_would_anyone_use_ppo_over_sac_td3_ddpg_and/) may provide you further indications. Finally, here you have [a post](https://medium.datadriveninvestor.com/which-reinforcement-learning-rl-algorithm-to-use-where-when-and-in-what-scenario-e3e7617fb0b1) with some additional information about DRL algorithms comparison.
  
  **In conclusion**: for a generic problem I would recommend using SAC as it has shown better performance in most problems. However, if in your case performance is not an aspect to evaluate and you value more the speed of training and lightness of implementation, I recommend PPO. Another option could be to use both and compare their performance on your specific problem, which could also be quite interesting. Finally, if you are open to more alternatives, don't rule out TD3 if none of the above convinces you!
  
  > **Comments:**
  > **username_4**: Actually, the SAC cost functions do not need to be adapted as per the linked article. You can obtain differentiator samples from a discrete distribution using the Gumbell Softmax trick.  
  > **username_0**: @username_3 I have read both the Medium post you linked and the article https://arxiv.org/pdf/1910.07207.pdf, explaining how to adapt SAC to discrete action settings and it seems quite straightforward. The only modification to the policy, actor and temperature cost functions is that, for the expectation over policy actions, we can now calculate it exactly as we have a discrete probability distribution (while before we had to approximate it with samples). So I don't know why in the Stable Baselines3 issue says the discrete SAC implementation is not trivial...  
  > **username_0**: @DavidIreland It's true that the Gumbell-softmax trick allows the gradients to flow across stochastic nodes (i.e., a node where a discrete probability distribution is sampled) but, related to my previous comment, it seems like there is no point in doing that. Instead, if we simply modify the cost functions (for Discrete SAC) we are now able to exactly calculate the expectation over policy actions. This reduces the variance in the estimation of the policy objective. This is explained in arxiv.org/pdf/1910.07207.pdf (paragraph on top of Equation 10).  
  > **username_4**: There is a point. For instance, if you have both continuous and discrete actions then it is natural to sample from your discrete actions using the Gumbell-softmax trick without making any of the aforementioned modifications. If you have only discrete actions then it is not advisable to use SAC when the DQN and it's variants perform so well.  
  > **username_0**: @DavidIreland I understand your point. However, even if only discrete actions are used, I think SAC is still a sensible alternative to DQN. The authors of the Discrete SAC paper (arxiv.org/pdf/1910.07207.pdf) show that the performance of discrete-SAC is very similar to that of Rainbow, even without tuning the SAC hyperparameters. That along with the fact that implementing discrete-SAC seems simpler than implementing Rainbow or other complex DQN variants seems like a good reason for using SAC over DQN (at least in some scenarios).  
  > **username_2**: @username_0 , I agree, as you pointed out it might be the necessary modification of the entropy estimation in the implementation + some details which make the adaptation not straightforward but still quite intuitive?  
  
  ---
  
  