Body: <p>Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, even so, it still uses the double Q trick. So it confuses me that if it doesn't serve the purpose of mitigating the maximum overestimation bias, what is the real intention of this use?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-24T08:09:04.953Z
Comments.Text: Just because the policy is stochastic, it doesn’t mean that we aren’t trying to take a maximum over the Q function.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-24T08:17:22.753Z
Comments.Text: @DavidIreland, thanks for your reply! What you mean is that actually all algorithms have the issue of the maximum overestimation, just differ in degree?
Comments.UserDisplayName: Magi Feeney
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-24T09:30:37.103Z
Comments.Text: More or less. Algorithms like SAC and TD3 use the Q-function as the objective for the policy to maximise, so this problem is inherent even though it may not be explicitly obvious. SAC and TD3 use two critics to avoid overestimation bias. If you sample an action from your and the Q-function gives an overinflated estimate of the returns, then your policy will be updated too optimistically, and so using two Q-functions and taking the minimum over these two for the policy update is a way of trying to control the overestimation bias.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-24T09:31:43.600Z
Comments.Text: Of course, 2 is just chosen to trade-off between computational efficiency and dealing with the problem. You could choose any reasonable number (choosing a high number would give a pessimistic value, which would potentially slow down learning too, but I think has a less dramatic effect than overestimation bias).
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-24T12:39:34.943Z
Comments.Text: @DavidIreland That means there are two sources of overestimation bias, one is from the Q function maximum (like in DDPG and TD3), the other is from the policy maximum? The latter seems like the main stream of such a bias in SAC, since SAC uses a stochastic policy, which chooses action randomly, rather than deterministically.  If this holds, however, can I infer that the overestimation bias of policy maximum is not as severe as the Q function maximum?
Comments.UserDisplayName: Magi Feeney
Tags: <ddpg><off-policy-methods><soft-actor-critic>
Title: Why Soft Actor-Critic (SAC) uses a double Q trick?
UserDisplayName: Magi Feeney
ThreadText: # Why Soft Actor-Critic (SAC) uses a double Q trick?
  
  (Asked by: username_0 on 2022-08-24)
  
  Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, even so, it still uses the double Q trick. So it confuses me that if it doesn't serve the purpose of mitigating the maximum overestimation bias, what is the real intention of this use?
  
  **Tags:** ddpg, off-policy-methods, soft-actor-critic
  > **Comments:**
  > **username_1**: Just because the policy is stochastic, it doesn’t mean that we aren’t trying to take a maximum over the Q function.  
  > **username_0**: @DavidIreland, thanks for your reply! What you mean is that actually all algorithms have the issue of the maximum overestimation, just differ in degree?  
  > **username_1**: More or less. Algorithms like SAC and TD3 use the Q-function as the objective for the policy to maximise, so this problem is inherent even though it may not be explicitly obvious. SAC and TD3 use two critics to avoid overestimation bias. If you sample an action from your and the Q-function gives an overinflated estimate of the returns, then your policy will be updated too optimistically, and so using two Q-functions and taking the minimum over these two for the policy update is a way of trying to control the overestimation bias.  
  > **username_1**: Of course, 2 is just chosen to trade-off between computational efficiency and dealing with the problem. You could choose any reasonable number (choosing a high number would give a pessimistic value, which would potentially slow down learning too, but I think has a less dramatic effect than overestimation bias).  
  > **username_0**: @DavidIreland That means there are two sources of overestimation bias, one is from the Q function maximum (like in DDPG and TD3), the other is from the policy maximum? The latter seems like the main stream of such a bias in SAC, since SAC uses a stochastic policy, which chooses action randomly, rather than deterministically.  If this holds, however, can I infer that the overestimation bias of policy maximum is not as severe as the Q function maximum?  
  
  ---
  
  ## 0 answers
  
  