Body: <p>While reading the original paper of <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="nofollow noreferrer">Soft Actor Critic</a>, I came across on page number 5, under equation (5) and (6)</p>
  <p><span class="math-container">$$
  J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\psi}\left(\mathbf{s}_{t}\right)-\mathbb{E}_{\mathbf{a}_{t} \sim \pi_{\phi}}\left[Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)-\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right]\right)^{2}\right]
  \tag{5}\label{5}
  $$</span></p>
  <p><span class="math-container">$$
  \hat{\nabla}_{\psi} J_{V}(\psi)=\nabla_{\psi} V_{\psi}\left(\mathbf{s}_{t}\right)\left(V_{\psi}\left(\mathbf{s}_{t}\right)-Q_{\theta}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\log \pi_{\phi}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right)
  \tag{6}\label{6}
  $$</span></p>
  <p>The following quote:</p>
  <blockquote>
  <p>where the actions are sampled according to the current policy, instead of the replay buffer</p>
  </blockquote>
  <p>In the context of deriving the formulation of the (estimated) gradient for the <strong>value function square residual error</strong> (Equation 5 in the paper)</p>
  <p>I'm having a hard time understanding <em>why</em> they use the action sampled from the current policy instead of the replay buffer. My intuition tells me that this is because SAC is an <em>off policy</em> Reinforcement Learning algorithm, and Q-learning uses <span class="math-container">$\max Q$</span> in one-step Q-value function update (to keep it off-policy), but why would sampling <em>one</em> action from the <em>current policy</em> still make it <em>off-policy</em>?</p>
  <p>I first asked a friend of mine (researcher in RL) and the answer I got was</p>
  <blockquote>
  <p>&quot;If the action is sampled with the current policy given any state the update is on-policy.&quot;</p>
  </blockquote>
  <p>I've checked <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">SpinningUpRL by OpenAI's explanation of SAC</a> but they only make it more clear <em>which</em> action is sampled from the current policy, and which one is from the replay buffer, but does not specify <em>why</em>.</p>
  <p>Does this have anything to do with the stochastic policy? Or the entropy term in the update equation?</p>
  <p>So I'm still quite confused. Link/references to explanation are also appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-11T14:17:02.323Z
Comments.Text: I was also not able to find an answer on the internet. I think the reason for using the current policy starts with the entropy term in the Q and policy updates. Reading very closely through [this section](https://spinningup.openai.com/en/latest/algorithms/sac.html#id6) of the Spinning Up explanation helped me. Maybe the reason is that the entropy is supposed to be minimized and it would not work to use entropy from the current policy but actions from a different policy. And it does not seem like using entropy from a past policy would make sense.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-11T14:18:04.080Z
Comments.Text: If I get the time, I might try to see if SAC can still learn with using a past policy for the entropy, next actions, or both.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-11T16:51:45.330Z
Comments.Text: Interestingly, SAC learned the Pendulum environment the same with and without a target actor giving the entropy term and the next actions, though it might have learned faster without the target actor. It even learned with the target actor not updating at all because I forgot to add the code in. Maybe there is no target actor just because it isn’t needed.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T05:52:59.447Z
Comments.Text: Hello, thank you for the comment (and also even trying it! :D ) S2673! Just to be sure we are on the same page, by "target actor" you mean the stored action in the replay buffer (from another policy, most often previous one), the a in $(s_t, a_t, r_t, s_{t+1}) \in D$ right?
Comments.UserDisplayName: DannyBoi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T12:48:16.697Z
Comments.Text: $a_t$ was from whatever policy was operating when the experience batch was collected. I used the target actor to get the next-state actions($a_{t+1}$) because that is how DDPG and TD3 work. SAC usually gets $a_{t+1}$ from the current policy.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T12:58:01.843Z
Comments.Text: Just like in TD3 and DDPG, the actor’s parameters were copied to the target actor at the beginning of training and every time the actor updates, the target actor was pushed slightly in the same direction with polyak averaging.  The difference between what I did and normal SAC, is the existence of a target actor and using it to get $a_{t+1}$ and the the log probabilities of the next actions (the log probabilities affect the entropy part of the update equation) during the updates of the Q networks.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-13T08:27:06.653Z
Comments.Text: Ah ok, thank you for the clarification! Hm still have slight hard time wrapping my head around it but I'll keep looking too.
Comments.UserDisplayName: DannyBoi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-13T09:17:11.503Z
Comments.Text: It is a really good question and I hope you get an answer.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-15T01:22:05.857Z
Comments.Text: I added a bounty to see if someone will answer.
Comments.UserDisplayName: S2673
Tags: <reinforcement-learning><policy-gradients><value-functions><off-policy-methods><soft-actor-critic>
Title: In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
UserDisplayName: DannyBoi
ThreadText: # In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
  
  (Asked by: username_0 on 2020-11-11)
  
  While reading the original paper of [Soft Actor Critic](https://arxiv.org/pdf/1801.01290.pdf), I came across on page number 5, under equation (5) and (6)
  
  $$
  J\_{V}(\psi)=\mathbb{E}\_{\mathbf{s}\_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V\_{\psi}\left(\mathbf{s}\_{t}\right)-\mathbb{E}\_{\mathbf{a}\_{t} \sim \pi\_{\phi}}\left[Q\_{\theta}\left(\mathbf{s}\_{t}, \mathbf{a}\_{t}\right)-\log \pi\_{\phi}\left(\mathbf{a}\_{t} \mid \mathbf{s}\_{t}\right)\right]\right)^{2}\right]
  \tag{5}\label{5}
  $$
  
  $$
  \hat{\nabla}\_{\psi} J\_{V}(\psi)=\nabla\_{\psi} V\_{\psi}\left(\mathbf{s}\_{t}\right)\left(V\_{\psi}\left(\mathbf{s}\_{t}\right)-Q\_{\theta}\left(\mathbf{s}\_{t}, \mathbf{a}\_{t}\right)+\log \pi\_{\phi}\left(\mathbf{a}\_{t} \mid \mathbf{s}\_{t}\right)\right)
  \tag{6}\label{6}
  $$
  
  The following quote:
  
  > where the actions are sampled according to the current policy, instead of the replay buffer
  
  In the context of deriving the formulation of the (estimated) gradient for the **value function square residual error** (Equation 5 in the paper)
  
  I'm having a hard time understanding *why* they use the action sampled from the current policy instead of the replay buffer. My intuition tells me that this is because SAC is an *off policy* Reinforcement Learning algorithm, and Q-learning uses $\max Q$ in one-step Q-value function update (to keep it off-policy), but why would sampling *one* action from the *current policy* still make it *off-policy*?
  
  I first asked a friend of mine (researcher in RL) and the answer I got was
  
  > "If the action is sampled with the current policy given any state the update is on-policy."
  
  I've checked [SpinningUpRL by OpenAI's explanation of SAC](https://spinningup.openai.com/en/latest/algorithms/sac.html) but they only make it more clear *which* action is sampled from the current policy, and which one is from the replay buffer, but does not specify *why*.
  
  Does this have anything to do with the stochastic policy? Or the entropy term in the update equation?
  
  So I'm still quite confused. Link/references to explanation are also appreciated!
  
  **Tags:** reinforcement-learning, policy-gradients, value-functions, off-policy-methods, soft-actor-critic
  
  > **Comments:**
  > **username_1**: I was also not able to find an answer on the internet. I think the reason for using the current policy starts with the entropy term in the Q and policy updates. Reading very closely through [this section](https://spinningup.openai.com/en/latest/algorithms/sac.html#id6) of the Spinning Up explanation helped me. Maybe the reason is that the entropy is supposed to be minimized and it would not work to use entropy from the current policy but actions from a different policy. And it does not seem like using entropy from a past policy would make sense.  
  > **username_1**: If I get the time, I might try to see if SAC can still learn with using a past policy for the entropy, next actions, or both.  
  > **username_1**: Interestingly, SAC learned the Pendulum environment the same with and without a target actor giving the entropy term and the next actions, though it might have learned faster without the target actor. It even learned with the target actor not updating at all because I forgot to add the code in. Maybe there is no target actor just because it isn’t needed.  
  > **username_0**: Hello, thank you for the comment (and also even trying it! :D ) username_1! Just to be sure we are on the same page, by "target actor" you mean the stored action in the replay buffer (from another policy, most often previous one), the a in $(s_t, a_t, r_t, s_{t+1}) \in D$ right?  
  > **username_1**: $a_t$ was from whatever policy was operating when the experience batch was collected. I used the target actor to get the next-state actions($a_{t+1}$) because that is how DDPG and TD3 work. SAC usually gets $a_{t+1}$ from the current policy.  
  > **username_1**: Just like in TD3 and DDPG, the actor’s parameters were copied to the target actor at the beginning of training and every time the actor updates, the target actor was pushed slightly in the same direction with polyak averaging.  The difference between what I did and normal SAC, is the existence of a target actor and using it to get $a_{t+1}$ and the the log probabilities of the next actions (the log probabilities affect the entropy part of the update equation) during the updates of the Q networks.  
  > **username_0**: Ah ok, thank you for the clarification! Hm still have slight hard time wrapping my head around it but I'll keep looking too.  
  > **username_1**: It is a really good question and I hope you get an answer.  
  > **username_1**: I added a bounty to see if someone will answer.  
  
  ---
  
  ## 0 answers
  
  