Answers.Body: <p>In the answer I'll be using notation similar to the one from the SAC paper.
  If we look at the standard objective function for policy gradient methods we have
  <span class="math-container">\begin{align}
  J_\pi &amp;= V_\pi(s_t)\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) +  \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\
  &amp;\cdots\\
  &amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)]
  \end{align}</span>
  If you keep unwinding this <span class="math-container">$V(s_{t+i})$</span> you will get expected sum of rewards.
  We can define soft state value as
  <span class="math-container">\begin{align}
  V(s_t) &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]
  \end{align}</span>
  third equality comes from the fact that <span class="math-container">$\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$</span> is nonrandom so its the same thing as if we are sampling over <span class="math-container">$\pi$</span> only once.</p>
  
  <p>In maximum entropy framework objective function would then be 
  <span class="math-container">\begin{align}
  J_\pi &amp;= V_\pi(s_t)\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\
  &amp;= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\
  &amp; \cdots\\
  &amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\
  &amp;= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)]
  \end{align}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-28T15:14:16.873Z
Answers.LastActivityDate: 2019-11-29T08:26:48.027Z
Answers.LastEditDate: 2019-11-29T08:26:48.027Z
Answers.UserDisplayName: Brale
Body: <p>I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. From <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">SpinningUp-SAC</a>, it looks like the entropy is entering through the value-function, so I'm thinking it enters by the <span class="math-container">$\log \pi_{\phi}(a_t \mid s_t)$</span> in the value function?</p>
  
  <p>I'm a little stuck on understanding SAC, can anyone confirm/explain this to me?</p>
  
  <p>Also, side-note question: is being a soft agent equivalent to including entropy in one of the object functions? </p>
  
Tags: <reinforcement-learning><actor-critic-methods><soft-actor-critic>
Title: Where does entropy enter in Soft Actor-Critic?
UserDisplayName: Vildemort
ThreadText: # Where does entropy enter in Soft Actor-Critic?
  
  (Asked by: username_0 on 2019-11-28)
  
  I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. From [SpinningUp-SAC](https://spinningup.openai.com/en/latest/algorithms/sac.html), it looks like the entropy is entering through the value-function, so I'm thinking it enters by the $\log \pi\_{\phi}(a\_t \mid s\_t)$ in the value function?
  
  I'm a little stuck on understanding SAC, can anyone confirm/explain this to me?
  
  Also, side-note question: is being a soft agent equivalent to including entropy in one of the object functions?
  
  **Tags:** reinforcement-learning, actor-critic-methods, soft-actor-critic
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-28. Score: 4 | Accepted answer
  
  In the answer I'll be using notation similar to the one from the SAC paper.
  If we look at the standard objective function for policy gradient methods we have
  \begin{align}
  J\_\pi &= V\_\pi(s\_t)\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[Q(s\_t, a\_t)]\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[ \mathbb E\_{s\_{t+1} \sim p(s|s\_t, a\_t)} [r(s\_t, a\_t) + V(s\_{t+1})]]\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[ \mathbb E\_{s\_{t+1} \sim p(s|s\_t, a\_t)} [r(s\_t, a\_t) + \mathbb E\_{a\_{t+1} \sim \pi(a|s\_{t+1})}[ \mathbb E\_{s\_{t+2} \sim p(s|s\_{t+1}, a\_{t+1})} [r(s\_{t+1}, a\_{t+1}) + V(s\_{t+2})]]]]\\
  &\cdots\\
  &= \sum\_t \mathbb E\_{(a\_t, s\_t) \sim \rho\_\pi} [r( s\_t, a\_t)]
  \end{align}
  If you keep unwinding this $V(s\_{t+i})$ you will get expected sum of rewards.
  We can define soft state value as
  \begin{align}
  V(s\_t) &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[Q(s\_t, a\_t) + \mathcal H(\cdot|s\_t)]\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[Q(s\_t, a\_t) + \mathbb E\_{a \sim \pi(a|s\_t)}[-\log(\pi(a|s\_t))]]\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[Q(s\_t, a\_t) - \log(\pi(a\_t|s\_t))]
  \end{align}
  third equality comes from the fact that $\mathbb E\_{a \sim \pi(a|s\_t)}[-\log(\pi(a|s\_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.
  
  In maximum entropy framework objective function would then be
  \begin{align}
  J\_\pi &= V\_\pi(s\_t)\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[Q(s\_t, a\_t) - \log(\pi(a\_t|s\_t))]\\
  &= \mathbb E\_{a\_t \sim \pi(a|s\_t)}[ \mathbb E\_{s\_{t+1} \sim p(s|s\_t, a\_t)} [r(s\_t, a\_t) - \log(\pi(a\_t|s\_t)) + V(s\_{t+1})]]\\
  & \cdots\\
  &= \sum\_t \mathbb E\_{(a\_t, s\_t) \sim \rho\_\pi} [r(s\_t, a\_t) -\log(\pi(a\_t|s\_t))]\\
  &= \sum\_t \mathbb E\_{(a\_t, s\_t) \sim \rho\_\pi} [r(s\_t, a\_t) + \mathcal H(\cdot|s\_t)]
  \end{align}
  
  
  ---
  
  