Answers.Body: <p>This relation is from the definition of action value function such as that defined in Sutton's RL book page 58.</p>
  <blockquote>
  <p>we define the value of taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> under a policy <span class="math-container">$\pi$</span>, denoted
  <span class="math-container">$q_{\pi}(s, a)$</span>, as the expected return starting from <span class="math-container">$s$</span>, taking the action <span class="math-container">$a$</span>, and thereafter following policy <span class="math-container">$\pi$</span></p>
  </blockquote>
  <p>Thus in general following your reference's notation we have the obvious equational relation between state and action value functions of the same state as <span class="math-container">$V^{\pi}(s)=Q^{\pi}(s, \pi(s))=\mathbb{E}_{a \sim \pi}[Q^{\pi}(s, a))]$</span>.</p>
  <p>For SAC we need to add an additional policy's entropy bonus from <em>every</em> timestep for both state and action value functions, <em>except</em> the first timestep for action value function as your reference mentions since the entropy bonus is associated with the uncertainty of the policy when choosing among possible actions, but when we condition on having taken a specific action <span class="math-container">$a$</span> for <span class="math-container">$Q^{\pi}(s, a)$</span>, we no longer consider the uncertainty of the policy at <span class="math-container">$s$</span> which has already been resolved by taking action <span class="math-container">$a$</span>. Therefore we have to add the additional entropy bonus at state <span class="math-container">$s$</span> at the RHS to balance above equation as <span class="math-container">$V^{\pi}(s)=\mathbb{E}_{a \sim \pi}[Q^{\pi}(s, a))] + \alpha H(\pi(\cdot|s))$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T04:47:45.057Z
Answers.LastActivityDate: 2024-09-08T04:47:45.057Z
Answers.UserDisplayName: cinch
Answers.Body: <p>A great intuitive answer was provided by @cinch; I will provide an answer based on notation conversion. First, note that the sampled trajectory <span class="math-container">$\tau$</span> does not include <span class="math-container">$s_0$</span> because <span class="math-container">$s_0{=}s$</span> by definition of <span class="math-container">$V^\pi(s)$</span>. We write out the first few terms of <span class="math-container">$\tau$</span> for clarity, condense <span class="math-container">$R(s_t, a_t, s_{t+1})$</span> into <span class="math-container">$R_t$</span>, and distribute <span class="math-container">$\gamma^t$</span>:</p>
  <p><span class="math-container">\begin{equation}
  \begin{split}
  V^{\pi}(s) &amp;= \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^\infty \gamma^t \Bigl(R(s_t, a_t, s_{t+1}) + \alpha H(\pi(\cdot|s_t))\Bigr)\ \Big|\ s_0{=}s\right], \hspace{10em}\\
  &amp;= \mathbb{E}_{(a_0, r_0, s_1,\ \ldots) \sim \pi}\left[\sum_{t=0}^\infty \gamma^t \Bigl(R_t + \alpha H(\pi(\cdot|s_t))\Bigr)\ \Big|\ s_0{=}s\right],\\
  &amp;= \mathbb{E}_{(a_0, r_0, s_1,\ \ldots) \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R_t + \sum_{t=0}^\infty \gamma^t\Bigl(\alpha H(\pi(\cdot|s_t))\Bigr)\ \Big|\ s_0{=}s\right].\\
  \end{split}
  \end{equation}</span></p>
  <p>Next, we use the fact that the expectation is not taken over <span class="math-container">$s_0{=}s$</span>, allowing us to change the index of summation and move the entropy term <span class="math-container">$\alpha H(\pi(\cdot|s))$</span> outside of the expectation.</p>
  <p><span class="math-container">\begin{equation}
  \begin{split}
   &amp;= \mathbb{E}_{(a_0, r_0, s_1,\ \ldots) \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R_t + \left(\sum_{t=1}^\infty \gamma^t\Bigr(\alpha H(\pi(\cdot|s_t))\Bigr)\right) + \alpha H(\pi(\cdot|s_0))\ \Big|\ s_0{=}s\right],\hspace{2.5em}\\
  &amp;= \mathbb{E}_{(a_0, r_0, s_1,\ \ldots) \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R_t + \alpha\sum_{t=1}^\infty \gamma^t H(\pi(\cdot|s_t))\ \Big|\ s_0{=}s\right] + \alpha H(\pi(\cdot|s)).\\
  \end{split}
  \end{equation}</span></p>
  <p>We continue by splitting the expectation of <span class="math-container">$\tau = (a_0, r_0, s_1, \ldots)$</span> into an expectation over <span class="math-container">$a_0$</span> and an expectation over <span class="math-container">$(r_0, s_1, \ldots)$</span>. Note that the expectation over <span class="math-container">$a_0$</span> is equivalent to the expectation over <span class="math-container">$a$</span> sampled from <span class="math-container">$\pi(\cdot|s)$</span> because <span class="math-container">$s_0{=}s$</span>. Finally, we simplify using the definition of <span class="math-container">$Q^\pi(s, a)$</span>:</p>
  <p><span class="math-container">\begin{equation}
  \begin{split}
  &amp;= \mathbb{E}_{a_0 \sim \pi(\cdot | s_0)}\left[\mathbb{E}_{(r_0, s_1,\ \ldots) \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R_t + \alpha\sum_{t=1}^\infty \gamma^t H(\pi(\cdot|s_t))\ \Big|\ s_0{=}s, a_0\right]\right] + \alpha H(\pi(\cdot|s)),\\
  &amp;= \mathbb{E}_{a \sim \pi(\cdot | s)}\left[\mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R_t + \alpha\sum_{t=1}^\infty \gamma^t H(\pi(\cdot|s_t))\ \Big|\ s_0{=}s, a_0{=}a\right]\right] + \alpha H(\pi(\cdot|s)),\hspace{-2.5em}\\
  &amp;= \mathbb{E}_{a \sim \pi(\cdot | s)}\left[Q^\pi(s, a)\right] + \alpha H(\pi(\cdot|s)),\\
  \end{split}
  \end{equation}</span></p>
  <p>as desired.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-10T19:22:35.663Z
Answers.LastActivityDate: 2024-09-10T19:22:35.663Z
Answers.UserDisplayName: DeepQZero
Body: <p>Below is the relationship between state-value function and action-value function based on the source: <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">https://spinningup.openai.com/en/latest/algorithms/sac.html</a></p>
  <hr />
  <p><span class="math-container">$$H(P) = \mathbb{E}_{x \sim P}[-\log P(x)].$$</span></p>
  <p>In entropy-regularized reinforcement learning, the agent gets a bonus reward at each time step proportional to the entropy of the policy at that time step. This changes the RL problem to:</p>
  <p><span class="math-container">$$\pi^* = \arg \max_{\pi} \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^\infty \gamma^t\Bigl(R(s_t, a_t, s_{t+1}) + \alpha H (\pi(\cdot | s_t))\Bigr)\right],$$</span></p>
  <p>where <span class="math-container">$\alpha &gt; 0$</span> is the trade-off coefficient. (Note: we’re assuming an infinite-horizon discounted setting here, and we’ll do the same for the rest of this page.) We can now define the slightly-different value functions in this setting. <span class="math-container">$V^{\pi}$</span> is changed to include the entropy bonuses from every time step:</p>
  <p><span class="math-container">$$V^{\pi}(s) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^\infty \gamma^t \Bigl(R(s_t, a_t, s_{t+1}) + \alpha H(\pi(\cdot|s_t))\Bigr)\ \Big|\ s_0{=}s\right].$$</span></p>
  <p><span class="math-container">$Q^{\pi}$</span> is changed to include the entropy bonuses from every time step except the first:</p>
  <p><span class="math-container">$$Q^{\pi}(s,a) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^\infty \gamma^t R(s_t, a_t, s_{t+1}) + \alpha\sum_{t=1}^\infty \gamma^t H(\pi(\cdot|s_t))\ \Big|\ s_0{=}s, a_0{=}a\right].$$</span></p>
  <p>With these definitions, <span class="math-container">$V^{\pi}$</span> and <span class="math-container">$Q^{\pi}$</span> are connected by:</p>
  <p><span class="math-container">$$V^{\pi}(s) = \mathbb{E}_{a \sim \pi}\left[Q^\pi(s, a)\right] + \alpha H(\pi(\cdot|s)).$$</span></p>
  <hr />
  <p><strong>Question</strong>: How is the derivation of the relationship between action-value function and state-value function given in the last line?</p>
  
Tags: <reinforcement-learning><soft-actor-critic>
Title: Derivation of the relationship between state-value function and action-value function in SAC
UserDisplayName: DSPinfinity
ThreadText: # Derivation of the relationship between state-value function and action-value function in SAC
  
  (Asked by: username_0 on 2024-09-08)
  
  Below is the relationship between state-value function and action-value function based on the source: <https://spinningup.openai.com/en/latest/algorithms/sac.html>
  
  ---
  
  $$H(P) = \mathbb{E}\_{x \sim P}[-\log P(x)].$$
  
  In entropy-regularized reinforcement learning, the agent gets a bonus reward at each time step proportional to the entropy of the policy at that time step. This changes the RL problem to:
  
  $$\pi^\* = \arg \max\_{\pi} \mathbb{E}\_{\tau \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t\Bigl(R(s\_t, a\_t, s\_{t+1}) + \alpha H (\pi(\cdot | s\_t))\Bigr)\right],$$
  
  where $\alpha > 0$ is the trade-off coefficient. (Note: we’re assuming an infinite-horizon discounted setting here, and we’ll do the same for the rest of this page.) We can now define the slightly-different value functions in this setting. $V^{\pi}$ is changed to include the entropy bonuses from every time step:
  
  $$V^{\pi}(s) = \mathbb{E}\_{\tau \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t \Bigl(R(s\_t, a\_t, s\_{t+1}) + \alpha H(\pi(\cdot|s\_t))\Bigr)\ \Big|\ s\_0{=}s\right].$$
  
  $Q^{\pi}$ is changed to include the entropy bonuses from every time step except the first:
  
  $$Q^{\pi}(s,a) = \mathbb{E}\_{\tau \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R(s\_t, a\_t, s\_{t+1}) + \alpha\sum\_{t=1}^\infty \gamma^t H(\pi(\cdot|s\_t))\ \Big|\ s\_0{=}s, a\_0{=}a\right].$$
  
  With these definitions, $V^{\pi}$ and $Q^{\pi}$ are connected by:
  
  $$V^{\pi}(s) = \mathbb{E}\_{a \sim \pi}\left[Q^\pi(s, a)\right] + \alpha H(\pi(\cdot|s)).$$
  
  ---
  
  **Question**: How is the derivation of the relationship between action-value function and state-value function given in the last line?
  
  **Tags:** reinforcement-learning, soft-actor-critic
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-08. Score: 2 | Accepted answer
  
  This relation is from the definition of action value function such as that defined in Sutton's RL book page 58.
  
  > we define the value of taking action $a$ in state $s$ under a policy $\pi$, denoted
  > $q\_{\pi}(s, a)$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$
  
  Thus in general following your reference's notation we have the obvious equational relation between state and action value functions of the same state as $V^{\pi}(s)=Q^{\pi}(s, \pi(s))=\mathbb{E}\_{a \sim \pi}[Q^{\pi}(s, a))]$.
  
  For SAC we need to add an additional policy's entropy bonus from *every* timestep for both state and action value functions, *except* the first timestep for action value function as your reference mentions since the entropy bonus is associated with the uncertainty of the policy when choosing among possible actions, but when we condition on having taken a specific action $a$ for $Q^{\pi}(s, a)$, we no longer consider the uncertainty of the policy at $s$ which has already been resolved by taking action $a$. Therefore we have to add the additional entropy bonus at state $s$ at the RHS to balance above equation as $V^{\pi}(s)=\mathbb{E}\_{a \sim \pi}[Q^{\pi}(s, a))] + \alpha H(\pi(\cdot|s))$.
  
  
  ---
  
  ### Answer by username_2 on 2024-09-10. Score: 0
  
  A great intuitive answer was provided by @cinch; I will provide an answer based on notation conversion. First, note that the sampled trajectory $\tau$ does not include $s\_0$ because $s\_0{=}s$ by definition of $V^\pi(s)$. We write out the first few terms of $\tau$ for clarity, condense $R(s\_t, a\_t, s\_{t+1})$ into $R\_t$, and distribute $\gamma^t$:
  
  \begin{equation}
  \begin{split}
  V^{\pi}(s) &= \mathbb{E}\_{\tau \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t \Bigl(R(s\_t, a\_t, s\_{t+1}) + \alpha H(\pi(\cdot|s\_t))\Bigr)\ \Big|\ s\_0{=}s\right], \hspace{10em}\\
  &= \mathbb{E}\_{(a\_0, r\_0, s\_1,\ \ldots) \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t \Bigl(R\_t + \alpha H(\pi(\cdot|s\_t))\Bigr)\ \Big|\ s\_0{=}s\right],\\
  &= \mathbb{E}\_{(a\_0, r\_0, s\_1,\ \ldots) \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R\_t + \sum\_{t=0}^\infty \gamma^t\Bigl(\alpha H(\pi(\cdot|s\_t))\Bigr)\ \Big|\ s\_0{=}s\right].\\
  \end{split}
  \end{equation}
  
  Next, we use the fact that the expectation is not taken over $s\_0{=}s$, allowing us to change the index of summation and move the entropy term $\alpha H(\pi(\cdot|s))$ outside of the expectation.
  
  \begin{equation}
  \begin{split}
  &= \mathbb{E}\_{(a\_0, r\_0, s\_1,\ \ldots) \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R\_t + \left(\sum\_{t=1}^\infty \gamma^t\Bigr(\alpha H(\pi(\cdot|s\_t))\Bigr)\right) + \alpha H(\pi(\cdot|s\_0))\ \Big|\ s\_0{=}s\right],\hspace{2.5em}\\
  &= \mathbb{E}\_{(a\_0, r\_0, s\_1,\ \ldots) \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R\_t + \alpha\sum\_{t=1}^\infty \gamma^t H(\pi(\cdot|s\_t))\ \Big|\ s\_0{=}s\right] + \alpha H(\pi(\cdot|s)).\\
  \end{split}
  \end{equation}
  
  We continue by splitting the expectation of $\tau = (a\_0, r\_0, s\_1, \ldots)$ into an expectation over $a\_0$ and an expectation over $(r\_0, s\_1, \ldots)$. Note that the expectation over $a\_0$ is equivalent to the expectation over $a$ sampled from $\pi(\cdot|s)$ because $s\_0{=}s$. Finally, we simplify using the definition of $Q^\pi(s, a)$:
  
  \begin{equation}
  \begin{split}
  &= \mathbb{E}\_{a\_0 \sim \pi(\cdot | s\_0)}\left[\mathbb{E}\_{(r\_0, s\_1,\ \ldots) \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R\_t + \alpha\sum\_{t=1}^\infty \gamma^t H(\pi(\cdot|s\_t))\ \Big|\ s\_0{=}s, a\_0\right]\right] + \alpha H(\pi(\cdot|s)),\\
  &= \mathbb{E}\_{a \sim \pi(\cdot | s)}\left[\mathbb{E}\_{\tau \sim \pi}\left[\sum\_{t=0}^\infty \gamma^t R\_t + \alpha\sum\_{t=1}^\infty \gamma^t H(\pi(\cdot|s\_t))\ \Big|\ s\_0{=}s, a\_0{=}a\right]\right] + \alpha H(\pi(\cdot|s)),\hspace{-2.5em}\\
  &= \mathbb{E}\_{a \sim \pi(\cdot | s)}\left[Q^\pi(s, a)\right] + \alpha H(\pi(\cdot|s)),\\
  \end{split}
  \end{equation}
  
  as desired.
  
  
  ---
  
  