Answers.Body: <blockquote>
  <p>isn't then <span class="math-container">$v_\pi(s)$</span> just the expected action value function at <span class="math-container">$s$</span> over all actions <span class="math-container">$a$</span> that are given by the policy <span class="math-container">$\pi$</span>, namely</p>
  </blockquote>
  <blockquote>
  <p><span class="math-container">$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$</span>?</p>
  </blockquote>
  <p>Yes this is 100% correct.</p>
  <p>There is no &quot;trick&quot; to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between <span class="math-container">$v_{\pi}$</span> and <span class="math-container">$q_{\pi}$</span> in that direction.</p>
  <p>Note that for a deterministic policy, with <span class="math-container">$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$</span> then the relationship is</p>
  <p><span class="math-container">$$v_\pi(s) = q_\pi(s, \pi(s))$$</span></p>
  <p>The related exercise in the book - expressing <span class="math-container">$q_{\pi}$</span> in terms of <span class="math-container">$v_{\pi}$</span> and the MDP characteristics - is more complex because it involves a time step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-21T11:12:53.443Z
Answers.Text: It's worth noting that your expression for the deterministic policy is really just a shorter notation for the general case, given that, in the general case, for a deterministic policy, you will multiply $1$ by $q$ and put $a = \pi(s)$ in there for the only possible action, and have $0$ multiplied by $q$ for all other actions. So, the general formula still applies to the deterministic policy case. It's just a shorter notation, but they are mathematically equivalent/equal.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T07:37:22.533Z
Answers.LastActivityDate: 2020-06-17T09:29:40.613Z
Answers.LastEditDate: 2020-06-17T09:29:40.613Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="https://i.sstatic.net/hX1jB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hX1jB.png" alt="enter image description here" /></a></p>
  <p>This is exercise 3.18 in Sutton and Barto's book.</p>
  <p>The task is to express <span class="math-container">$v_\pi(s)$</span> using <span class="math-container">$q_\pi(s,a)$</span>.</p>
  <p>Looking at the diagram above, the value of <span class="math-container">$q_\pi(s,a)$</span> at <span class="math-container">$s$</span> for each <span class="math-container">$a \in A$</span> we take gives us the value function at <span class="math-container">$s$</span> after taking the action <span class="math-container">$a$</span> and then following the policy <span class="math-container">$\pi$</span>.</p>
  <p>This is probably wrong, but if</p>
  <p><span class="math-container">$$v_\pi(s) = E_\pi[G_t | S_t = s]$$</span></p>
  <p>and</p>
  <p><span class="math-container">$$q_\pi(s) = E_\pi[G_t | S_t = s, A_t = a]$$</span></p>
  <p>isn't then <span class="math-container">$v_\pi(s)$</span> just the expected action value function at <span class="math-container">$s$</span> over all actions <span class="math-container">$a$</span> that are given by the policy <span class="math-container">$\pi$</span>, namely</p>
  <p><span class="math-container">$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$</span>?</p>
  
Tags: <reinforcement-learning><comparison><value-functions><sutton-barto>
Title: How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
UserDisplayName: tmaric
ThreadText: # How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
  
  (Asked by: username_0 on 2020-06-17)
  
  [![enter image description here](https://i.sstatic.net/hX1jB.png)](https://i.sstatic.net/hX1jB.png)
  
  This is exercise 3.18 in Sutton and Barto's book.
  
  The task is to express $v\_\pi(s)$ using $q\_\pi(s,a)$.
  
  Looking at the diagram above, the value of $q\_\pi(s,a)$ at $s$ for each $a \in A$ we take gives us the value function at $s$ after taking the action $a$ and then following the policy $\pi$.
  
  This is probably wrong, but if
  
  $$v\_\pi(s) = E\_\pi[G\_t | S\_t = s]$$
  
  and
  
  $$q\_\pi(s) = E\_\pi[G\_t | S\_t = s, A\_t = a]$$
  
  isn't then $v\_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely
  
  $$v\_\pi(s) = E\_{a \sim \pi}[q\_\pi(s,a) | S\_t = s, A\_t = a] = \sum\_{a \in A}\pi(a|s) q\_\pi(s,a)$$?
  
  **Tags:** reinforcement-learning, comparison, value-functions, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-17. Score: 5 | Accepted answer
  
  > isn't then $v\_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely
  
  > $v\_\pi(s) = E\_{a \sim \pi}[q\_\pi(s,a) | S\_t = s, A\_t = a] = \sum\_{a \in A}\pi(a|s) q\_\pi(s,a)$?
  
  Yes this is 100% correct.
  
  There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $v\_{\pi}$ and $q\_{\pi}$ in that direction.
  
  Note that for a deterministic policy, with $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ then the relationship is
  
  $$v\_\pi(s) = q\_\pi(s, \pi(s))$$
  
  The related exercise in the book - expressing $q\_{\pi}$ in terms of $v\_{\pi}$ and the MDP characteristics - is more complex because it involves a time step.
  
  > **Comments:**
  > **username_2**: It's worth noting that your expression for the deterministic policy is really just a shorter notation for the general case, given that, in the general case, for a deterministic policy, you will multiply $1$ by $q$ and put $a = \pi(s)$ in there for the only possible action, and have $0$ multiplied by $q$ for all other actions. So, the general formula still applies to the deterministic policy case. It's just a shorter notation, but they are mathematically equivalent/equal.  
  
  ---
  
  