Answers.Body: <p>I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that</p>
  <blockquote>
  <p>the value of a state under an optimal policy must equal the expected return for the best action from that state.</p>
  </blockquote>
  <p>However, in chapter 4, they present and prove the &quot;Policy Improvement Theorem&quot; and deduce that</p>
  <blockquote>
  <p>It is a natural extension to consider changes at all states, selecting at each state the action that appears best according to <span class="math-container">$q_\pi(s, a)$</span>. In
  other words, to consider the new greedy policy, <span class="math-container">$\pi'$</span>, given by
  <span class="math-container">$\pi'(s) = \underset{a}{argmax} (q_\pi(s, a))$</span></p>
  </blockquote>
  <p>The fact that <span class="math-container">$v_∗(s)=\underset{a}{max} \ q_∗(s,a)$</span> is immediate from there, given that <span class="math-container">$v_*$</span> is the optimal value function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-14T21:30:37.350Z
Answers.LastActivityDate: 2024-05-14T21:30:37.350Z
Answers.UserDisplayName: AMT
Body: <p>The Sutton and Barto reinforcement learning textbook states that</p>
  <blockquote>
  <p>the value of a state under an optimal policy must equal the expected return for the best action from that state.</p>
  </blockquote>
  <p>That is,
  <span class="math-container">$$v_*(s) = \max_a q_*(s, a).$$</span></p>
  <p>I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how</p>
  <p><span class="math-container">$$v_*(s) = \sum_a \pi_*(a|s)q_*(s,a) = \max_a q_*(s, a).$$</span></p>
  <p>I'd appreciate any insights!</p>
  
Tags: <reinforcement-learning><value-functions><policies><bellman-equations>
Title: Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
UserDisplayName: bonzo_pippinpaddle
ThreadText: # Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
  
  (Asked by: username_0 on 2021-08-22)
  
  The Sutton and Barto reinforcement learning textbook states that
  
  > the value of a state under an optimal policy must equal the expected return for the best action from that state.
  
  That is,
  $$v\_\*(s) = \max\_a q\_\*(s, a).$$
  
  I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how
  
  $$v\_\*(s) = \sum\_a \pi\_\*(a|s)q\_\*(s,a) = \max\_a q\_\*(s, a).$$
  
  I'd appreciate any insights!
  
  **Tags:** reinforcement-learning, value-functions, policies, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-14. Score: 1
  
  I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that
  
  > the value of a state under an optimal policy must equal the expected return for the best action from that state.
  
  However, in chapter 4, they present and prove the "Policy Improvement Theorem" and deduce that
  
  > It is a natural extension to consider changes at all states, selecting at each state the action that appears best according to $q\_\pi(s, a)$. In
  > other words, to consider the new greedy policy, $\pi'$, given by
  > $\pi'(s) = \underset{a}{argmax} (q\_\pi(s, a))$
  
  The fact that $v\_∗(s)=\underset{a}{max} \ q\_∗(s,a)$ is immediate from there, given that $v\_\*$ is the optimal value function.
  
  
  ---
  
  