Answers.Body: <p>Your first equation is the definition of any state value function, so it must also be definition of the optimal state value function <span class="math-container">$v_*$</span>.</p>
  <p>The second equation is the definition of <span class="math-container">$v_*$</span> in terms of the state-action value function <span class="math-container">$\color{green}{q_*}$</span>.</p>
  <p>In reality, the first equation is also the definition of <span class="math-container">$v_*$</span> in terms of <span class="math-container">$\color{green}{q_*}$</span>, which is what you want to see :)</p>
  <p>First, note that</p>
  <p><span class="math-container">\begin{align}
  v_\pi(s) 
  &amp;=
   \sum_a \pi(a \mid s) 
  \color{blue}{\sum_{s'} \sum_r p(s', r|s, a) [r + \gamma v_\pi(s')]} \\
  &amp;= \sum_a \pi(a \mid s) \color{blue}{q_\pi(s, a)} \tag{1}\label{1}.
  \end{align}</span>
  Now, we denote the optimal state value function as <span class="math-container">$v_{\pi_*} = v_*$</span>. If we plug <span class="math-container">$v_*$</span>, <span class="math-container">$\pi_*$</span> and <span class="math-container">$\color{green}{q_*}$</span> in the equation above, we get your first equation, but we write it as</p>
  <p><span class="math-container">\begin{align}
  v_*(s) 
  &amp;= \sum_a \pi_*(a \mid s) \color{green}{q_*(s, a)} \tag{2}\label{2}.
  \end{align}</span>
  For finite MDPs, the optimal policy is deterministic, i.e. it chooses one action (the optimal one) with probability <span class="math-container">$1$</span>, so that means that \ref{2} can be written as</p>
  <p><span class="math-container">\begin{align}
  v_*(s) 
  &amp;= 0 \color{green}{q_*(s, a_1)} + \cdots + 1 \color{green}{q_*(s, a_*)} + 0 \color{green}{q_*(s, a_N)} \\
  &amp;= \color{green}{q_*(s, a_*)} \\
  &amp;= \max_a \color{green}{q_*(s, a)} \\
  &amp;=\max_a \color{green}{ \sum_{s'} \sum\limits_r p(s', r|s,a) [r + \gamma v_{*}(s')]}
  \tag{3}\label{3},
  \end{align}</span>
  where <span class="math-container">$a_* = \text{argmax}_a\pi_*(a \mid s) = \text{argmax}_a \color{green}{q_*(s, a)}$</span> is the optimal action. By definition, the optimal action in state <span class="math-container">$s$</span> is the one that leads to the highest expected return. See also <a href="https://ai.stackexchange.com/a/36209/2444">this answer</a>.</p>
  <p>Finally, note that optimal value functions are unique for finite MPDs, so <span class="math-container">$\color{green}{q_*}$</span> and <span class="math-container">$v_*$</span> are unique.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-24T03:45:09.597Z
Answers.Text: still, there are some typos and unclear points: (1) $a_*$ is an action value and $\pi_{*}(s)$ is a probability value and hence they cannot be equal as you wrote in the bottom line. (2) why do we have $q_{*}(s,a_*)= \max_{a}q_{*}(s,a)?$
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-24T08:49:35.100Z
Answers.Text: @DSPinfinity I used my notation inconsistently. If $\pi_*(s)$ is a probability distribution, then $\pi_*(s)$ would be a probability. However, there, I used $\pi_*$ to refer to a function (or decision rule) that is derived from the stochastic policy. $a_* = \text{argmax}_a \pi_*(s, a)$. $a_*$ is an action, not an action value, $q_*(s, a_*)$ would be the (action) value of $a_*$ in $s$. You have $q_*(s, a_*) = \max_a q_*(s, a)$ because the highest action value in state $s$ of the optimal value function is associated with the optimal action, $a_*$ That's the definition of the optimal action.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-24T08:51:58.870Z
Answers.Text: @DSPinfinity Maybe [this](https://ai.stackexchange.com/a/36209/2444) can also help.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T23:39:17.970Z
Answers.LastActivityDate: 2023-01-24T10:52:54.580Z
Answers.LastEditDate: 2023-01-24T10:52:54.580Z
Answers.UserDisplayName: nbro
Body: <p>By substituting the optimal policy <span class="math-container">$\pi_{\star}$</span> into the Bellman equation, we get the Bellman equation for <span class="math-container">$v_{\pi_{\star}}(s)=v_{\star}(s)$</span>:</p>
  <p><span class="math-container">$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\limits_{s'} \sum_r p(s', r | s, a)[r + \gamma v_{\star}(s')]$$</span></p>
  <p>From the above equation, how can we obtain the this one?</p>
  <p><span class="math-container">$$ v_{\star}(s) = \max\limits_a \sum\limits_{s'} \sum\limits_r p(s', r|s,a) [r + \gamma v_{\star}(s')]$$</span></p>
  
Tags: <reinforcement-learning><comparison><value-functions><bellman-equations><optimality>
Title: How are these two equations for the optimal state-value function equivalent?
UserDisplayName: DSPinfinity
ThreadText: # How are these two equations for the optimal state-value function equivalent?
  
  (Asked by: username_0 on 2023-01-22)
  
  By substituting the optimal policy $\pi\_{\star}$ into the Bellman equation, we get the Bellman equation for $v\_{\pi\_{\star}}(s)=v\_{\star}(s)$:
  
  $$ v\_{\star}(s) = \sum\limits\_a \pi\_{\star}(a|s) \sum\limits\_{s'} \sum\_r p(s', r | s, a)[r + \gamma v\_{\star}(s')]$$
  
  From the above equation, how can we obtain the this one?
  
  $$ v\_{\star}(s) = \max\limits\_a \sum\limits\_{s'} \sum\limits\_r p(s', r|s,a) [r + \gamma v\_{\star}(s')]$$
  
  **Tags:** reinforcement-learning, comparison, value-functions, bellman-equations, optimality
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-23. Score: 3 | Accepted answer
  
  Your first equation is the definition of any state value function, so it must also be definition of the optimal state value function $v\_\*$.
  
  The second equation is the definition of $v\_\*$ in terms of the state-action value function $\color{green}{q\_\*}$.
  
  In reality, the first equation is also the definition of $v\_\*$ in terms of $\color{green}{q\_\*}$, which is what you want to see :)
  
  First, note that
  
  \begin{align}
  v\_\pi(s)
  &=
  \sum\_a \pi(a \mid s)
  \color{blue}{\sum\_{s'} \sum\_r p(s', r|s, a) [r + \gamma v\_\pi(s')]} \\
  &= \sum\_a \pi(a \mid s) \color{blue}{q\_\pi(s, a)} \tag{1}\label{1}.
  \end{align}
  Now, we denote the optimal state value function as $v\_{\pi\_\*} = v\_\*$. If we plug $v\_\*$, $\pi\_\*$ and $\color{green}{q\_\*}$ in the equation above, we get your first equation, but we write it as
  
  \begin{align}
  v\_\*(s)
  &= \sum\_a \pi\_\*(a \mid s) \color{green}{q\_\*(s, a)} \tag{2}\label{2}.
  \end{align}
  For finite MDPs, the optimal policy is deterministic, i.e. it chooses one action (the optimal one) with probability $1$, so that means that \ref{2} can be written as
  
  \begin{align}
  v\_\*(s)
  &= 0 \color{green}{q\_\*(s, a\_1)} + \cdots + 1 \color{green}{q\_\*(s, a\_\*)} + 0 \color{green}{q\_\*(s, a\_N)} \\
  &= \color{green}{q\_\*(s, a\_\*)} \\
  &= \max\_a \color{green}{q\_\*(s, a)} \\
  &=\max\_a \color{green}{ \sum\_{s'} \sum\limits\_r p(s', r|s,a) [r + \gamma v\_{\*}(s')]}
  \tag{3}\label{3},
  \end{align}
  where $a\_\* = \text{argmax}\_a\pi\_\*(a \mid s) = \text{argmax}\_a \color{green}{q\_\*(s, a)}$ is the optimal action. By definition, the optimal action in state $s$ is the one that leads to the highest expected return. See also [this answer](https://ai.stackexchange.com/a/36209/2444).
  
  Finally, note that optimal value functions are unique for finite MPDs, so $\color{green}{q\_\*}$ and $v\_\*$ are unique.
  
  > **Comments:**
  > **username_0**: still, there are some typos and unclear points: (1) $a_*$ is an action value and $\pi_{*}(s)$ is a probability value and hence they cannot be equal as you wrote in the bottom line. (2) why do we have $q_{*}(s,a_*)= \max_{a}q_{*}(s,a)?$  
  > **username_1**: @username_0 I used my notation inconsistently. If $\pi_*(s)$ is a probability distribution, then $\pi_*(s)$ would be a probability. However, there, I used $\pi_*$ to refer to a function (or decision rule) that is derived from the stochastic policy. $a_* = \text{argmax}_a \pi_*(s, a)$. $a_*$ is an action, not an action value, $q_*(s, a_*)$ would be the (action) value of $a_*$ in $s$. You have $q_*(s, a_*) = \max_a q_*(s, a)$ because the highest action value in state $s$ of the optimal value function is associated with the optimal action, $a_*$ That's the definition of the optimal action.  
  > **username_1**: @username_0 Maybe [this](https://ai.stackexchange.com/a/36209/2444) can also help.  
  
  ---
  
  