Answers.Body: <p>In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from <a href="https://ai.stanford.edu/%7Eang/papers/icml00-irl.pdf" rel="nofollow noreferrer">Andrew Ng's Algorithms for Inverse Reinforcement Learning</a> <span class="math-container">$$V^{\pi}(s) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$$</span></p>
  <p>In this setting, <span class="math-container">$Q^{\pi}$</span> can be obtained from <span class="math-container">$V^{\pi}$</span> because we have access to the transition model <span class="math-container">$P(s'|s,a)$</span>. The <span class="math-container">$Q^{\pi}$</span> values allow us to carry out a step in <span class="math-container">$\textbf{policy improvement}$</span> as in policy iteration.</p>
  <p>To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for <span class="math-container">$\textbf{state values}$</span> is not helpful in determining how to carry out model free control because we cannot compute <span class="math-container">$Q^{\pi}(s,a)$</span> from <span class="math-container">$V^{\pi}$</span> in the model free case.</p>
  <p>The update for SARSA in model free control is <span class="math-container">$$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$</span></p>
  <p>Even though we do not know the transition model, we are essentially <span class="math-container">$\textbf{sampling}$</span> from <span class="math-container">$P(s'|s,a)$</span> by allowing the environment to provide us the possible next states <span class="math-container">$s'$</span> that we may end up in. The following update for SARSA is equivalent to computing <span class="math-container">$$Q^{\pi}(s,a) = R(s) + \gamma E_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$</span>
  Essentially this should give the same <span class="math-container">$Q^{\pi}(s,a)$</span> values when we have the ground truth <span class="math-container">$P(s'|s,a)$</span> values for the model free case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T05:43:15.997Z
Answers.Text: Awesome! Why are the formulas for V and Q the same in your answer? Isn't V = Pi * Q ?
Answers.UserDisplayName: Aung Khant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T07:25:30.800Z
Answers.Text: Because SARSA update on the RHS utilises $Q(s',a')$. Also, as $a'$ is selected from the policy $\pi$, this is equivalent to the definition of $V^{\pi}$
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T01:55:48.037Z
Answers.LastActivityDate: 2020-11-20T14:09:51.520Z
Answers.LastEditDate: 2020-11-20T14:09:51.520Z
Answers.UserDisplayName: calveeen
Body: <p><a href="https://www.youtube.com/watch?v=0g4j2k_Ggc4&amp;list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&amp;index=5" rel="nofollow noreferrer">Here is David Silver's lecture on that</a>. Look at 9:30 to 10:30.</p>
  <p>He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function <span class="math-container">$Q$</span> is used.</p>
  <ul>
  <li><p>But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?</p>
  </li>
  <li><p>Secondly, even if we were to use <span class="math-container">$Q$</span>, we have <span class="math-container">$Q^{\pi}(s,a) = R(s) + \gamma \sum_{s'}P(s'|s,a)V^{\pi}(s')$</span>, so we still need to know the transition model, which is unknown.</p>
  </li>
  </ul>
  <p>What am I missing here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T02:11:06.920Z
Comments.Text: This seems to be a duplicate of [this](https://ai.stackexchange.com/q/24465/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><monte-carlo-methods><value-functions><model-based-methods><model-free-methods>
Title: Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
UserDisplayName: Aung Khant
ThreadText: # Why does Monte Carlo policy evaluation relies on action-value function rather than state-value function?
  
  (Asked by: username_0 on 2020-11-19)
  
  [Here is David Silver's lecture on that](https://www.youtube.com/watch?v=0g4j2k_Ggc4&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=5). Look at 9:30 to 10:30.
  
  He says that, since it is model-free learning, the environment's dynamics are unknown, so the action-value function $Q$ is used.
  
  * But then state-values are already calculated (via first-visit or every-visit). So, why aren't these values used?
  * Secondly, even if we were to use $Q$, we have $Q^{\pi}(s,a) = R(s) + \gamma \sum\_{s'}P(s'|s,a)V^{\pi}(s')$, so we still need to know the transition model, which is unknown.
  
  What am I missing here?
  
  **Tags:** reinforcement-learning, monte-carlo-methods, value-functions, model-based-methods, model-free-methods
  
  > **Comments:**
  > **username_1**: This seems to be a duplicate of [this](https://ai.stackexchange.com/q/24465/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-20. Score: 3 | Accepted answer
  
  In Model Based Reinforcement learning, state and state-action values for all states can be calculated based on the bellman equations. The equations are taken from [Andrew Ng's Algorithms for Inverse Reinforcement Learning](https://ai.stanford.edu/%7Eang/papers/icml00-irl.pdf) $$V^{\pi}(s) = R(s) + \gamma \sum\_{s'}P(s'|s,a)V^{\pi}(s') \\ Q^{\pi}(s,a) = R(s) + \gamma \sum\_{s'}P(s'|s,a)V^{\pi}(s')$$
  
  In this setting, $Q^{\pi}$ can be obtained from $V^{\pi}$ because we have access to the transition model $P(s'|s,a)$. The $Q^{\pi}$ values allow us to carry out a step in $\textbf{policy improvement}$ as in policy iteration.
  
  To answer the first bullet point, the first visit or every state visit policy evaluation in the model free setting for $\textbf{state values}$ is not helpful in determining how to carry out model free control because we cannot compute $Q^{\pi}(s,a)$ from $V^{\pi}$ in the model free case.
  
  The update for SARSA in model free control is $$Q(s,a) \rightarrow Q(s,a) + \alpha (r(s) + \gamma Q(s',a') - Q(s,a))$$
  
  Even though we do not know the transition model, we are essentially $\textbf{sampling}$ from $P(s'|s,a)$ by allowing the environment to provide us the possible next states $s'$ that we may end up in. The following update for SARSA is equivalent to computing $$Q^{\pi}(s,a) = R(s) + \gamma E\_{s' \sim P(s'|s,a)}[Q^{\pi}(s',a')]$$
  Essentially this should give the same $Q^{\pi}(s,a)$ values when we have the ground truth $P(s'|s,a)$ values for the model free case.
  
  > **Comments:**
  > **username_0**: Awesome! Why are the formulas for V and Q the same in your answer? Isn't V = Pi * Q ?  
  > **username_2**: Because SARSA update on the RHS utilises $Q(s',a')$. Also, as $a'$ is selected from the policy $\pi$, this is equivalent to the definition of $V^{\pi}$  
  
  ---
  
  