Answers.Body: <blockquote>
  <p>why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values?</p>
  </blockquote>
  <p>A policy function takes state as an argument and returns an action <span class="math-container">$a = \pi(s)$</span>, or it may return a probability distribution over actions <span class="math-container">$\mathbf{Pr}\{A_t=a|S_t=s \} =\pi(a|s)$</span>.</p>
  <p>In order to do this rationally, an agent needs to use the knowledge it has gained to select the best action. In value-based methods, the agent needs to identify the action that has the highest expected return. As an aside, whilst learning it may not take that action because it has decided to explore, but if it is not capable of even identifying a best action then there is no hope of it ever finding an optimal policy, and it cannot even perform <span class="math-container">$\epsilon$</span>-greedy action selection, which is a very basic exploration approach.</p>
  <p>If you use an action value estimate, then the agent can select the greedy action simply:</p>
  <p><span class="math-container">$$\pi(s) = \text{argmax}_a Q(s,a)$$</span></p>
  <p>If you have state values, then the agent can select the greedy action directly only if it knows the model distribution <span class="math-container">$p(r,s'|s,a)$</span>:</p>
  <p><span class="math-container">$$\pi(s) = \text{argmax}_a \sum_{r,s'}p(r,s'|s,a)(r + \gamma V(s'))$$</span></p>
  <p>In other words, to find the best action to take the agent needs to look ahead a time step to find out what the distribution of next states would be following that action. If the only values the agent knows are state values, this is the only way the agent can determine the impact of any specific action.</p>
  <p>Although there are alternatives to this specific equation, there is no alternative that does not use a model in some form. For instance, if you can simulate the environment, you could simulate taking each action in turn, and look over multiple simulation runs to see which choice ends up with the best <span class="math-container">$(r + \gamma V(s'))$</span> on average. That would be a type of planning, and perhaps the start of a more sophisticated approach such as MCTS. However, that simulation <em>is</em> a model - it needs access to the transition probabilities in some form in order to correctly run.</p>
  <p>It is possible to have an entirely <em>separate</em> policy function that you train alongside a state value function. This is the basis of Actor-Critic methods, which make use of policy gradients to adjust the policy function, and one of the value-based methods, such as TD learning, to learn a value function that assists with calculating the updates to the policy function. In that case you would not be using a value-based method on its own, so the quote from that part of Sutton &amp; Barto does not apply.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-09T06:08:07.587Z
Answers.Text: To clarify, we need the state-action values for model-free environments only, right? On the other hand, model-based environments should be able to estimate the best policy using the transition model, reward model, and state values - without needing to explicitly compute the state-action values. Thanks!
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-09T08:48:33.283Z
Answers.Text: @cogito_ai Yes, although if is not common to call an environment that you don't have the model for "model-free". Also, even if you don't need action values strictly, one of the model-free value-based methods could still be a good choice for other reasons - e.g. the calculation for greedy action using the model might take too long
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-28T16:17:00.107Z
Answers.Text: @NeilSlater I turn the question upside down. If we have a model of the environment, why should we don't calculate directly state_action_values and directly infer the optimal policy? Why should we do value and policy iteration if it is guaranteed to directly  find the optimal policy knowing the state_action values?
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-28T19:14:54.997Z
Answers.Text: @HermesMorales: Short version - an environment model is not the same as the action value function, and it takes some effort (maybe a lot of effort) to calculate e.g. $Q(s,a)$ from a known environment. Value or policy iteration are types of model-based methods to do just that. If that is not clear from the comment, please ask a new question on the site.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-07T08:55:59.810Z
Answers.LastActivityDate: 2020-08-07T09:24:55.150Z
Answers.LastEditDate: 2020-08-07T09:24:55.150Z
Answers.UserDisplayName: Neil Slater
Body: <blockquote>
  <p>&quot;If a model is not available, then it is particularly useful to estimate action values (the
  values of state-action pairs) rather than state values. With a model, state values alone are
  sufficient to determine a policy; one simply looks ahead one step and chooses whichever
  action leads to the best combination of reward and next state, as we did in the chapter on
  DP. <strong>Without a model, however, state values alone are not sufficient. One must explicitly
  estimate the value of each action in order for the values to be useful in suggesting a policy.</strong>&quot;</p>
  </blockquote>
  <p>The above extract is from Sutton and Barto's Reinforcement Learning, Section 5.2 - part of the chapter on Monte Carlo Methods.</p>
  <p>Could someone please explain in some more detail, as to <strong>why it is necessary to determine the value of each action</strong> (i.e. state-values alone are not sufficient) for suggesting a policy in a <strong>model-free setting</strong>?</p>
  <hr />
  <p>P.S.</p>
  <p>From what I know, state-values basically refer to the expected return one gets when starting from a state (we know that we'll reach a terminal state, since we're dealing with Monte Carlo methods which, at least in the book, look at only episodic MDPs). That being said, why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values? I'm a little confused, it'd really help if someone could clear it up.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T22:19:26.910Z
Comments.Text: [Here](https://ai.stackexchange.com/q/10584/2444) and [here](https://ai.stackexchange.com/q/24465/2444) are 2 related (if not duplicate) questions.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><monte-carlo-methods><model-free-methods>
Title: Why are state-values alone not sufficient in determining a policy (without a model)?
UserDisplayName: stoic-santiago
ThreadText: # Why are state-values alone not sufficient in determining a policy (without a model)?
  
  (Asked by: username_0 on 2020-08-07)
  
  > "If a model is not available, then it is particularly useful to estimate action values (the
  > values of state-action pairs) rather than state values. With a model, state values alone are
  > sufficient to determine a policy; one simply looks ahead one step and chooses whichever
  > action leads to the best combination of reward and next state, as we did in the chapter on
  > DP. **Without a model, however, state values alone are not sufficient. One must explicitly
  > estimate the value of each action in order for the values to be useful in suggesting a policy.**"
  
  The above extract is from Sutton and Barto's Reinforcement Learning, Section 5.2 - part of the chapter on Monte Carlo Methods.
  
  Could someone please explain in some more detail, as to **why it is necessary to determine the value of each action** (i.e. state-values alone are not sufficient) for suggesting a policy in a **model-free setting**?
  
  ---
  
  P.S.
  
  From what I know, state-values basically refer to the expected return one gets when starting from a state (we know that we'll reach a terminal state, since we're dealing with Monte Carlo methods which, at least in the book, look at only episodic MDPs). That being said, why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values? I'm a little confused, it'd really help if someone could clear it up.
  
  **Tags:** reinforcement-learning, monte-carlo-methods, model-free-methods
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/10584/2444) and [here](https://ai.stackexchange.com/q/24465/2444) are 2 related (if not duplicate) questions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-07. Score: 3 | Accepted answer
  
  > why is it not possible to suggest a policy solely on the basis of state-values; why do we need state-action values?
  
  A policy function takes state as an argument and returns an action $a = \pi(s)$, or it may return a probability distribution over actions $\mathbf{Pr}\{A\_t=a|S\_t=s \} =\pi(a|s)$.
  
  In order to do this rationally, an agent needs to use the knowledge it has gained to select the best action. In value-based methods, the agent needs to identify the action that has the highest expected return. As an aside, whilst learning it may not take that action because it has decided to explore, but if it is not capable of even identifying a best action then there is no hope of it ever finding an optimal policy, and it cannot even perform $\epsilon$-greedy action selection, which is a very basic exploration approach.
  
  If you use an action value estimate, then the agent can select the greedy action simply:
  
  $$\pi(s) = \text{argmax}\_a Q(s,a)$$
  
  If you have state values, then the agent can select the greedy action directly only if it knows the model distribution $p(r,s'|s,a)$:
  
  $$\pi(s) = \text{argmax}\_a \sum\_{r,s'}p(r,s'|s,a)(r + \gamma V(s'))$$
  
  In other words, to find the best action to take the agent needs to look ahead a time step to find out what the distribution of next states would be following that action. If the only values the agent knows are state values, this is the only way the agent can determine the impact of any specific action.
  
  Although there are alternatives to this specific equation, there is no alternative that does not use a model in some form. For instance, if you can simulate the environment, you could simulate taking each action in turn, and look over multiple simulation runs to see which choice ends up with the best $(r + \gamma V(s'))$ on average. That would be a type of planning, and perhaps the start of a more sophisticated approach such as MCTS. However, that simulation *is* a model - it needs access to the transition probabilities in some form in order to correctly run.
  
  It is possible to have an entirely *separate* policy function that you train alongside a state value function. This is the basis of Actor-Critic methods, which make use of policy gradients to adjust the policy function, and one of the value-based methods, such as TD learning, to learn a value function that assists with calculating the updates to the policy function. In that case you would not be using a value-based method on its own, so the quote from that part of Sutton & Barto does not apply.
  
  > **Comments:**
  > **username_0**: To clarify, we need the state-action values for model-free environments only, right? On the other hand, model-based environments should be able to estimate the best policy using the transition model, reward model, and state values - without needing to explicitly compute the state-action values. Thanks!  
  > **username_2**: @cogito_ai Yes, although if is not common to call an environment that you don't have the model for "model-free". Also, even if you don't need action values strictly, one of the model-free value-based methods could still be a good choice for other reasons - e.g. the calculation for greedy action using the model might take too long  
  > **username_3**: @username_2 I turn the question upside down. If we have a model of the environment, why should we don't calculate directly state_action_values and directly infer the optimal policy? Why should we do value and policy iteration if it is guaranteed to directly  find the optimal policy knowing the state_action values?  
  > **username_2**: @username_3: Short version - an environment model is not the same as the action value function, and it takes some effort (maybe a lot of effort) to calculate e.g. $Q(s,a)$ from a known environment. Value or policy iteration are types of model-based methods to do just that. If that is not clear from the comment, please ask a new question on the site.  
  
  ---
  
  