Answers.Body: <p>The core differences between using <span class="math-container">$V(s)$</span> or <span class="math-container">$Q(s,a)$</span> are:</p>
  <h2><span class="math-container">$V(s)$</span> cannot be used stand-alone to decide a policy.</h2>
  <p>You either need a separate policy function <span class="math-container">$\pi(a|s)$</span> that it is the value function for, or you can derive a policy from it <strong>if</strong> you have access to the environment's distribution model <span class="math-container">$p(r,s'|s,a)$</span> - the probability of receiving reward <span class="math-container">$r$</span> and arriving in next state <span class="math-container">$s'$</span> given start <span class="math-container">$s,a$</span>. The derived policy would be deterministic <span class="math-container">$\pi(s) = \text{argmax}_a \sum_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$</span></p>
  <p>In comparison, <span class="math-container">$Q(s,a)$</span> can be used to derive a policy without reference to any model: <span class="math-container">$\pi(s) = \text{argmax}_a Q(s,a)$</span>. This makes the action value function Q necessary for model-free value-based control methods. Hence Monte-Carlo Control, SARSA, Q-Learning will all use action values.</p>
  <h2><span class="math-container">$V(s)$</span> is usually more compact than <span class="math-container">$Q(s,a)$</span>.</h2>
  <p>Space used by a state value table is <span class="math-container">$O(|\mathcal{S}|)$</span></p>
  <p>Space used by an action-value table is <span class="math-container">$O(|\mathcal{S} \times \mathcal{A}|)$</span></p>
  <p>These space concerns also apply when you are using approximation - the input domain you need to learn to approximate <span class="math-container">$Q(s,a)$</span> is larger than the one you need to learn <span class="math-container">$V(s)$</span>. Although for large state spaces with lots of dimensions, the relative difference after approximation may be small enough that you don't really care.</p>
  <p>The smaller space makes <span class="math-container">$V(s)$</span> a good choice if you <em>can</em> use it. The benefits are smaller memory footprint, and often faster learning because there are fewer separate values to learn (not always faster, as you still need to explore all actions in all states to learn accurate values).</p>
  <h2>Typical scenarios where you can use <span class="math-container">$V(s)$</span> instead of <span class="math-container">$Q(s,a)$</span></h2>
  <ul>
  <li>You have access to the full environment model, as described in the first section, above.</li>
  </ul>
  <p>or</p>
  <ul>
  <li>You are working with a separate policy function. In optimal control, this typically occurs in actor-critic methods, which are relatively advanced RL approach, but a popular one, especially when the action space is large. The situation also occurs in prediction problems where the task is not to optimise anything, but to evaluate a fixed policy.</li>
  </ul>
  <p>or</p>
  <ul>
  <li>Your actions are directly choosing the next state. This is called afterstate representation, and is subtly different to state values <span class="math-container">$V(s)$</span>, but has similar benefit of smaller space used. It is useful in determining values in board games and similar environments.</li>
  </ul>
  <p>If none of these situations applies, then you will have to use action-value function <span class="math-container">$Q(s,a)$</span>. You might also choose to use <span class="math-container">$Q(s,a)$</span> because it is easier to stick with it in whatever framework you are working in. Sometimes the benefits of using <span class="math-container">$V(s)$</span> instead are not large enough to be worth the effort.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-20T01:03:29.090Z
Answers.Text: @NeilSlater, Thank you answering my question.  I have now asked my scenario specific question. Can you please see if you can answer or suggest something to that as well? Maybe then you can know why  I asked the question on this page. I really need help on this please.
Answers.UserDisplayName: knowledge_seeker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-19T16:10:03.913Z
Answers.LastActivityDate: 2021-11-20T00:40:37.767Z
Answers.LastEditDate: 2021-11-20T00:40:37.767Z
Answers.UserDisplayName: Neil Slater
Body: <p>I saw the difference between value function <span class="math-container">$V(s)$</span> and <span class="math-container">$Q(s, a)$</span>. But when do I use each one? When I coded in Matlab I only used <span class="math-container">$Q(s, a)$</span> directly (as I was thinking of a tabular approach). So, when is more beneficial than the other? I have a large state space.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T22:26:47.353Z
Comments.Text: [Here](https://ai.stackexchange.com/q/18017/2444), [here](https://ai.stackexchange.com/q/10584/2444) and [here](https://ai.stackexchange.com/q/21723/2444) are 3 related questions.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><comparison><value-functions>
Title: When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
UserDisplayName: knowledge_seeker
ThreadText: # When to use the state value function $V(s)$ and when to use the state-action value function $Q(s, a)$?
  
  (Asked by: username_0 on 2021-11-19)
  
  I saw the difference between value function $V(s)$ and $Q(s, a)$. But when do I use each one? When I coded in Matlab I only used $Q(s, a)$ directly (as I was thinking of a tabular approach). So, when is more beneficial than the other? I have a large state space.
  
  **Tags:** reinforcement-learning, comparison, value-functions
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/18017/2444), [here](https://ai.stackexchange.com/q/10584/2444) and [here](https://ai.stackexchange.com/q/21723/2444) are 3 related questions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-19. Score: 9 | Accepted answer
  
  The core differences between using $V(s)$ or $Q(s,a)$ are:
  
  ## $V(s)$ cannot be used stand-alone to decide a policy.
  
  You either need a separate policy function $\pi(a|s)$ that it is the value function for, or you can derive a policy from it **if** you have access to the environment's distribution model $p(r,s'|s,a)$ - the probability of receiving reward $r$ and arriving in next state $s'$ given start $s,a$. The derived policy would be deterministic $\pi(s) = \text{argmax}\_a \sum\_{r,s'} p(r,s'|s,a)(r + \gamma V(s'))$
  
  In comparison, $Q(s,a)$ can be used to derive a policy without reference to any model: $\pi(s) = \text{argmax}\_a Q(s,a)$. This makes the action value function Q necessary for model-free value-based control methods. Hence Monte-Carlo Control, SARSA, Q-Learning will all use action values.
  
  ## $V(s)$ is usually more compact than $Q(s,a)$.
  
  Space used by a state value table is $O(|\mathcal{S}|)$
  
  Space used by an action-value table is $O(|\mathcal{S} \times \mathcal{A}|)$
  
  These space concerns also apply when you are using approximation - the input domain you need to learn to approximate $Q(s,a)$ is larger than the one you need to learn $V(s)$. Although for large state spaces with lots of dimensions, the relative difference after approximation may be small enough that you don't really care.
  
  The smaller space makes $V(s)$ a good choice if you *can* use it. The benefits are smaller memory footprint, and often faster learning because there are fewer separate values to learn (not always faster, as you still need to explore all actions in all states to learn accurate values).
  
  ## Typical scenarios where you can use $V(s)$ instead of $Q(s,a)$
  
  * You have access to the full environment model, as described in the first section, above.
  
  or
  
  * You are working with a separate policy function. In optimal control, this typically occurs in actor-critic methods, which are relatively advanced RL approach, but a popular one, especially when the action space is large. The situation also occurs in prediction problems where the task is not to optimise anything, but to evaluate a fixed policy.
  
  or
  
  * Your actions are directly choosing the next state. This is called afterstate representation, and is subtly different to state values $V(s)$, but has similar benefit of smaller space used. It is useful in determining values in board games and similar environments.
  
  If none of these situations applies, then you will have to use action-value function $Q(s,a)$. You might also choose to use $Q(s,a)$ because it is easier to stick with it in whatever framework you are working in. Sometimes the benefits of using $V(s)$ instead are not large enough to be worth the effort.
  
  > **Comments:**
  > **username_0**: @username_2, Thank you answering my question.  I have now asked my scenario specific question. Can you please see if you can answer or suggest something to that as well? Maybe then you can know why  I asked the question on this page. I really need help on this please.  
  
  ---
  
  