Answers.Body: <blockquote>
    <p>Can we say that <span class="math-container">$Q^\pi(s, a) = V^\pi(s)$</span></p>
  </blockquote>
  
  <p>No.</p>
  
  <p>The correct relationship is this:</p>
  
  <p><span class="math-container">$$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$$</span></p>
  
  <p>or, if you have a <em>deterministic</em> policy <span class="math-container">$a = \pi(s)$</span> you can instead write:</p>
  
  <p><span class="math-container">$$V^\pi(s) = Q^\pi(s, \pi(s))$$</span></p>
  
  <p>Intuitively, this is because the <span class="math-container">$V^\pi(s)$</span> is the expected future return when following the policy <span class="math-container">$\pi$</span> from state <span class="math-container">$s$</span>, whilst <span class="math-container">$Q^\pi(s, a)$</span> is the expected future return where it <em>ignores</em> the policy for only the next action <span class="math-container">$a$</span> (which will decide immediate reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> independently of the policy), and thereafter follows <span class="math-container">$\pi$</span>. </p>
  
  <p>The above equations essentially show what happens when you apply the policy in state <span class="math-container">$s$</span> to decide which Q value(s) to use, they remove the independent choice of <span class="math-container">$a$</span> in <span class="math-container">$Q^\pi(s,a)$</span>. </p>
  
  <p>One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When <em>any</em> algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state <span class="math-container">$s$</span> when making a free choice of <span class="math-container">$a$</span> for the next step, and thereafter strictly following the policy being evaluated. </p>
  
  <p>What <em>is</em> different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different <em>target policy</em>. Both approaches have the same interpretation of what <span class="math-container">$Q(s,a)$</span> means otherwise, and have the same relationship between Q and V for their repsective policies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-19T10:30:37.230Z
Answers.LastActivityDate: 2020-03-19T14:16:01.977Z
Answers.LastEditDate: 2020-03-19T14:16:01.977Z
Answers.UserDisplayName: Neil Slater
Body: <p>Suppose we have a policy <span class="math-container">$\pi$</span> and we use <em>SARSA</em> to evaluate <span class="math-container">$Q^\pi(s, a)$</span>, where <span class="math-container">$a$</span> is the policy <span class="math-container">$\pi$</span>.</p>
  
  <p>Can we say that <span class="math-container">$Q^\pi(s, a) = V^\pi(s)$</span>? </p>
  
  <p>The reason why I think this can be the case is because <span class="math-container">$Q^\pi(s, a)$</span> is defined as the value obtained from taking action <span class="math-container">$a$</span> and then following policy <span class="math-container">$\pi$</span> thereafter. However, the action <span class="math-container">$a$</span> taken is the policy according to <span class="math-container">$\pi$</span> for all <span class="math-container">$s \in S $</span>. This seems to corresponds to the value function equation of <span class="math-container">$V^\pi(s_t) = r(s_t) + \gamma V^\pi(s_{t+1})$</span>.</p>
  
Tags: <reinforcement-learning><comparison><sarsa><value-functions>
Title: What is the relationship between the Q and V functions?
UserDisplayName: calveeen
ThreadText: # What is the relationship between the Q and V functions?
  
  (Asked by: username_0 on 2020-03-19)
  
  Suppose we have a policy $\pi$ and we use *SARSA* to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.
  
  Can we say that $Q^\pi(s, a) = V^\pi(s)$?
  
  The reason why I think this can be the case is because $Q^\pi(s, a)$ is defined as the value obtained from taking action $a$ and then following policy $\pi$ thereafter. However, the action $a$ taken is the policy according to $\pi$ for all $s \in S $. This seems to corresponds to the value function equation of $V^\pi(s\_t) = r(s\_t) + \gamma V^\pi(s\_{t+1})$.
  
  **Tags:** reinforcement-learning, comparison, sarsa, value-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-19. Score: 2 | Accepted answer
  
  > Can we say that $Q^\pi(s, a) = V^\pi(s)$
  
  No.
  
  The correct relationship is this:
  
  $$V^\pi(s) = \sum\_a \pi(a|s) Q^\pi(s, a)$$
  
  or, if you have a *deterministic* policy $a = \pi(s)$ you can instead write:
  
  $$V^\pi(s) = Q^\pi(s, \pi(s))$$
  
  Intuitively, this is because the $V^\pi(s)$ is the expected future return when following the policy $\pi$ from state $s$, whilst $Q^\pi(s, a)$ is the expected future return where it *ignores* the policy for only the next action $a$ (which will decide immediate reward $r$ and next state $s'$ independently of the policy), and thereafter follows $\pi$.
  
  The above equations essentially show what happens when you apply the policy in state $s$ to decide which Q value(s) to use, they remove the independent choice of $a$ in $Q^\pi(s,a)$.
  
  One possible misunderstanding that you have is that "on-policy" means the same thing as the equations show when considering action values (Q values) - it does not. When *any* algorithm learns action values, it learns the same thing conceptually, i.e. the expected (and maybe discounted) sum of future rewards given state $s$ when making a free choice of $a$ for the next step, and thereafter strictly following the policy being evaluated.
  
  What *is* different between on-policy and off-policy is which policy gets evaluated - for on-policy methods like SARSA you evaluate action values for the same policy that you use to generate actions. For off-policy methods like Q learning you evaluate a different *target policy*. Both approaches have the same interpretation of what $Q(s,a)$ means otherwise, and have the same relationship between Q and V for their repsective policies.
  
  
  ---
  
  