Answers.Body: <p>In general they are <em>not</em> the same and that should be clear as to why -- mathematically you are conditioning on an extra random variable being known in the state-action value function. You have the correct relationship between them, but I think your understanding of the two may be slightly off. The state-action value function is a function of both <span class="math-container">$s$</span> and <span class="math-container">$a$</span> whereas the value function is a function of just <span class="math-container">$s$</span>. As you have noted, the value function is equal to the expected value of the state-action value function, with the expectation taken over the action distribution induced by the policy.</p>
  <p>However, if you have a deterministic policy, then two are equivalent <em>only if you evaluate the state-action value function at the action that the deterministic policy gives</em>. That is, <span class="math-container">$v_\pi(s) = Q(s, \pi(s))$</span>. This is because, as you say, for a given state the policy will assign probability one to a certain action and 0 to all other actions in the action space and so the equation I just wrote is what the expectation in your question reduces to.</p>
  <p>Now, you might think 'well the policy is deterministic so we are always going to choose this action, so my theory was correct'. This is not true. The definition of the state-action value function is the value of the expected future (possibly discounted) returns given that we are in state <span class="math-container">$s$</span> and given that we have taken action <span class="math-container">$a$</span>. This means that we have already chosen our action and we are <em>not</em> choosing it according to the policy -- the policy will only choose all <em>future</em> actions.</p>
  <p>You also say that &quot;If we are operating a greedy policy and hence acting optimally&quot; which is not true. Just because we act greedily it does <em>not</em> imply we are acting optimally. Value functions are functions of a policy. If the policy is completely random then you would get certain value and state-action value functions which you could act greedily to but this would not be optimal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T12:26:13.783Z
Answers.Text: Hi @DavidIreland I completely agree about the last paragraph. What I was trying to say was that acting greedily WITH an optimal policy then we would be acting optimally. Would this be correct? Given we have found an optimal policy and there is no noise in the environment if we act greedily then it should also be optimal, correct?
Answers.UserDisplayName: InvestingScientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T12:27:45.677Z
Answers.Text: Your other points also makes sense, we are saying that in fact they wont be equivalent because although we evaluate all future steps with the optimal policy, the current step was chosen for this state, as defined by the definition of state action values. Is this correct?
Answers.UserDisplayName: InvestingScientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T12:34:15.020Z
Answers.Text: If you are acting greedily according to optimal action-value functions then yes you are acting optimally. 
  
  with regards to the second comment, yes that is indeed what we are saying. All future steps would agree except the returns would differ depending on what the current action taken was.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T12:17:56.733Z
Answers.LastActivityDate: 2020-12-15T12:17:56.733Z
Answers.UserDisplayName: David
Body: <p>Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as <span class="math-container">$V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$</span>. If we are operating a greedy policy and hence acting optimally, doesn't this mean that in fact the policy is deterministic and then <span class="math-container">$\pi(a|s)$</span> is <span class="math-container">$1$</span> for the optimal action and <span class="math-container">$0$</span> for all others? Would this then lead to an equivalence between the two?</p>
  <p>Here is my work to formulate some form of proof where I start with the idea that a policy is defined to be better than a current policy if for all states then <span class="math-container">$Q_{\pi}(S,\pi^∗(s))\geq Vπ_{\pi}(s)$</span> :</p>
  <p>I iteratively apply the optimal policy to each time step until I eventually get to a fully optimal time step of rewards</p>
  <p><span class="math-container">$$Vπ_{\pi}(s)≤Q_{\pi}(S,\pi^∗(s))$$</span>
  <span class="math-container">$$=Eπ[R_{t+1}+\gamma V_{\pi}(St+1)|St=s]$$</span>
  <span class="math-container">$$\leq E[Rt+1+\gamma Q_{\pi}(S_{t+1},\pi^∗(S_{t+1})|S_t=s]$$</span>
  <span class="math-container">$$\leq E[Rt+1+\gamma Rt+2+\gamma 2Q \pi^*(S_{t+2},\pi^∗(S_{t+2})|S_t=s]$$</span>
  <span class="math-container">$$\leq E[R_{t+1}+\gamma R_{t+2}+....|S_t=s]$$</span>
  <span class="math-container">$$=V\pi^∗(s)$$</span></p>
  <p>I would say that our final two lines are in fact inequalities, and for me this makes intuitive sense in that if we are always taking a deterministic greedy action our value function and Q function are the same. As detailed here, for a given policy and state we have that <span class="math-container">$V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$</span> and if the policy is optimal and hence greedy then <span class="math-container">$\pi(a|s)$</span> is deterministic.</p>
  
Tags: <reinforcement-learning><comparison><value-functions><bellman-equations>
Title: Are the state-action values and the state value function equivalent for a given policy?
UserDisplayName: InvestingScientist
ThreadText: # Are the state-action values and the state value function equivalent for a given policy?
  
  (Asked by: username_0 on 2020-12-14)
  
  Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum\_a \pi(a|s)Q\_{\pi}(s,a)$. If we are operating a greedy policy and hence acting optimally, doesn't this mean that in fact the policy is deterministic and then $\pi(a|s)$ is $1$ for the optimal action and $0$ for all others? Would this then lead to an equivalence between the two?
  
  Here is my work to formulate some form of proof where I start with the idea that a policy is defined to be better than a current policy if for all states then $Q\_{\pi}(S,\pi^∗(s))\geq Vπ\_{\pi}(s)$ :
  
  I iteratively apply the optimal policy to each time step until I eventually get to a fully optimal time step of rewards
  
  $$Vπ\_{\pi}(s)≤Q\_{\pi}(S,\pi^∗(s))$$
  $$=Eπ[R\_{t+1}+\gamma V\_{\pi}(St+1)|St=s]$$
  $$\leq E[Rt+1+\gamma Q\_{\pi}(S\_{t+1},\pi^∗(S\_{t+1})|S\_t=s]$$
  $$\leq E[Rt+1+\gamma Rt+2+\gamma 2Q \pi^\*(S\_{t+2},\pi^∗(S\_{t+2})|S\_t=s]$$
  $$\leq E[R\_{t+1}+\gamma R\_{t+2}+....|S\_t=s]$$
  $$=V\pi^∗(s)$$
  
  I would say that our final two lines are in fact inequalities, and for me this makes intuitive sense in that if we are always taking a deterministic greedy action our value function and Q function are the same. As detailed here, for a given policy and state we have that $V(s)=\sum\_a \pi(a|s)Q\_{\pi}(s,a)$ and if the policy is optimal and hence greedy then $\pi(a|s)$ is deterministic.
  
  **Tags:** reinforcement-learning, comparison, value-functions, bellman-equations
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-15. Score: 1 | Accepted answer
  
  In general they are *not* the same and that should be clear as to why -- mathematically you are conditioning on an extra random variable being known in the state-action value function. You have the correct relationship between them, but I think your understanding of the two may be slightly off. The state-action value function is a function of both $s$ and $a$ whereas the value function is a function of just $s$. As you have noted, the value function is equal to the expected value of the state-action value function, with the expectation taken over the action distribution induced by the policy.
  
  However, if you have a deterministic policy, then two are equivalent *only if you evaluate the state-action value function at the action that the deterministic policy gives*. That is, $v\_\pi(s) = Q(s, \pi(s))$. This is because, as you say, for a given state the policy will assign probability one to a certain action and 0 to all other actions in the action space and so the equation I just wrote is what the expectation in your question reduces to.
  
  Now, you might think 'well the policy is deterministic so we are always going to choose this action, so my theory was correct'. This is not true. The definition of the state-action value function is the value of the expected future (possibly discounted) returns given that we are in state $s$ and given that we have taken action $a$. This means that we have already chosen our action and we are *not* choosing it according to the policy -- the policy will only choose all *future* actions.
  
  You also say that "If we are operating a greedy policy and hence acting optimally" which is not true. Just because we act greedily it does *not* imply we are acting optimally. Value functions are functions of a policy. If the policy is completely random then you would get certain value and state-action value functions which you could act greedily to but this would not be optimal.
  
  > **Comments:**
  > **username_0**: Hi @DavidIreland I completely agree about the last paragraph. What I was trying to say was that acting greedily WITH an optimal policy then we would be acting optimally. Would this be correct? Given we have found an optimal policy and there is no noise in the environment if we act greedily then it should also be optimal, correct?  
  > **username_0**: Your other points also makes sense, we are saying that in fact they wont be equivalent because although we evaluate all future steps with the optimal policy, the current step was chosen for this state, as defined by the definition of state action values. Is this correct?  
  > **username_1**: If you are acting greedily according to optimal action-value functions then yes you are acting optimally.   with regards to the second comment, yes that is indeed what we are saying. All future steps would agree except the returns would differ depending on what the current action taken was.  
  
  ---
  
  