Answers.Body: <p>Ok, Q is the reward associated with being in a given state, following a certain action and then following the given policy.</p>
  
  <p>You need to take the expectation of the sum the immediate reward plus the value function which is defined by the policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T01:05:35.663Z
Answers.LastActivityDate: 2020-04-16T01:05:35.663Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>First of all, <span class="math-container">$Q_\pi(s, a)$</span> IS DEFINED AS the value (i.e. the expected return) of taking some action <span class="math-container">$a$</span> in some state <span class="math-container">$s$</span>, AND THEN following some given policy <span class="math-container">$\pi$</span> (until e.g. the end of the game or your life). In other words, suppose that you take action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>, AND THEN use the policy <span class="math-container">$\pi$</span> to behave in the world until you die, then <span class="math-container">$Q_\pi(s, a)$</span> would represent the value that you would obtain.</p>
  
  <p>So, we are DEFINING <span class="math-container">$Q_\pi(s, a)$</span> in a certain way.  This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.</p>
  
  <blockquote>
    <p>From this, I infer that the <span class="math-container">$Q$</span> value (the action-value function) will be affected by the policy <span class="math-container">$\pi$</span>.</p>
  </blockquote>
  
  <p>So, <span class="math-container">$Q_\pi(s, a)$</span> will not keep changing. You could say that <span class="math-container">$Q_\pi(s, a)$</span> (which is a function) is "affected by" <span class="math-container">$\pi$</span> ONLY in the sense that it is "defined in terms of" <span class="math-container">$\pi$</span>. To be precise, <span class="math-container">$Q_\pi(s, a)$</span> is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.</p>
  
  <blockquote>
    <p>Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?</p>
  </blockquote>
  
  <p>Again, there's the distinction between the algorithm that you actually use to find the function <span class="math-container">$Q_\pi(s, a)$</span> and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Reinforcement learning: an introduction</a> by Sutton and Barto).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T01:41:47.133Z
Answers.LastActivityDate: 2020-04-16T01:41:47.133Z
Answers.UserDisplayName: nbro
Body: <p>From my understanding, the policy <span class="math-container">$\pi$</span> is basically how the agent acts (i.e. the actions it will take in each state).</p>
  
  <p>However, I am confused about the Q value and how it is "affected" by a policy. <a href="https://datascience.stackexchange.com/a/31791/10640">This answer</a> says</p>
  
  <blockquote>
    <p><span class="math-container">$Q^\pi(s, a)$</span> is the action-value function. It is the expected return starting from state <span class="math-container">$s$</span>, following policy <span class="math-container">$\pi$</span>, taking action <span class="math-container">$a$</span>. It's focusing on the particular action at the particular state.</p>
  </blockquote>
  
  <p>From this, I infer that the <span class="math-container">$Q$</span> value (the action-value function) will be affected by the policy <span class="math-container">$\pi$</span>. Why? So, why does the Q value change according to policy <span class="math-container">$\pi$</span>? </p>
  
  <p>Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)? </p>
  
  <p>All the policy does is find out the max Q values and bases its policy on that information.</p>
  
Tags: <reinforcement-learning><q-learning><policies><value-functions>
Title: Why does the policy $\pi$ affect the Q value?
UserDisplayName: BG10
ThreadText: # Why does the policy $\pi$ affect the Q value?
  
  (Asked by: username_0 on 2020-04-16)
  
  From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).
  
  However, I am confused about the Q value and how it is "affected" by a policy. [This answer](https://datascience.stackexchange.com/a/31791/10640) says
  
  > $Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.
  
  From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$. Why? So, why does the Q value change according to policy $\pi$?
  
  Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?
  
  All the policy does is find out the max Q values and bases its policy on that information.
  
  **Tags:** reinforcement-learning, q-learning, policies, value-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-04-16. Score: 2 | Accepted answer
  
  First of all, $Q\_\pi(s, a)$ IS DEFINED AS the value (i.e. the expected return) of taking some action $a$ in some state $s$, AND THEN following some given policy $\pi$ (until e.g. the end of the game or your life). In other words, suppose that you take action $a$ in state $s$, AND THEN use the policy $\pi$ to behave in the world until you die, then $Q\_\pi(s, a)$ would represent the value that you would obtain.
  
  So, we are DEFINING $Q\_\pi(s, a)$ in a certain way. This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.
  
  > From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$.
  
  So, $Q\_\pi(s, a)$ will not keep changing. You could say that $Q\_\pi(s, a)$ (which is a function) is "affected by" $\pi$ ONLY in the sense that it is "defined in terms of" $\pi$. To be precise, $Q\_\pi(s, a)$ is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.
  
  > Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?
  
  Again, there's the distinction between the algorithm that you actually use to find the function $Q\_\pi(s, a)$ and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book [Reinforcement learning: an introduction](http://incompleteideas.net/book/RLbook2020.pdf) by Sutton and Barto).
  
  
  ---
  
  ### Answer by username_1 on 2020-04-16. Score: 0
  
  Ok, Q is the reward associated with being in a given state, following a certain action and then following the given policy.
  
  You need to take the expectation of the sum the immediate reward plus the value function which is defined by the policy.
  
  
  ---
  
  