Answers.Body: <p>The true value <span class="math-container">$v_{\pi}(s)$</span> is a conceptual target for the <span class="math-container">$\overline{VE}$</span> in the book. You often do not know it in real problems. However, it is still used in two main ways in the book:</p>
  
  <ul>
  <li><p>Theoretically for analysis of different aprpoximation schemes, which can be shown to converge to minimise the <span class="math-container">$\overline{VE}$</span> objective, or a related one. </p></li>
  <li><p>In toy problems when exploring the nature of approximation in Reinforcement Learning (RL), it is possible to use tabular methods guranateed to get close to zero error, and then compare them to approximate methods. There are several plots of this type in the book.</p></li>
  </ul>
  
  <p>The book shows derivation of gradient descent methods that start with minimising <span class="math-container">$\overline{VE}$</span> as an objective, and that use <em>samples</em> of <span class="math-container">$v_{\pi}(s)$</span> such as Monte Carlo return <span class="math-container">$G_t$</span> or the TD target) in place of the unknown <span class="math-container">$v_{\pi}(s)$</span> in the loss function. These also rely on the fact that the sample distribution will be weighted by <span class="math-container">$\mu(s)$</span> if they are taken naturally from the environment whilst the agent is following the policy <span class="math-container">$\pi$</span> so that <span class="math-container">$\mu(s)$</span> also does not need to be explicitly known. </p>
  
  <p>Outside of toy problems deliberately constructed to demonstrate that this theory is correct, you will not know <span class="math-container">$v_{\pi}(s)$</span> or be able to calculate <span class="math-container">$\overline{VE}$</span>. However, you will know from the theory that if you follow the update rules derived in the book for approximate gradient descent methods, that the process should find a local minimum for <span class="math-container">$\overline{VE}$</span>, for whatever state approximation scheme you have chosen to use.</p>
  
  <p>Usually you cannot even <em>approximate</em> <span class="math-container">$\overline{VE}$</span> from raw data, as the variance in returns will add noise to the signal, and there is no way to separate variance in returns from error in approximation in the general case. However there are a couple of scenarios that do lend themselves to measuring this objective, provided you already have your estimate <span class="math-container">$\hat{v}(s,w)$</span> and the policy remains fixed throughout:</p>
  
  <ul>
  <li><p>Simple, fast (perhaps simulated), environments which can be solved to arbitrary accuracy using tabular methods. In this case, you first calculate <span class="math-container">$v_{\pi}(s)$</span> using a non-approximate method, then sample many approximations by running the environment using policy <span class="math-container">$\pi$</span> and treating that as your data set.</p></li>
  <li><p>Fully deterministic environments where <span class="math-container">$\pi$</span> is also deterministic. These have a variance of <span class="math-container">$0$</span> for Monte Carlo returns, so each observed return from any given state is already the true value of <span class="math-container">$v_{\pi}(s)$</span>. Again you can just run the environment many times to get your data set to calculate  <span class="math-container">$v_{\pi}(s)$</span> and <span class="math-container">$\hat{v}(s,w)$</span> for the observed states in the correct frequencies, and thus have data to calculate <span class="math-container">$\overline{VE}$</span>.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-07T16:53:03.427Z
Answers.Text: This is a great description, thank you for going through the documentation and fleshing this out.
Answers.UserDisplayName: Zakk Diaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-05T09:26:37.997Z
Answers.LastActivityDate: 2019-10-05T11:00:54.360Z
Answers.LastEditDate: 2019-10-05T11:00:54.360Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I may be wrong about this but my best interpretation without having access to the book is that</p>
  
  <p><em>How do I get the true value vπ(s)?</em> </p>
  
  <p>The True value I think is whatever the most correct answer is for the prediction. This should be training data. Some companies like facebook spend a lot of money to hire people to create hand-detailed data to fill in this value.</p>
  
  <p><em>And if there is a way to obtain the value, why would I need to approximate it?</em></p>
  
  <p>You are approximating it to test the accuracy of your model - your prediction. It seems to me this equation is only necessary when training the model.</p>
  
  <p>The result of this is your total error between all your predictions. The lower the value, the better your model.
  <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Mean_squared_error</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-28T19:09:59.700Z
Answers.Text: So the target (which from my understanding does not has to be the true value) is referenced there as "true" value?
Answers.UserDisplayName: F.M.F.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-28T19:12:52.517Z
Answers.Text: It's hard to say without the book. The target value & true value seem like technical terms in which I would be wrong trying to assume their intention.
Answers.UserDisplayName: Zakk Diaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-28T19:16:31.013Z
Answers.Text: Okay. Thank you! You got my upvote but I will wait before accepting, if anyone has a more specific answer!
Answers.UserDisplayName: F.M.F.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-05T09:28:01.867Z
Answers.Text: "Some companies like facebook spend a lot of money to hire people to create hand-detailed data to fill in this value" this is not something anyone can do for complex RL problems, even with the resources of Facebook. Instead proxy values are used and VE is unknown. Yet it can still be used as an objective, and analysed to create update rules. Also, you do have access to the book. It is freely available online as HTML and a PDF: http://incompleteideas.net/book/the-book.html
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-28T18:27:52.987Z
Answers.LastActivityDate: 2019-05-28T18:27:52.987Z
Answers.UserDisplayName: Zakk Diaz
Body: <p>In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective (<span class="math-container">$\overline{VE}$</span>) as follows (page 199):
  <span class="math-container">$$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{\pi}(s)-\hat{v}(s,w)]^2$$</span>
  Where <span class="math-container">$v_{\pi}(s)$</span> is the true value of <span class="math-container">$s$</span> and <span class="math-container">$\hat{v}(s,w)$</span> is the approximation of it. Furthermore it is stated that this is "often used in plots".</p>
  
  <p><em>How do I get the true value <span class="math-container">$v_{\pi}(s)$</span>? And if there is a way to obtain the value, why would I need to approximate it?</em></p>
  
Tags: <reinforcement-learning><objective-functions><sutton-barto>
Title: How do we get the true value in the prediction objective in reinforcement learning?
UserDisplayName: F.M.F.
ThreadText: # How do we get the true value in the prediction objective in reinforcement learning?
  
  (Asked by: username_0 on 2019-05-28)
  
  In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199):
  $$\overline{VE}\doteq\sum\_{s\epsilon S} \mu(s)[v\_{\pi}(s)-\hat{v}(s,w)]^2$$
  Where $v\_{\pi}(s)$ is the true value of $s$ and $\hat{v}(s,w)$ is the approximation of it. Furthermore it is stated that this is "often used in plots".
  
  *How do I get the true value $v\_{\pi}(s)$? And if there is a way to obtain the value, why would I need to approximate it?*
  
  **Tags:** reinforcement-learning, objective-functions, sutton-barto
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-05. Score: 1 | Accepted answer
  
  The true value $v\_{\pi}(s)$ is a conceptual target for the $\overline{VE}$ in the book. You often do not know it in real problems. However, it is still used in two main ways in the book:
  
  * Theoretically for analysis of different aprpoximation schemes, which can be shown to converge to minimise the $\overline{VE}$ objective, or a related one.
  * In toy problems when exploring the nature of approximation in Reinforcement Learning (RL), it is possible to use tabular methods guranateed to get close to zero error, and then compare them to approximate methods. There are several plots of this type in the book.
  
  The book shows derivation of gradient descent methods that start with minimising $\overline{VE}$ as an objective, and that use *samples* of $v\_{\pi}(s)$ such as Monte Carlo return $G\_t$ or the TD target) in place of the unknown $v\_{\pi}(s)$ in the loss function. These also rely on the fact that the sample distribution will be weighted by $\mu(s)$ if they are taken naturally from the environment whilst the agent is following the policy $\pi$ so that $\mu(s)$ also does not need to be explicitly known.
  
  Outside of toy problems deliberately constructed to demonstrate that this theory is correct, you will not know $v\_{\pi}(s)$ or be able to calculate $\overline{VE}$. However, you will know from the theory that if you follow the update rules derived in the book for approximate gradient descent methods, that the process should find a local minimum for $\overline{VE}$, for whatever state approximation scheme you have chosen to use.
  
  Usually you cannot even *approximate* $\overline{VE}$ from raw data, as the variance in returns will add noise to the signal, and there is no way to separate variance in returns from error in approximation in the general case. However there are a couple of scenarios that do lend themselves to measuring this objective, provided you already have your estimate $\hat{v}(s,w)$ and the policy remains fixed throughout:
  
  * Simple, fast (perhaps simulated), environments which can be solved to arbitrary accuracy using tabular methods. In this case, you first calculate $v\_{\pi}(s)$ using a non-approximate method, then sample many approximations by running the environment using policy $\pi$ and treating that as your data set.
  * Fully deterministic environments where $\pi$ is also deterministic. These have a variance of $0$ for Monte Carlo returns, so each observed return from any given state is already the true value of $v\_{\pi}(s)$. Again you can just run the environment many times to get your data set to calculate $v\_{\pi}(s)$ and $\hat{v}(s,w)$ for the observed states in the correct frequencies, and thus have data to calculate $\overline{VE}$.
  
  > **Comments:**
  > **username_2**: This is a great description, thank you for going through the documentation and fleshing this out.  
  
  ---
  
  ### Answer by username_2 on 2019-05-28. Score: 0
  
  I may be wrong about this but my best interpretation without having access to the book is that
  
  *How do I get the true value vπ(s)?*
  
  The True value I think is whatever the most correct answer is for the prediction. This should be training data. Some companies like facebook spend a lot of money to hire people to create hand-detailed data to fill in this value.
  
  *And if there is a way to obtain the value, why would I need to approximate it?*
  
  You are approximating it to test the accuracy of your model - your prediction. It seems to me this equation is only necessary when training the model.
  
  The result of this is your total error between all your predictions. The lower the value, the better your model.
  <https://en.wikipedia.org/wiki/Mean_squared_error>
  
  > **Comments:**
  > **username_0**: So the target (which from my understanding does not has to be the true value) is referenced there as "true" value?  
  > **username_2**: It's hard to say without the book. The target value & true value seem like technical terms in which I would be wrong trying to assume their intention.  
  > **username_0**: Okay. Thank you! You got my upvote but I will wait before accepting, if anyone has a more specific answer!  
  > **username_1**: "Some companies like facebook spend a lot of money to hire people to create hand-detailed data to fill in this value" this is not something anyone can do for complex RL problems, even with the resources of Facebook. Instead proxy values are used and VE is unknown. Yet it can still be used as an objective, and analysed to create update rules. Also, you do have access to the book. It is freely available online as HTML and a PDF: http://incompleteideas.net/book/the-book.html  
  
  ---
  
  