Answers.Body: <p>In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T09:01:01.363Z
Answers.Text: What you mean by saying expectation will be approximated by Monte Carlo?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-10T09:41:33.797Z
Answers.Text: You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-08T22:19:56.180Z
Answers.LastActivityDate: 2020-05-08T22:19:56.180Z
Answers.UserDisplayName: David
Body: <p>I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation. </p>
  
  <p>I marked this point in this image.</p>
  
  <p><a href="https://i.sstatic.net/idsbt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/idsbt.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T15:24:45.160Z
Comments.Text: This is just a simple SGD algorithm so one sample is enough.
Comments.UserDisplayName: bitWise
Tags: <machine-learning><deep-learning><reinforcement-learning><math><deep-rl>
Title: How can a single sample represent the expectation in gradient temporal difference learning?
UserDisplayName: Swakshar Deb
ThreadText: # How can a single sample represent the expectation in gradient temporal difference learning?
  
  (Asked by: username_0 on 2020-04-26)
  
  I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation.
  
  I marked this point in this image.
  
  [![enter image description here](https://i.sstatic.net/idsbt.png)](https://i.sstatic.net/idsbt.png)
  
  **Tags:** machine-learning, deep-learning, reinforcement-learning, math, deep-rl
  > **Comments:**
  > **username_1**: This is just a simple SGD algorithm so one sample is enough.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-08. Score: 2
  
  In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.
  
  > **Comments:**
  > **username_0**: What you mean by saying expectation will be approximated by Monte Carlo?  
  > **username_2**: You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment.  
  
  ---
  
  