<p>In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.</p>

CC BY-SA 4.0
2020-05-09T09:01:01.363Z
What you mean by saying expectation will be approximated by Monte Carlo?
Swakshar Deb
CC BY-SA 4.0
2020-05-10T09:41:33.797Z
You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment.
David
CC BY-SA 4.0
2020-05-08T22:19:56.180Z
2020-05-08T22:19:56.180Z
David
<p>I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation. </p>

<p>I marked this point in this image.</p>

<p><a href="https://i.sstatic.net/idsbt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/idsbt.png" alt="enter image description here"></a></p>

CC BY-SA 4.0
2020-04-27T15:24:45.160Z
This is just a simple SGD algorithm so one sample is enough.
bitWise
<machine-learning><deep-learning><reinforcement-learning><math><deep-rl>
How can a single sample represent the expectation in gradient temporal difference learning?
Swakshar Deb
# How can a single sample represent the expectation in gradient temporal difference learning?

(Asked by: username_0 on 2020-04-26)

I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation.

I marked this point in this image.

[![enter image description here](https://i.sstatic.net/idsbt.png)](https://i.sstatic.net/idsbt.png)

**Tags:** machine-learning, deep-learning, reinforcement-learning, math, deep-rl

> **Comments:**
> **username_1**: This is just a simple SGD algorithm so one sample is enough.  

---

## 1 answers

### Answer by username_2 on 2020-05-08. Score: 2

In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.

> **Comments:**
> **username_0**: What you mean by saying expectation will be approximated by Monte Carlo?  
> **username_2**: You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment.  

---

