<p>This is from Sutton-Barto, second edt, page 126:</p>
<blockquote>
<p>Suppose there is available only a finite amount of experience, say 10 episodes or 100
time steps. In this case, a common approach with incremental learning methods is to
present the experience repeatedly until the method converges upon an answer. Given an
approximate value function, V , the increments specified by (6.1) or (6.2) are computed
for every time step t at which a nonterminal state is visited, but the value function is
changed only once, by the sum of all the increments. Then all the available experience is
processed again with the new value function to produce a new overall increment, and so
on, until the value function converges. We call this batch updating because updates are
made only after processing each complete batch of training data.</p>
</blockquote>
<p>However, there is no pseudocode for &quot;batch TD(0)&quot;. Could an expert give it or the source to find it?</p>

CC BY-SA 4.0
2024-04-20T00:05:23.757Z
Is https://stats.stackexchange.com/questions/297708/batch-reinforcement-learning-algorithm-example what you are looking for?
Neil Slater
CC BY-SA 4.0
2024-04-20T00:08:41.137Z
not exactly, I need a better and sure answer. I mean the pseudocode for batch TD(0).
DSPinfinity
CC BY-SA 4.0
2024-04-20T00:10:51.763Z
TD(0) isn't really an algorithm, the answer I linked shows a batch SARSA(0) which is a fully-realised batch TD(0) model-free control method. What will you be using this for? If for control, will it be model-based?
Neil Slater
CC BY-SA 4.0
2024-04-20T00:13:25.510Z
I think I have found a source (see page 31 of the this link): https://www.khoury.northeastern.edu/home/rplatt/cs7180_fall2018/slides/temporal_difference.pdf
DSPinfinity
CC BY-SA 4.0
2024-04-20T00:15:40.003Z
Yes that will evaluate the value function for an observed policy using temporal difference, assuming you have a condition you can insert for "until converged". It also matches S&B's description that you quoted. Is that what you want to do?
Neil Slater
CC BY-SA 4.0
2024-04-20T00:19:55.293Z
Yes that is what I wanted but this source is still not clear for me:  (1) it is not clear how  of the value function is changed only once by the sum of all the increments.  (2) Based on this source, we may have V'(s)=a and V'(s)=b. Then, is the last one valid? I am totally confused. I think this source is not correct.
DSPinfinity
CC BY-SA 4.0
2024-04-20T00:29:04.527Z
Yes you seem to be right, the algorithm in the PDF is discarding experience when $s$ is repreated in the dataset, only using the last one. It works as per S&B only if $s$ is not repeated in the batch, or if some additional data structure not shown is assumed to allow increments and multiple changes to the same $V'(s)$
Neil Slater
CC BY-SA 4.0
2024-04-20T00:34:01.187Z
so, still I wonder the exact batch TD(0) algorithm meant in  S&B.
DSPinfinity
<sutton-barto><temporal-difference-methods>
Pseudocode for batch TD(0)
DSPinfinity
# Pseudocode for batch TD(0)

(Asked by: username_0 on 2024-04-20)

This is from Sutton-Barto, second edt, page 126:

> Suppose there is available only a finite amount of experience, say 10 episodes or 100
> time steps. In this case, a common approach with incremental learning methods is to
> present the experience repeatedly until the method converges upon an answer. Given an
> approximate value function, V , the increments specified by (6.1) or (6.2) are computed
> for every time step t at which a nonterminal state is visited, but the value function is
> changed only once, by the sum of all the increments. Then all the available experience is
> processed again with the new value function to produce a new overall increment, and so
> on, until the value function converges. We call this batch updating because updates are
> made only after processing each complete batch of training data.

However, there is no pseudocode for "batch TD(0)". Could an expert give it or the source to find it?

**Tags:** sutton-barto, temporal-difference-methods

> **Comments:**
> **username_1**: Is https://stats.stackexchange.com/questions/297708/batch-reinforcement-learning-algorithm-example what you are looking for?  
> **username_0**: not exactly, I need a better and sure answer. I mean the pseudocode for batch TD(0).  
> **username_1**: TD(0) isn't really an algorithm, the answer I linked shows a batch SARSA(0) which is a fully-realised batch TD(0) model-free control method. What will you be using this for? If for control, will it be model-based?  
> **username_0**: I think I have found a source (see page 31 of the this link): https://www.khoury.northeastern.edu/home/rplatt/cs7180_fall2018/slides/temporal_difference.pdf  
> **username_1**: Yes that will evaluate the value function for an observed policy using temporal difference, assuming you have a condition you can insert for "until converged". It also matches S&B's description that you quoted. Is that what you want to do?  
> **username_0**: Yes that is what I wanted but this source is still not clear for me:  (1) it is not clear how  of the value function is changed only once by the sum of all the increments.  (2) Based on this source, we may have V'(s)=a and V'(s)=b. Then, is the last one valid? I am totally confused. I think this source is not correct.  
> **username_1**: Yes you seem to be right, the algorithm in the PDF is discarding experience when $s$ is repreated in the dataset, only using the last one. It works as per S&B only if $s$ is not repeated in the batch, or if some additional data structure not shown is assumed to allow increments and multiple changes to the same $V'(s)$  
> **username_0**: so, still I wonder the exact batch TD(0) algorithm meant in  S&B.  

---

## 0 answers

