Body: <p>This is from Sutton-Barto, second edt, page 126:</p>
  <blockquote>
  <p>Suppose there is available only a finite amount of experience, say 10 episodes or 100
  time steps. In this case, a common approach with incremental learning methods is to
  present the experience repeatedly until the method converges upon an answer. Given an
  approximate value function, V , the increments specified by (6.1) or (6.2) are computed
  for every time step t at which a nonterminal state is visited, but the value function is
  changed only once, by the sum of all the increments. Then all the available experience is
  processed again with the new value function to produce a new overall increment, and so
  on, until the value function converges. We call this batch updating because updates are
  made only after processing each complete batch of training data.</p>
  </blockquote>
  <p>However, there is no pseudocode for &quot;batch TD(0)&quot;. Could an expert give it or the source to find it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:05:23.757Z
Comments.Text: Is https://stats.stackexchange.com/questions/297708/batch-reinforcement-learning-algorithm-example what you are looking for?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:08:41.137Z
Comments.Text: not exactly, I need a better and sure answer. I mean the pseudocode for batch TD(0).
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:10:51.763Z
Comments.Text: TD(0) isn't really an algorithm, the answer I linked shows a batch SARSA(0) which is a fully-realised batch TD(0) model-free control method. What will you be using this for? If for control, will it be model-based?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:13:25.510Z
Comments.Text: I think I have found a source (see page 31 of the this link): https://www.khoury.northeastern.edu/home/rplatt/cs7180_fall2018/slides/temporal_difference.pdf
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:15:40.003Z
Comments.Text: Yes that will evaluate the value function for an observed policy using temporal difference, assuming you have a condition you can insert for "until converged". It also matches S&B's description that you quoted. Is that what you want to do?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:19:55.293Z
Comments.Text: Yes that is what I wanted but this source is still not clear for me:  (1) it is not clear how  of the value function is changed only once by the sum of all the increments.  (2) Based on this source, we may have V'(s)=a and V'(s)=b. Then, is the last one valid? I am totally confused. I think this source is not correct.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:29:04.527Z
Comments.Text: Yes you seem to be right, the algorithm in the PDF is discarding experience when $s$ is repreated in the dataset, only using the last one. It works as per S&B only if $s$ is not repeated in the batch, or if some additional data structure not shown is assumed to allow increments and multiple changes to the same $V'(s)$
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-20T00:34:01.187Z
Comments.Text: so, still I wonder the exact batch TD(0) algorithm meant in  S&B.
Comments.UserDisplayName: DSPinfinity
Tags: <sutton-barto><temporal-difference-methods>
Title: Pseudocode for batch TD(0)
UserDisplayName: DSPinfinity
ThreadText: # Pseudocode for batch TD(0)
  
  (Asked by: username_0 on 2024-04-20)
  
  This is from Sutton-Barto, second edt, page 126:
  
  > Suppose there is available only a finite amount of experience, say 10 episodes or 100
  > time steps. In this case, a common approach with incremental learning methods is to
  > present the experience repeatedly until the method converges upon an answer. Given an
  > approximate value function, V , the increments specified by (6.1) or (6.2) are computed
  > for every time step t at which a nonterminal state is visited, but the value function is
  > changed only once, by the sum of all the increments. Then all the available experience is
  > processed again with the new value function to produce a new overall increment, and so
  > on, until the value function converges. We call this batch updating because updates are
  > made only after processing each complete batch of training data.
  
  However, there is no pseudocode for "batch TD(0)". Could an expert give it or the source to find it?
  
  **Tags:** sutton-barto, temporal-difference-methods
  
  > **Comments:**
  > **username_1**: Is https://stats.stackexchange.com/questions/297708/batch-reinforcement-learning-algorithm-example what you are looking for?  
  > **username_0**: not exactly, I need a better and sure answer. I mean the pseudocode for batch TD(0).  
  > **username_1**: TD(0) isn't really an algorithm, the answer I linked shows a batch SARSA(0) which is a fully-realised batch TD(0) model-free control method. What will you be using this for? If for control, will it be model-based?  
  > **username_0**: I think I have found a source (see page 31 of the this link): https://www.khoury.northeastern.edu/home/rplatt/cs7180_fall2018/slides/temporal_difference.pdf  
  > **username_1**: Yes that will evaluate the value function for an observed policy using temporal difference, assuming you have a condition you can insert for "until converged". It also matches S&B's description that you quoted. Is that what you want to do?  
  > **username_0**: Yes that is what I wanted but this source is still not clear for me:  (1) it is not clear how  of the value function is changed only once by the sum of all the increments.  (2) Based on this source, we may have V'(s)=a and V'(s)=b. Then, is the last one valid? I am totally confused. I think this source is not correct.  
  > **username_1**: Yes you seem to be right, the algorithm in the PDF is discarding experience when $s$ is repreated in the dataset, only using the last one. It works as per S&B only if $s$ is not repeated in the batch, or if some additional data structure not shown is assumed to allow increments and multiple changes to the same $V'(s)$  
  > **username_0**: so, still I wonder the exact batch TD(0) algorithm meant in  S&B.  
  
  ---
  
  ## 0 answers
  
  