Body: <p>I'm reading the article <a href="https://www.mit.edu/%7Ejnt/Papers/J063-97-bvr-td.pdf" rel="nofollow noreferrer">An Analysis of Temporal-Difference Learning
  with Function Approximation</a> (1997), but the mathematics inside seems overly complicated for me. Answers to some similar questions had pointed out that these proof typically involves stochastic approximation.</p>
  <p>My question is: are there any good tutorials (textbooks, list of papers, etc.) on stochastic approximation (or similar topics) that prepare you for reading proof like this? It will be best if it is rather self-contained under my mathematical maturity.</p>
  <p>I have an undergraduate-level mathematical analysis and probability theory foundation and have touched some measure theory.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-26T16:07:37.470Z
Comments.Text: This paper didn't even prove the hard results in SA. The book neurodynamic programming proves the basic SA result but, it uses a martingale convergence theorem which requires measure theory based probability. To be honest this is all kind of moot, nobody is going to be proving anything useful with NN based RL approximations.
Comments.UserDisplayName: FourierFlux
Tags: <reinforcement-learning><reference-request><convergence><temporal-difference-methods>
Title: Is there a tutorial for understanding the proof of convergence for TD learning?
UserDisplayName: heiyezhu6324
ThreadText: # Is there a tutorial for understanding the proof of convergence for TD learning?
  
  (Asked by: username_0 on 2021-10-26)
  
  I'm reading the article [An Analysis of Temporal-Difference Learning
  with Function Approximation](https://www.mit.edu/%7Ejnt/Papers/J063-97-bvr-td.pdf) (1997), but the mathematics inside seems overly complicated for me. Answers to some similar questions had pointed out that these proof typically involves stochastic approximation.
  
  My question is: are there any good tutorials (textbooks, list of papers, etc.) on stochastic approximation (or similar topics) that prepare you for reading proof like this? It will be best if it is rather self-contained under my mathematical maturity.
  
  I have an undergraduate-level mathematical analysis and probability theory foundation and have touched some measure theory.
  
  **Tags:** reinforcement-learning, reference-request, convergence, temporal-difference-methods
  > **Comments:**
  > **username_1**: This paper didn't even prove the hard results in SA. The book neurodynamic programming proves the basic SA result but, it uses a martingale convergence theorem which requires measure theory based probability. To be honest this is all kind of moot, nobody is going to be proving anything useful with NN based RL approximations.  
  
  ---
  
  ## 0 answers
  
  