Answers.Body: <p>I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any theoretical foundation. But nowadays,  I understand it simply as a mean's calculation, using the recurrent formula that states that when you a have a mean and a new value arrives, it modifies the mean by an amount equal to its difference with it (the mean) divided by the new values number.</p>
  <p>To summarize, the exposed mean calculation is an instance of a general formula of recurrent mean calculation that uses as increasing factor for the difference between the new value and the actual mean multiplied by any number between 0 and 1. By the way, this number - usually called  <em>step size parameter</em> - can be dynamic, and in the first paragraph (usual mean calculation) its amount is the inverse of the number of values considered in the mean's calculation.</p>
  <p>Intuitively, we can understand that it is an accurate estimation procedure independently the initial (guessed or not) value. With a high number of estimates (new values arriving), the initial values fades its importance, and that it can be extended to treat many (lambda) new values simultaneously.</p>
  <p>Until now I have not found this explanation nowhere, even if it is very simple, and I am not so sure that it is sound. I will appreciate if someone let me know if this intuition is correct and if it has already been exposed somewhere.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T14:35:58.553Z
Answers.LastActivityDate: 2021-03-14T23:17:34.447Z
Answers.LastEditDate: 2021-03-14T23:17:34.447Z
Answers.UserDisplayName: Hermes Morales
Answers.Body: <p>I would argue that maybe the word &quot;incremental&quot; could be better.</p>
  <p>Instead of directly comparing the Monte Carlo and the TD(<span class="math-container">$\lambda$</span>), I would like to compare the <span class="math-container">$\lambda$</span>-return and the TD(<span class="math-container">$\lambda$</span>) algorithm, since Monte Carlo is an extreme case of <span class="math-container">$\lambda$</span>-return method.</p>
  <ul>
  <li><p>The <span class="math-container">$\lambda$</span>-return algorithm is a forward view algorithm, which means that it requires waiting up to <span class="math-container">$n$</span> steps after the current step to gather all the necessary information for updating the current state. This can be unsatisfactory because updates are not made in real time. An extreme case of this scenario is the <strong>Monte Carlo method</strong>, in which updates for all states are delayed until the end of an episode.</p>
  </li>
  <li><p>TD(<span class="math-container">$\lambda$</span>) is a backward view algorithm that can alleviate this issue. Unlike the <span class="math-container">$\lambda$</span>-return, which gathers all information at once and then performs a &quot;complete update&quot; of the current state via n-step bootstrapping, TD(<span class="math-container">$\lambda$</span>) performs &quot;<strong>incremental updates</strong>&quot; through eligibility traces. As it encounters a new state, it uses the new information to update those previous states that need this piece of information. Since those previous states might need more information than this single piece, they are &quot;incrementally updated&quot; or &quot;partially updated&quot; by this new information. As the process proceeds, the information that a previous state needs to complete its update will eventually be gathered completely, allowing for a complete update of that state.</p>
  </li>
  </ul>
  <p><em>Chapter 12 - Eligibility Traces</em> of Sutton and Barto's <a href="https://mitpress.mit.edu/9780262039246/reinforcement-learning/" rel="nofollow noreferrer">book</a> provides more accurate information about this topic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-29T08:04:55.240Z
Answers.LastActivityDate: 2024-06-29T08:04:55.240Z
Answers.UserDisplayName: Jiangpeng Li
Answers.Body: <p>TD(<span class="math-container">$\lambda$</span>) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.</p>
  <p>More precisely, TD(<span class="math-container">$\lambda$</span>) is temporal-difference learning with a <span class="math-container">$\lambda$</span>-return, which is defined as an average of all <span class="math-container">$n$</span>-step returns, for all <span class="math-container">$n$</span>, where an <span class="math-container">$n$</span>-step return is the <em>target</em> used to update the estimate of the value function that contains <span class="math-container">$n$</span> future rewards (plus an estimate of the value function of the state <span class="math-container">$n$</span> steps in the future). For example, TD(0) (e.g. Q-learning is usually presented as a TD(0) method) uses a <span class="math-container">$1$</span>-step return, that is, it uses one future reward (plus an estimate of the value of the next state) to compute the target. The letter <span class="math-container">$\lambda$</span> actually refers to a parameter used in this context to weigh the combination of TD and MC methods. There are actually two different perspectives of TD(<span class="math-container">$\lambda$</span>), the <em>forward view</em> and the <em>backward view</em> (eligibility traces).</p>
  <p>The blog post <a href="https://amreis.github.io/ml/reinf-learn/2017/11/02/reinforcement-learning-eligibility-traces.html" rel="nofollow noreferrer">Reinforcement Learning: Eligibility Traces and TD(lambda)</a> gives a quite intuitive overview of TD(<span class="math-container">$\lambda$</span>), and, for more details, read <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=309" rel="nofollow noreferrer">the related chapter of the book <em>Reinforcement Learning: An Introduction</em></a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T03:01:45.903Z
Answers.LastActivityDate: 2021-03-10T17:45:20.583Z
Answers.LastEditDate: 2021-03-10T17:45:20.583Z
Answers.UserDisplayName: nbro
Body: <p>I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD(<span class="math-container">$\lambda$</span>) as a type of "truncated" Monte Carlo learning?</p>
  
Tags: <reinforcement-learning><comparison><monte-carlo-methods><temporal-difference-methods><td-lambda>
Title: What is the intuition behind TD($\lambda$)?
UserDisplayName: Nick Kunz
ThreadText: # What is the intuition behind TD($\lambda$)?
  
  (Asked by: username_0 on 2020-01-21)
  
  I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD($\lambda$) as a type of "truncated" Monte Carlo learning?
  
  **Tags:** reinforcement-learning, comparison, monte-carlo-methods, temporal-difference-methods, td-lambda
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2020-01-22. Score: 5 | Accepted answer
  
  TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.
  
  More precisely, TD($\lambda$) is temporal-difference learning with a $\lambda$-return, which is defined as an average of all $n$-step returns, for all $n$, where an $n$-step return is the *target* used to update the estimate of the value function that contains $n$ future rewards (plus an estimate of the value function of the state $n$ steps in the future). For example, TD(0) (e.g. Q-learning is usually presented as a TD(0) method) uses a $1$-step return, that is, it uses one future reward (plus an estimate of the value of the next state) to compute the target. The letter $\lambda$ actually refers to a parameter used in this context to weigh the combination of TD and MC methods. There are actually two different perspectives of TD($\lambda$), the *forward view* and the *backward view* (eligibility traces).
  
  The blog post [Reinforcement Learning: Eligibility Traces and TD(lambda)](https://amreis.github.io/ml/reinf-learn/2017/11/02/reinforcement-learning-eligibility-traces.html) gives a quite intuitive overview of TD($\lambda$), and, for more details, read [the related chapter of the book *Reinforcement Learning: An Introduction*](http://incompleteideas.net/book/RLbook2020.pdf#page=309).
  
  
  ---
  
  ### Answer by username_1 on 2021-03-10. Score: 2
  
  I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any theoretical foundation. But nowadays, I understand it simply as a mean's calculation, using the recurrent formula that states that when you a have a mean and a new value arrives, it modifies the mean by an amount equal to its difference with it (the mean) divided by the new values number.
  
  To summarize, the exposed mean calculation is an instance of a general formula of recurrent mean calculation that uses as increasing factor for the difference between the new value and the actual mean multiplied by any number between 0 and 1. By the way, this number - usually called *step size parameter* - can be dynamic, and in the first paragraph (usual mean calculation) its amount is the inverse of the number of values considered in the mean's calculation.
  
  Intuitively, we can understand that it is an accurate estimation procedure independently the initial (guessed or not) value. With a high number of estimates (new values arriving), the initial values fades its importance, and that it can be extended to treat many (lambda) new values simultaneously.
  
  Until now I have not found this explanation nowhere, even if it is very simple, and I am not so sure that it is sound. I will appreciate if someone let me know if this intuition is correct and if it has already been exposed somewhere.
  
  
  ---
  
  ### Answer by username_2 on 2024-06-29. Score: 1
  
  I would argue that maybe the word "incremental" could be better.
  
  Instead of directly comparing the Monte Carlo and the TD($\lambda$), I would like to compare the $\lambda$-return and the TD($\lambda$) algorithm, since Monte Carlo is an extreme case of $\lambda$-return method.
  
  * The $\lambda$-return algorithm is a forward view algorithm, which means that it requires waiting up to $n$ steps after the current step to gather all the necessary information for updating the current state. This can be unsatisfactory because updates are not made in real time. An extreme case of this scenario is the **Monte Carlo method**, in which updates for all states are delayed until the end of an episode.
  * TD($\lambda$) is a backward view algorithm that can alleviate this issue. Unlike the $\lambda$-return, which gathers all information at once and then performs a "complete update" of the current state via n-step bootstrapping, TD($\lambda$) performs "**incremental updates**" through eligibility traces. As it encounters a new state, it uses the new information to update those previous states that need this piece of information. Since those previous states might need more information than this single piece, they are "incrementally updated" or "partially updated" by this new information. As the process proceeds, the information that a previous state needs to complete its update will eventually be gathered completely, allowing for a complete update of that state.
  
  *Chapter 12 - Eligibility Traces* of Sutton and Barto's [book](https://mitpress.mit.edu/9780262039246/reinforcement-learning/) provides more accurate information about this topic.
  
  
  ---
  
  