Body: <p>I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far:</p>
  <ul>
  <li><p>TD(0): increment learning, can learn after each step instead of waiting for the episode to end. Update bases on one reward, then low variance. However, high bias.</p>
  </li>
  <li><p>MC: Learn after each episode, the calculation of return is correct. However, its drawback is high variance. And you have to make decisions in the whole episode without an update of parameters.</p>
  </li>
  <li><p>GAE: combine returns in all steps, get a better trade-off between variance and bias. However, still has to wait until the end of the episode for an update.</p>
  </li>
  </ul>
  <p>I have some questions as follows:</p>
  <ol>
  <li><p>Is variance and bias about the return of each episode? What are their effects on the outcomes (convergence speed of training process, performance of the model)?</p>
  </li>
  <li><p>Is increment learning important? The ability to correct behaviors after each step may improve convergence speed. However, it can lead to unstable learning ( if I understand correctly, this is why the Target model in Double DQL only updates its parameters for each k mini-batches ). Which scenarios should I use TD(0) or GAE?</p>
  </li>
  <li><p>Concretely, in my case, I run parallelly a batch with 12 environments, each with 1000 steps. If I use GAE, I make 12000 decisions for each update. All losses of the model are summed up and calculate gradients, after that, I clip gradients to 2.0. Is that too expensive to learn the correct direction? Should I consider using TD(0) here?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-04T10:07:04.083Z
Comments.Text: Hello. Could you please edit your post to focus on and ask only 1 specific question? If you have multiple questions, split the post into multiple ones.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-06T15:19:16.030Z
Comments.Text: @nbro Thanks for your advice, but actually there are three related questions. I combine them with the expectation of helping others better understand the problem I have. Anyway, question 3 is the one that I want to answer most now.
Comments.UserDisplayName: Ngoc Bui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-06T18:06:15.593Z
Comments.Text: I am suggesting you leave 1 question because this also simplifies the life of readers and people that may attempt to answer your question. Even if they are related, I would still suggest this.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><training><monte-carlo-methods><temporal-difference-methods>
Title: Comparison between TD(0) and MC ( or GAE )?
UserDisplayName: Ngoc Bui
ThreadText: # Comparison between TD(0) and MC ( or GAE )?
  
  (Asked by: username_0 on 2021-05-04)
  
  I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far:
  
  * TD(0): increment learning, can learn after each step instead of waiting for the episode to end. Update bases on one reward, then low variance. However, high bias.
  * MC: Learn after each episode, the calculation of return is correct. However, its drawback is high variance. And you have to make decisions in the whole episode without an update of parameters.
  * GAE: combine returns in all steps, get a better trade-off between variance and bias. However, still has to wait until the end of the episode for an update.
  
  I have some questions as follows:
  
  1. Is variance and bias about the return of each episode? What are their effects on the outcomes (convergence speed of training process, performance of the model)?
  2. Is increment learning important? The ability to correct behaviors after each step may improve convergence speed. However, it can lead to unstable learning ( if I understand correctly, this is why the Target model in Double DQL only updates its parameters for each k mini-batches ). Which scenarios should I use TD(0) or GAE?
  3. Concretely, in my case, I run parallelly a batch with 12 environments, each with 1000 steps. If I use GAE, I make 12000 decisions for each update. All losses of the model are summed up and calculate gradients, after that, I clip gradients to 2.0. Is that too expensive to learn the correct direction? Should I consider using TD(0) here?
  
  **Tags:** reinforcement-learning, training, monte-carlo-methods, temporal-difference-methods
  > **Comments:**
  > **username_1**: Hello. Could you please edit your post to focus on and ask only 1 specific question? If you have multiple questions, split the post into multiple ones.  
  > **username_0**: @nbro Thanks for your advice, but actually there are three related questions. I combine them with the expectation of helping others better understand the problem I have. Anyway, question 3 is the one that I want to answer most now.  
  > **username_1**: I am suggesting you leave 1 question because this also simplifies the life of readers and people that may attempt to answer your question. Even if they are related, I would still suggest this.  
  
  ---
  
  ## 0 answers
  
  