Answers.Body: <p>The bias-variance trade-off that you're referring to has to do with the <strong>return estimator</strong>. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many sources of randomness, such as stochastic transitions or rewards.</p>
  <p>Monte Carlo RL algorithms estimate returns by running full trajectories and literally averaging the return achieved for each state. This imposes very few assumptions on the system (in fact, you don't even need the Markovian property for these methods), so bias is low. However, variance is high since each estimate depends on the literal trajectories that you observe. As such, you'll need many, many trajectories to get a good estimate of the value function.</p>
  <p>On the other hand, with TD methods, you estimate returns as <span class="math-container">$R_t + \gamma V(S_{t+1})$</span>, where <span class="math-container">$V$</span> is your estimate of the value function. Using <span class="math-container">$V$</span> this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value &quot;piecewise&quot; rather than just via literal trajectories that you happened to witness.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T20:42:55.063Z
Answers.Text: It may also be a good idea to mention [TD($\lambda$)](https://ai.stackexchange.com/q/17605/2444), so that people don't think that the distinction between the 2 approaches is always sharp.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T01:00:36.763Z
Answers.LastActivityDate: 2021-12-19T20:41:07.260Z
Answers.LastEditDate: 2021-12-19T20:41:07.260Z
Answers.UserDisplayName: harwiltz
Body: <p>I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-difference methods. I understood the reasoning behind that, but I wanted to know what one means when they refer to bias-variance tradeoff in RL.</p>
  <p>Is bias-variance trade-off used in the same way as in machine learning or deep learning?</p>
  <p>(I am just a beginner and have just started learning RL, so I apologize if it is a silly question.)</p>
  
Tags: <reinforcement-learning><deep-rl><monte-carlo-methods><temporal-difference-methods><bias-variance-tradeoff>
Title: What is the bias-variance trade-off in reinforcement learning?
UserDisplayName: Aman Savaria
ThreadText: # What is the bias-variance trade-off in reinforcement learning?
  
  (Asked by: username_0 on 2020-06-23)
  
  I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-difference methods. I understood the reasoning behind that, but I wanted to know what one means when they refer to bias-variance tradeoff in RL.
  
  Is bias-variance trade-off used in the same way as in machine learning or deep learning?
  
  (I am just a beginner and have just started learning RL, so I apologize if it is a silly question.)
  
  **Tags:** reinforcement-learning, deep-rl, monte-carlo-methods, temporal-difference-methods, bias-variance-tradeoff
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-24. Score: 3 | Accepted answer
  
  The bias-variance trade-off that you're referring to has to do with the **return estimator**. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many sources of randomness, such as stochastic transitions or rewards.
  
  Monte Carlo RL algorithms estimate returns by running full trajectories and literally averaging the return achieved for each state. This imposes very few assumptions on the system (in fact, you don't even need the Markovian property for these methods), so bias is low. However, variance is high since each estimate depends on the literal trajectories that you observe. As such, you'll need many, many trajectories to get a good estimate of the value function.
  
  On the other hand, with TD methods, you estimate returns as $R\_t + \gamma V(S\_{t+1})$, where $V$ is your estimate of the value function. Using $V$ this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value "piecewise" rather than just via literal trajectories that you happened to witness.
  
  > **Comments:**
  > **username_2**: It may also be a good idea to mention [TD($\lambda$)](https://ai.stackexchange.com/q/17605/2444), so that people don't think that the distinction between the 2 approaches is always sharp.  
  
  ---
  
  