Answers.Body: <p>Technically, it would not be Q-learning if you used Monte Carlo returns, because Q-learning is specifically a temporal difference (TD) method.</p>
  <p>However, your intuition about the estimates being interchangeable is correct. There is nothing preventing you in principle from swapping out the TD return estimate for a Monte Carlo sample to generate data for training a neural network that is approximating <span class="math-container">$Q(s, a)$</span>. In some ways it is a better estimate, as it will be unbiased. If you stop using bootstrapping, then the learning process is theoretically more stable and less likely to diverge.</p>
  <p>There are some caveats:</p>
  <ul>
  <li><p>Experience replay becomes less practical. You could store the final return from each <span class="math-container">$(s,a)$</span> pair, but it will quickly become off-policy to a degree that it could not be used. Some extensions to DQN do use a small n with n-step returns and no correction. However, you have to be careful with this, and full trajectories off-policy would need better correction - many would be useless, especially at the start of learning.</p>
  </li>
  <li><p>Without experience replay, you will need more samples of the current policy output before training. Training on a batch generated from just a single trajectory could easily result in poor learning performance, because records would be closely correlated with each other. So you would need to borrow ideas from CEM or A2C where you generate many trajectories for each training batch.</p>
  </li>
  </ul>
  <p>These issues could make a Monte-Carlo based deep RL slower to train than DQN. However, perhaps the extra stability of the approach may appeal for a specific problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T11:52:33.067Z
Answers.Text: Thanks for the reply! But I don't quite understand why is "Q-learning is specifically a TD method"? I mean, what characteristic of Q-learning make it a TD method? A second question is what is the "corrections" you mentioned when using a n-step return? Is there a paper talking about the correction method?
Answers.UserDisplayName: Cloudy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T13:15:37.470Z
Answers.Text: Q-learning uses TD targets as value estimates, that makes it a TD method. The equivalent MC method is called "off-policy Monte Carlo control", it is not called "Q-learning with MC return estmates", although it could be in principle that's not how the original designers of Q-learning chose to categorise what they created. For corrections required for n-step returns see Sutton & Barto chapters on off-policy Monte Carlo. For the n-step augmentation in DQN (which deliberately ignores those corrections and "gets away with it"), see the "Rainbow" paper: https://arxiv.org/abs/1710.02298
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T09:06:34.730Z
Answers.LastActivityDate: 2022-03-24T09:06:34.730Z
Answers.UserDisplayName: Neil Slater
Body: <p>In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function <span class="math-container">$Q(s,a)$</span> is learned by temporal difference learning. However, in policy gradient methods, it is also possible to learn by using Monte-Carlo estimate of returns (i.e. observing all the rewards in a trajectory), if the task is episodic.</p>
  <p>I would like to know if it is also possible to do deep Q-learning by using Monte-Carlo estimate of returns in episodic task?</p>
  
Tags: <reinforcement-learning><deep-rl><q-learning><dqn>
Title: Is using Monte-Carlo estimate of returns in Deep Q Learning possible?
UserDisplayName: Cloudy
ThreadText: # Is using Monte-Carlo estimate of returns in Deep Q Learning possible?
  
  (Asked by: username_0 on 2022-03-24)
  
  In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient methods, it is also possible to learn by using Monte-Carlo estimate of returns (i.e. observing all the rewards in a trajectory), if the task is episodic.
  
  I would like to know if it is also possible to do deep Q-learning by using Monte-Carlo estimate of returns in episodic task?
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-24. Score: 2
  
  Technically, it would not be Q-learning if you used Monte Carlo returns, because Q-learning is specifically a temporal difference (TD) method.
  
  However, your intuition about the estimates being interchangeable is correct. There is nothing preventing you in principle from swapping out the TD return estimate for a Monte Carlo sample to generate data for training a neural network that is approximating $Q(s, a)$. In some ways it is a better estimate, as it will be unbiased. If you stop using bootstrapping, then the learning process is theoretically more stable and less likely to diverge.
  
  There are some caveats:
  
  * Experience replay becomes less practical. You could store the final return from each $(s,a)$ pair, but it will quickly become off-policy to a degree that it could not be used. Some extensions to DQN do use a small n with n-step returns and no correction. However, you have to be careful with this, and full trajectories off-policy would need better correction - many would be useless, especially at the start of learning.
  * Without experience replay, you will need more samples of the current policy output before training. Training on a batch generated from just a single trajectory could easily result in poor learning performance, because records would be closely correlated with each other. So you would need to borrow ideas from CEM or A2C where you generate many trajectories for each training batch.
  
  These issues could make a Monte-Carlo based deep RL slower to train than DQN. However, perhaps the extra stability of the approach may appeal for a specific problem.
  
  > **Comments:**
  > **username_0**: Thanks for the reply! But I don't quite understand why is "Q-learning is specifically a TD method"? I mean, what characteristic of Q-learning make it a TD method? A second question is what is the "corrections" you mentioned when using a n-step return? Is there a paper talking about the correction method?  
  > **username_1**: Q-learning uses TD targets as value estimates, that makes it a TD method. The equivalent MC method is called "off-policy Monte Carlo control", it is not called "Q-learning with MC return estmates", although it could be in principle that's not how the original designers of Q-learning chose to categorise what they created. For corrections required for n-step returns see Sutton & Barto chapters on off-policy Monte Carlo. For the n-step augmentation in DQN (which deliberately ignores those corrections and "gets away with it"), see the "Rainbow" paper: https://arxiv.org/abs/1710.02298  
  
  ---
  
  