Answers.Body: <p>Probably adding more Q estimators that are trained on separate data would not improve performance, and may even degrade it.</p>
  <p>At least there is no theoretical justification. Double Q learning addresses a specific problem with maximisation bias: When your estimators are noisy and you select the highest estimate (the greedy action), there will be a bias towards overestimating its value (in the update step) when using the same estimator for both. There is no equivalent bias to consider that would be impacted moving from 2 to 3 estimators.</p>
  <p>In addition, Q learning can make use of both estimators in each update - one to select, the other to use value. A higher number of estimators would need to be rotated through.</p>
  <p>However, it is possible that some other factor would make a 3 or 4 estimator agent effective. I have not experimented with this, and not aware of anything published. So you could always try the experiment. I suggest pick an environment in which double Q learning is already shown to perform well, and give it a go. These kinds of &quot;what if I changed this thing?&quot; experiments usually come to nothing, but they can be fun.</p>
  <p>I suspect what you will find is that the learning is slower, but a little bit more robust against some kinds of error. However, in a double Q learner, decreasing the learning rate and/or increasing the time between copying to the frozen copies of estimators should have a very similar effect.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-05T08:45:32.797Z
Answers.LastActivityDate: 2023-12-05T09:17:20.440Z
Answers.LastEditDate: 2023-12-05T09:17:20.440Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Yes, there are variations of Q-learning which use <span class="math-container">$n$</span> Q-functions named &quot;enseble Q-learning&quot; or &quot;ensemble Q-functions&quot;. You can have a look at <a href="http://arxiv.org/abs/2101.05982" rel="nofollow noreferrer">REDQ</a> algorithm.</p>
  <p>The main benefit of having multiple but <em>uncorrelated</em> Q-functions is that you can reduce the overestimation bias of the q-values, and therefore update them more often (with the same data) such that to achieve a better sample-efficiency, or, equivalently, <em>update-to-data</em> (UTD) ratio.</p>
  <p>The drawback is that an ensemble is always costly (so it does NOT scale with <span class="math-container">$n$</span>, at least naively), and so training is slowed down (regarding wall-clock time) although you use less data to learn.</p>
  <p>To fix this issue, <a href="http://arxiv.org/abs/2110.02034" rel="nofollow noreferrer">DroQ</a> uses <em>dropout</em> to easily create an ensemble of Q-functions, which is also cheap to evaluate. Therefore, you get speed, efficiency and SOTA performance.</p>
  <p><strong>Update:</strong> Ensemble of Q-functions are also popular in <em>offline RL</em> (where you want to learn a policy from a previously collected dataset of demonstrations), where the ensemble allows you to select the next best experience tuple by evaluating the uncertainty about the predicted q-values: <a href="http://arxiv.org/abs/2110.01548" rel="nofollow noreferrer">http://arxiv.org/abs/2110.01548</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-05T18:08:15.450Z
Answers.Text: Anzaone Good answer! I voted for you, ma potevo scriverlo anche in italiano che siamo entrambi italiani :)
Answers.UserDisplayName: Zollikofen4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T11:06:05.337Z
Answers.Text: FWIW, there is a paper that shows an ensemble of MLP's can efficiently utilise a GPU for ensemble sizes of up to 50. So it does not necessarily need to be so costly.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T15:08:04.770Z
Answers.Text: @David Do you happen to have a paper name for that?  I could probably find it but figured I'd ask.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-06T15:32:41.207Z
Answers.Text: @foreverska [this vectorised linear function](https://github.com/tinkoff-ai/CORL/blob/6afec90484bbf47dee05fdf525e26a3ebe028e9b/algorithms/offline/sac_n.py#L174) is the implementation. There is a paper [here](https://arxiv.org/abs/2303.14716) that shows in Table 8 some speed comparisons using this implementation.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-05T17:42:03.570Z
Answers.LastActivityDate: 2023-12-07T21:25:18.403Z
Answers.LastEditDate: 2023-12-07T21:25:18.403Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>Q-Learning (Watkins, 1989) uses a single function to estimate the value of actions and to choose the next action.  Double Q-Learning (Hasselt, 2010) extends this and uses two functions which are updated using different subsets of experience.  The paper claims this tends to calm down the overestimation of Q-Learning and Hasselt further claims: &quot;Therefore, this algorithm is not less data-efficient than Q-learning.&quot;</p>
  <p>So my question enters, is there a Q-Learning with n functions?  If so, does it scale on the two claims of Double Q?</p>
  
Tags: <reinforcement-learning><q-learning>
Title: Does increasing the number of Q functions in Q-Learning scale?
UserDisplayName: foreverska
ThreadText: # Does increasing the number of Q functions in Q-Learning scale?
  
  (Asked by: username_0 on 2023-12-05)
  
  Q-Learning (Watkins, 1989) uses a single function to estimate the value of actions and to choose the next action. Double Q-Learning (Hasselt, 2010) extends this and uses two functions which are updated using different subsets of experience. The paper claims this tends to calm down the overestimation of Q-Learning and Hasselt further claims: "Therefore, this algorithm is not less data-efficient than Q-learning."
  
  So my question enters, is there a Q-Learning with n functions? If so, does it scale on the two claims of Double Q?
  
  **Tags:** reinforcement-learning, q-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-12-05. Score: 8 | Accepted answer
  
  Yes, there are variations of Q-learning which use $n$ Q-functions named "enseble Q-learning" or "ensemble Q-functions". You can have a look at [REDQ](http://arxiv.org/abs/2101.05982) algorithm.
  
  The main benefit of having multiple but *uncorrelated* Q-functions is that you can reduce the overestimation bias of the q-values, and therefore update them more often (with the same data) such that to achieve a better sample-efficiency, or, equivalently, *update-to-data* (UTD) ratio.
  
  The drawback is that an ensemble is always costly (so it does NOT scale with $n$, at least naively), and so training is slowed down (regarding wall-clock time) although you use less data to learn.
  
  To fix this issue, [DroQ](http://arxiv.org/abs/2110.02034) uses *dropout* to easily create an ensemble of Q-functions, which is also cheap to evaluate. Therefore, you get speed, efficiency and SOTA performance.
  
  **Update:** Ensemble of Q-functions are also popular in *offline RL* (where you want to learn a policy from a previously collected dataset of demonstrations), where the ensemble allows you to select the next best experience tuple by evaluating the uncertainty about the predicted q-values: <http://arxiv.org/abs/2110.01548>.
  
  > **Comments:**
  > **username_3**: Anzaone Good answer! I voted for you, ma potevo scriverlo anche in italiano che siamo entrambi italiani :)  
  > **username_4**: FWIW, there is a paper that shows an ensemble of MLP's can efficiently utilise a GPU for ensemble sizes of up to 50. So it does not necessarily need to be so costly.  
  > **username_0**: @David Do you happen to have a paper name for that?  I could probably find it but figured I'd ask.  
  > **username_4**: @foreverska [this vectorised linear function](https://github.com/tinkoff-ai/CORL/blob/6afec90484bbf47dee05fdf525e26a3ebe028e9b/algorithms/offline/sac_n.py#L174) is the implementation. There is a paper [here](https://arxiv.org/abs/2303.14716) that shows in Table 8 some speed comparisons using this implementation.  
  
  ---
  
  ### Answer by username_1 on 2023-12-05. Score: 7
  
  Probably adding more Q estimators that are trained on separate data would not improve performance, and may even degrade it.
  
  At least there is no theoretical justification. Double Q learning addresses a specific problem with maximisation bias: When your estimators are noisy and you select the highest estimate (the greedy action), there will be a bias towards overestimating its value (in the update step) when using the same estimator for both. There is no equivalent bias to consider that would be impacted moving from 2 to 3 estimators.
  
  In addition, Q learning can make use of both estimators in each update - one to select, the other to use value. A higher number of estimators would need to be rotated through.
  
  However, it is possible that some other factor would make a 3 or 4 estimator agent effective. I have not experimented with this, and not aware of anything published. So you could always try the experiment. I suggest pick an environment in which double Q learning is already shown to perform well, and give it a go. These kinds of "what if I changed this thing?" experiments usually come to nothing, but they can be fun.
  
  I suspect what you will find is that the learning is slower, but a little bit more robust against some kinds of error. However, in a double Q learner, decreasing the learning rate and/or increasing the time between copying to the frozen copies of estimators should have a very similar effect.
  
  
  ---
  
  