Answers.Body: <p>I would say that in general situation more estimators are better.</p>
  <p>RandomForest fits a lot of estimators - decision trees that take a subset of data (obtained sampling with replacement) and subset of features (by default <code>sqrt(n_features)</code> in sklearn).</p>
  <p>Each of these estimators is noisy and prone to overfitting, producing a complicated decision surface.</p>
  <p>But when you take sufficiently many of them, noisy artifacts, produced by individual estimators, are smoothed and you can get pretty accurate classifier or regressor.</p>
  <p>It can be the case, that some added estimators are too noisy and worsen the ensemble, but overall, quality is expected to improve. At some point, the amount of estimators would be sufficient, and additional change won't change the result a lot.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-23T06:55:53.867Z
Answers.LastActivityDate: 2022-01-23T06:55:53.867Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>For Random Forests in particular, you will find a nice collection on references regarding research on the choice of number of trees in <a href="https://arxiv.org/pdf/1511.05741.pdf" rel="nofollow noreferrer">BiauScornet2015</a>, section 2.4.</p>
  <p>Further, a Random Forest can be considered an ensemble of decision trees. There is a bunch of literature on what, in fact, makes an ensemble effective. As far as I could gather, the gist of it seems to be that we need the right kind of &quot;diversity&quot; in the ensemble. <em>Diversity</em> is the keyword that seems to be used in the literature.</p>
  <p>Coming from the practical side, you may find some research that investigates the relationship between ad-hoc/intuitive diversity measures (see also Zhou2012 sec 5.3) and ensemble performance for the classification task. The results: &quot;it's complicated&quot;. [<a href="https://dl.acm.org/doi/abs/10.1016/j.ijar.2011.12.011" rel="nofollow noreferrer">1</a>, <a href="https://link.springer.com/article/10.1023/A:1022859003006" rel="nofollow noreferrer">2</a>, <a href="https://scialert.net/fulltext/?doi=jse.2017.60.65" rel="nofollow noreferrer">3</a>]</p>
  <p>However, from the theoretical side, I feel like there is plenty of motivation for this notion. A very good read is &quot;Zhou2012: Ensemble Methods: Foundations and Algorithms&quot;. Let me quote two points from there in a condensed manner.</p>
  <h1>Error-Ambiguity Decomposition</h1>
  <p>Assume that the task is to use an ensemble of <span class="math-container">$T$</span> individual learners <span class="math-container">$h_1, \ldots, h_T$</span> to approximate a function <span class="math-container">$f: R^d \mapsto R$</span>, and the final prediction of the ensemble is obtained through weighted averaging (4.9), i.e.,
  <span class="math-container">$$
  H(\boldsymbol{x})=\sum_{i=1}^T w_i h_i(\boldsymbol{x})
  $$</span>
  where <span class="math-container">$w_i$</span> is the weight for the learner <span class="math-container">$h_i$</span>, and the weights are constrained by <span class="math-container">$w_i \geq 0$</span> and <span class="math-container">$\sum_{i=1}^T w_i=1$</span></p>
  <p>Let the errors of an individual learner (submodel) and the error of the ensemble be, resp.
  <span class="math-container">$$
  \begin{aligned}
  &amp; \operatorname{err}\left(h_i \mid \boldsymbol{x}\right)=\left(f(\boldsymbol{x})-h_i(\boldsymbol{x})\right)^2 \\
  &amp; \operatorname{err}(H \mid \boldsymbol{x})=(f(\boldsymbol{x})-H(\boldsymbol{x}))^2
  \end{aligned}
  $$</span></p>
  <p>Likewise, we can define the <em>ambiguity</em>, a measure of disagreement among individual learners on instance (point / sample) <span class="math-container">$x$</span>:
  <span class="math-container">$$
  \operatorname{ambi}\left(h_i \mid \boldsymbol{x}\right)=\left(h_i(\boldsymbol{x})-H(\boldsymbol{x})\right)^2,
  $$</span></p>
  <p>The generalization error and the ambiguity of the individual learner <span class="math-container">$h_i$</span> can be written respectively as
  <span class="math-container">$$
  \begin{aligned}
  \operatorname{err}\left(h_i\right) &amp; =\int \operatorname{err}\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x}, \\
  \operatorname{ambi}\left(h_i\right) &amp; =\int a m b i\left(h_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x} .
  \end{aligned}
  $$</span></p>
  <p>The generalization error of the ensemble can be written as
  <span class="math-container">$$
  \operatorname{err}(H)=\int \operatorname{err}(H \mid \boldsymbol{x}) p(\boldsymbol{x}) d \boldsymbol{x}
  $$</span></p>
  <p>Based on the above notations, we can get the error-ambiguity decomposition [Krogh and Vedelsby, 1995]
  <span class="math-container">$$
  \operatorname{err}(H)=\overline{\operatorname{err}}(h)-\overline{a m b i}(h),
  $$</span></p>
  <p>where <span class="math-container">$\overline{\operatorname{err}}(h)=\sum_{i=1}^T w_i \cdot \operatorname{err}\left(h_i\right)$</span> is the weighted average of individual generalization errors, and <span class="math-container">$\overline{a m b i}(h)=\sum_{i=1}^T w_i \cdot \operatorname{ambi}\left(h_i\right)$</span> is the weighted average of ambiguities that is also referred to as the ensemble ambiguity.</p>
  <p>This shows that the the more accurate and the more diverse the individual learners, the better the ensemble.</p>
  <h1>Bias-Variance-Covariance Decomposition</h1>
  <p>The bias-variance-covariance decomposition of squared error of ensemble is
  <span class="math-container">$$
  \operatorname{err}(H)=\overline{\operatorname{bias}}(H)^2+\frac{1}{T} \overline{\operatorname{variance}}(H)+\left(1-\frac{1}{T}\right) \overline{\operatorname{covariance}}(H) .
  $$</span>
  where
  <span class="math-container">\begin{aligned}
  &amp; \overline{\operatorname{bias}}(H)=\frac{1}{T} \sum_{i=1}^T\left(\mathbb{E}\left[h_i\right]-f\right) \\
  &amp; \overline{\operatorname{variance}}(H)=\frac{1}{T} \sum_{i=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right)^2, \\
  &amp; \overline{\operatorname{covariance}}(H)=\frac{1}{T(T-1)} \sum_{i=1}^T \sum_{j=1}^T \mathbb{E}\left(h_i-\mathbb{E}\left[h_i\right]\right) \mathbb{E}\left(h_j-\mathbb{E}\left[h_j\right]\right)
  \end{aligned}</span></p>
  <p>The smaller the covariance, the better the ensemble. It is obvious that if all the learners make similar errors, the covariance will be large, and therefore it is preferred that the individual learners make different errors. Thus, through the covariance term, (5.18) shows that the diversity is important for ensemble performance. Notice that the bias and variance terms are constrained to be positive, while the covariance termcan be negative</p>
  <h1>See Also</h1>
  <ul>
  <li><a href="https://machinelearningmastery.com/ensemble-diversity-for-machine-learning/" rel="nofollow noreferrer">https://machinelearningmastery.com/ensemble-diversity-for-machine-learning/</a></li>
  <li><a href="https://stats.stackexchange.com/questions/328798/defining-diversity-in-ensemble-learning">https://stats.stackexchange.com/questions/328798/defining-diversity-in-ensemble-learning</a></li>
  <li><a href="https://stats.stackexchange.com/questions/576777/diversity-between-classifiers-in-ensemble-learning/576827#576827">https://stats.stackexchange.com/questions/576777/diversity-between-classifiers-in-ensemble-learning/576827#576827</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T13:36:21.087Z
Answers.LastActivityDate: 2023-04-06T13:36:21.087Z
Answers.UserDisplayName: ngmir
Body: <p>I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the <code>scikit-optimize</code> package.  For those unfamiliar with the package, it can be used easily with model classes from <code>scikit-learn</code>, in this case the random forest classes such as <code>RandomForestClassifier</code>, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.</p>
  <p>I noticed that in some examples, the <code>n_estimators</code> hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect.  The <code>n_estimators</code> hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in <code>scikit-optimize</code>, only single variables).</p>
  <p>Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model?  If so, what is the rationale?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-23T03:31:25.750Z
Comments.Text: Any time there is randomness, there is a chance the estimator could produce better results.
Comments.UserDisplayName: FourierFlux
Tags: <machine-learning><hyperparameter-optimization><decision-trees><random-forests>
Title: Random forests - are more estimators always better?
UserDisplayName: SuperCodeBrah
ThreadText: # Random forests - are more estimators always better?
  
  (Asked by: username_0 on 2022-01-22)
  
  I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the `scikit-optimize` package. For those unfamiliar with the package, it can be used easily with model classes from `scikit-learn`, in this case the random forest classes such as `RandomForestClassifier`, and it provides more intelligent alternatives to traditional hyperparameter optimization methods like grid search.
  
  I noticed that in some examples, the `n_estimators` hyperparameter (of the random forest) is included in the optimization, which I wouldn't expect. The `n_estimators` hyperparameter determines the number of component decision trees in the random forest, so I would expect that more estimators always results in a better model with respect to a single target variable (for clarity, I'm not referring to anything having to do with optimizing a custom objective function in `scikit-optimize`, only single variables).
  
  Ignoring practical issues like training time as well as the potential effects of randomness (i.e., that different random seeds could lead to models with varying effectiveness), are there situations where fewer estimators could result in a more accurate model? If so, what is the rationale?
  
  **Tags:** machine-learning, hyperparameter-optimization, decision-trees, random-forests
  
  > **Comments:**
  > **username_1**: Any time there is randomness, there is a chance the estimator could produce better results.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-01-23. Score: 1
  
  I would say that in general situation more estimators are better.
  
  RandomForest fits a lot of estimators - decision trees that take a subset of data (obtained sampling with replacement) and subset of features (by default `sqrt(n_features)` in sklearn).
  
  Each of these estimators is noisy and prone to overfitting, producing a complicated decision surface.
  
  But when you take sufficiently many of them, noisy artifacts, produced by individual estimators, are smoothed and you can get pretty accurate classifier or regressor.
  
  It can be the case, that some added estimators are too noisy and worsen the ensemble, but overall, quality is expected to improve. At some point, the amount of estimators would be sufficient, and additional change won't change the result a lot.
  
  
  ---
  
  ### Answer by username_3 on 2023-04-06. Score: 0
  
  For Random Forests in particular, you will find a nice collection on references regarding research on the choice of number of trees in [BiauScornet2015](https://arxiv.org/pdf/1511.05741.pdf), section 2.4.
  
  Further, a Random Forest can be considered an ensemble of decision trees. There is a bunch of literature on what, in fact, makes an ensemble effective. As far as I could gather, the gist of it seems to be that we need the right kind of "diversity" in the ensemble. *Diversity* is the keyword that seems to be used in the literature.
  
  Coming from the practical side, you may find some research that investigates the relationship between ad-hoc/intuitive diversity measures (see also Zhou2012 sec 5.3) and ensemble performance for the classification task. The results: "it's complicated". [[1](https://dl.acm.org/doi/abs/10.1016/j.ijar.2011.12.011), [2](https://link.springer.com/article/10.1023/A:1022859003006), [3](https://scialert.net/fulltext/?doi=jse.2017.60.65)]
  
  However, from the theoretical side, I feel like there is plenty of motivation for this notion. A very good read is "Zhou2012: Ensemble Methods: Foundations and Algorithms". Let me quote two points from there in a condensed manner.
  
  # Error-Ambiguity Decomposition
  
  Assume that the task is to use an ensemble of $T$ individual learners $h\_1, \ldots, h\_T$ to approximate a function $f: R^d \mapsto R$, and the final prediction of the ensemble is obtained through weighted averaging (4.9), i.e.,
  $$
  H(\boldsymbol{x})=\sum\_{i=1}^T w\_i h\_i(\boldsymbol{x})
  $$
  where $w\_i$ is the weight for the learner $h\_i$, and the weights are constrained by $w\_i \geq 0$ and $\sum\_{i=1}^T w\_i=1$
  
  Let the errors of an individual learner (submodel) and the error of the ensemble be, resp.
  $$
  \begin{aligned}
  & \operatorname{err}\left(h\_i \mid \boldsymbol{x}\right)=\left(f(\boldsymbol{x})-h\_i(\boldsymbol{x})\right)^2 \\
  & \operatorname{err}(H \mid \boldsymbol{x})=(f(\boldsymbol{x})-H(\boldsymbol{x}))^2
  \end{aligned}
  $$
  
  Likewise, we can define the *ambiguity*, a measure of disagreement among individual learners on instance (point / sample) $x$:
  $$
  \operatorname{ambi}\left(h\_i \mid \boldsymbol{x}\right)=\left(h\_i(\boldsymbol{x})-H(\boldsymbol{x})\right)^2,
  $$
  
  The generalization error and the ambiguity of the individual learner $h\_i$ can be written respectively as
  $$
  \begin{aligned}
  \operatorname{err}\left(h\_i\right) & =\int \operatorname{err}\left(h\_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x}, \\
  \operatorname{ambi}\left(h\_i\right) & =\int a m b i\left(h\_i \mid \boldsymbol{x}\right) p(\boldsymbol{x}) d \boldsymbol{x} .
  \end{aligned}
  $$
  
  The generalization error of the ensemble can be written as
  $$
  \operatorname{err}(H)=\int \operatorname{err}(H \mid \boldsymbol{x}) p(\boldsymbol{x}) d \boldsymbol{x}
  $$
  
  Based on the above notations, we can get the error-ambiguity decomposition [Krogh and Vedelsby, 1995]
  $$
  \operatorname{err}(H)=\overline{\operatorname{err}}(h)-\overline{a m b i}(h),
  $$
  
  where $\overline{\operatorname{err}}(h)=\sum\_{i=1}^T w\_i \cdot \operatorname{err}\left(h\_i\right)$ is the weighted average of individual generalization errors, and $\overline{a m b i}(h)=\sum\_{i=1}^T w\_i \cdot \operatorname{ambi}\left(h\_i\right)$ is the weighted average of ambiguities that is also referred to as the ensemble ambiguity.
  
  This shows that the the more accurate and the more diverse the individual learners, the better the ensemble.
  
  # Bias-Variance-Covariance Decomposition
  
  The bias-variance-covariance decomposition of squared error of ensemble is
  $$
  \operatorname{err}(H)=\overline{\operatorname{bias}}(H)^2+\frac{1}{T} \overline{\operatorname{variance}}(H)+\left(1-\frac{1}{T}\right) \overline{\operatorname{covariance}}(H) .
  $$
  where
  \begin{aligned}
  & \overline{\operatorname{bias}}(H)=\frac{1}{T} \sum\_{i=1}^T\left(\mathbb{E}\left[h\_i\right]-f\right) \\
  & \overline{\operatorname{variance}}(H)=\frac{1}{T} \sum\_{i=1}^T \mathbb{E}\left(h\_i-\mathbb{E}\left[h\_i\right]\right)^2, \\
  & \overline{\operatorname{covariance}}(H)=\frac{1}{T(T-1)} \sum\_{i=1}^T \sum\_{j=1}^T \mathbb{E}\left(h\_i-\mathbb{E}\left[h\_i\right]\right) \mathbb{E}\left(h\_j-\mathbb{E}\left[h\_j\right]\right)
  \end{aligned}
  
  The smaller the covariance, the better the ensemble. It is obvious that if all the learners make similar errors, the covariance will be large, and therefore it is preferred that the individual learners make different errors. Thus, through the covariance term, (5.18) shows that the diversity is important for ensemble performance. Notice that the bias and variance terms are constrained to be positive, while the covariance termcan be negative
  
  # See Also
  
  * <https://machinelearningmastery.com/ensemble-diversity-for-machine-learning/>
  * <https://stats.stackexchange.com/questions/328798/defining-diversity-in-ensemble-learning>
  * <https://stats.stackexchange.com/questions/576777/diversity-between-classifiers-in-ensemble-learning/576827#576827>
  
  
  ---
  
  