Answers.Body: <p>Lets consider binary classification.</p>
  <p>Imagine you have an ensemble made up of <span class="math-container">$K$</span> models. Assume each model has exactly <span class="math-container">$51\%$</span> accuracy. Further assume each model's error is uncorrelated with each other model. The ensemble simply takes a majority vote.</p>
  <p>For <span class="math-container">$K = 11$</span>, what is the probability we get a single question right?
  Note: This problem can be simply modeled as flipping an unfair coin <span class="math-container">$K$</span> times and calculating the probability that one side will be the majority of flips.</p>
  <blockquote>
  <p><span class="math-container">$P(correct\ \geq\ 6)\ =\ \sum_{k=6}^{11}{\binom{11}{k}({0.51}^k)({0.49}^{11-k})= 0.53}$</span></p>
  </blockquote>
  <p>What about when <span class="math-container">$K = 101$</span>?</p>
  <blockquote>
  <p><span class="math-container">$P(correct\ \geq\ 51)\ =\ \sum_{k=51}^{101}{\binom{101}{k}({0.51}^k)({0.49}^{101-k})= 0.58}$</span></p>
  </blockquote>
  <p>If you do the math you will find that as <span class="math-container">$K$</span> approaches <span class="math-container">$\infty$</span>, the odds of the ensemble being correct asymptotically approaches <span class="math-container">$100\%$</span>.</p>
  <blockquote>
  <p><span class="math-container">$\lim_{K\rightarrow\infty}{P\left(correct\ \geq\frac{K+1}{2}\right)}={lim}_{K\rightarrow\infty}\left(\sum_{k=\frac{K+1}{2}}^{K}{\binom{K}{k}({0.51}^k)({0.49}^{K-k}})\right)=1$</span></p>
  </blockquote>
  <p>Now, we made 2 key assumptions:</p>
  <ol>
  <li>Errors are uncorreleated between models</li>
  <li>Each model has more than <span class="math-container">$50\%$</span> accuracy</li>
  </ol>
  <p>In practice these assumptions often may not hold, especially the first one. But if they do, this simple probability analysis shows ensembles will be better</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T21:34:33.147Z
Answers.Text: Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.
Answers.UserDisplayName: Oscar Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T22:15:26.377Z
Answers.Text: @OscarSmith, can you explain how that is possible, maybe along with a concrete example?
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T22:29:13.743Z
Answers.Text: you take the model and make a new model that predicts the opposite. (for the n way version it's a bit harder, but you can make a model that chooses a random model other than the one your original model produced)
Answers.UserDisplayName: Oscar Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T23:52:02.330Z
Answers.Text: Sorry, I may have missed something here.  Is it implied somewhere in the first comment (Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.) that it is referring to a multiclass situation?
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-04T19:46:55.580Z
Answers.Text: The generalization of the 50% accuracy condition (which is better than random) is the weak assumption.
Answers.UserDisplayName: Oscar Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T16:09:44.650Z
Answers.LastActivityDate: 2022-11-29T15:59:28.800Z
Answers.LastEditDate: 2022-11-29T15:59:28.800Z
Answers.UserDisplayName: chessprogrammer
Body: <p><strong>BACKGROUND:</strong>  Ensemble classifiers are said to reduce <em>bias</em> by taking an &quot;average&quot; of predictions of several base classifiers that comprise the ensemble.  However, I am uncertain if this necessarily means that they can increase <em>accuracy</em>.  My intuition tells me that the ensemble classifier should perform <strong>no better and possibly even worse</strong> than the <em>best</em> base classifier in the ensemble.  This seems especially true for bagging approaches which use strong classifiers anyway.  When you have a &quot;star performer&quot;, it just doesn't seem to make intuitive sense to &quot;dilute&quot; its performance with subpar performers.</p>
  <p>Nonetheless, from my novice-level reading, it seems that ensembles can be <strong>as good or possibly even better</strong> than all of the individual base classifiers, but I'm still not clear why.</p>
  <p><strong>QUESTION:</strong>  How can an ensemble be more accurate than the best base classifier in that ensemble?</p>
  
Tags: <ensemble-learning>
Title: How can an ensemble be more accurate than the best base classifier in that ensemble?
UserDisplayName: Snehal Patel
ThreadText: # How can an ensemble be more accurate than the best base classifier in that ensemble?
  
  (Asked by: username_0 on 2022-11-27)
  
  **BACKGROUND:** Ensemble classifiers are said to reduce *bias* by taking an "average" of predictions of several base classifiers that comprise the ensemble. However, I am uncertain if this necessarily means that they can increase *accuracy*. My intuition tells me that the ensemble classifier should perform **no better and possibly even worse** than the *best* base classifier in the ensemble. This seems especially true for bagging approaches which use strong classifiers anyway. When you have a "star performer", it just doesn't seem to make intuitive sense to "dilute" its performance with subpar performers.
  
  Nonetheless, from my novice-level reading, it seems that ensembles can be **as good or possibly even better** than all of the individual base classifiers, but I'm still not clear why.
  
  **QUESTION:** How can an ensemble be more accurate than the best base classifier in that ensemble?
  
  **Tags:** ensemble-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-27. Score: 15 | Accepted answer
  
  Lets consider binary classification.
  
  Imagine you have an ensemble made up of $K$ models. Assume each model has exactly $51\%$ accuracy. Further assume each model's error is uncorrelated with each other model. The ensemble simply takes a majority vote.
  
  For $K = 11$, what is the probability we get a single question right?
  Note: This problem can be simply modeled as flipping an unfair coin $K$ times and calculating the probability that one side will be the majority of flips.
  
  > $P(correct\ \geq\ 6)\ =\ \sum\_{k=6}^{11}{\binom{11}{k}({0.51}^k)({0.49}^{11-k})= 0.53}$
  
  What about when $K = 101$?
  
  > $P(correct\ \geq\ 51)\ =\ \sum\_{k=51}^{101}{\binom{101}{k}({0.51}^k)({0.49}^{101-k})= 0.58}$
  
  If you do the math you will find that as $K$ approaches $\infty$, the odds of the ensemble being correct asymptotically approaches $100\%$.
  
  > $\lim\_{K\rightarrow\infty}{P\left(correct\ \geq\frac{K+1}{2}\right)}={lim}\_{K\rightarrow\infty}\left(\sum\_{k=\frac{K+1}{2}}^{K}{\binom{K}{k}({0.51}^k)({0.49}^{K-k}})\right)=1$
  
  Now, we made 2 key assumptions:
  
  1. Errors are uncorreleated between models
  2. Each model has more than $50\%$ accuracy
  
  In practice these assumptions often may not hold, especially the first one. But if they do, this simple probability analysis shows ensembles will be better
  
  > **Comments:**
  > **username_2**: Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.  
  > **username_0**: @username_2, can you explain how that is possible, maybe along with a concrete example?  
  > **username_2**: you take the model and make a new model that predicts the opposite. (for the n way version it's a bit harder, but you can make a model that chooses a random model other than the one your original model produced)  
  > **username_0**: Sorry, I may have missed something here.  Is it implied somewhere in the first comment (Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.) that it is referring to a multiclass situation?  
  > **username_2**: The generalization of the 50% accuracy condition (which is better than random) is the weak assumption.  
  
  ---
  
  