Answers.Body: <p>These are generally known as <a href="https://en.wikipedia.org/wiki/Ensemble_learning" rel="nofollow noreferrer">ensemble methods</a>. Your method is essentially what Scikit-Learn's <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html" rel="nofollow noreferrer"><code>VotingClassifier</code></a> does, which is perfectly reasonable and might give you better results. Of course, if you have an ensemble of classifiers and some of them perform quite poorly, the ensemble might not be able to beat the best classifier: you'll need to check this in your cross-validation. Be aware that any confidence or probability estimates may not be well-calibrated, so the predictions from the ensemble may not be particularly meaningful.</p>
  <p>There are more elaborate ways of ensembling classifiers. Random forest classifiers, for example, are just ensembles of decision trees; a technique called <a href="https://en.wikipedia.org/wiki/Bootstrap_aggregating" rel="nofollow noreferrer">bagging</a> is also used here to improve performance.</p>
  <p>The technique of using a second model to weight the predictions from an ensemble is known as <em>stacked generalisation</em>. It was introduced by Wolpert in 1991 [1], and you can find plenty of interesting examples of using the technique, e.g. on <a href="https://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python" rel="nofollow noreferrer">Kaggle</a>.</p>
  <p>[1] David H. Wolpert. <em>Stacked generalization</em>. Neural Networks 5.2 (1992), pp. 241-259.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-02T09:38:20.743Z
Answers.LastActivityDate: 2021-03-02T09:38:20.743Z
Answers.UserDisplayName: htl
Answers.Body: <p>It is a simple way to do it but it is not wrong.</p>
  <p>If you are getting probabilities for each model, then, you can average them. Then, you can do the classification.</p>
  <p>Also, you assign weights to each model based on a validation set and regressing the weights for each models prediction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-02T05:53:15.997Z
Answers.LastActivityDate: 2021-03-02T05:53:15.997Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I am working on a classification problem.</p>
  <p>I have a dataset <span class="math-container">$S$</span> and I am training several prediction algorithms using S: Naive Bayes, SVM, classification trees.</p>
  <p>Intuitively, I was planning to combine my models, and, for each data point in the test sample <span class="math-container">$S'$</span>, take the majority vote as my prediction.</p>
  <p>Does that make sense? I feel this is a very simplistic way to combine different models.</p>
  
Tags: <machine-learning><classification><ensemble-learning>
Title: Does it make sense to combine classifiers trained on the same dataset?
UserDisplayName: ℕʘʘḆḽḘ
ThreadText: # Does it make sense to combine classifiers trained on the same dataset?
  
  (Asked by: username_0 on 2021-03-02)
  
  I am working on a classification problem.
  
  I have a dataset $S$ and I am training several prediction algorithms using S: Naive Bayes, SVM, classification trees.
  
  Intuitively, I was planning to combine my models, and, for each data point in the test sample $S'$, take the majority vote as my prediction.
  
  Does that make sense? I feel this is a very simplistic way to combine different models.
  
  **Tags:** machine-learning, classification, ensemble-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-02. Score: 3
  
  These are generally known as [ensemble methods](https://en.wikipedia.org/wiki/Ensemble_learning). Your method is essentially what Scikit-Learn's [`VotingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html) does, which is perfectly reasonable and might give you better results. Of course, if you have an ensemble of classifiers and some of them perform quite poorly, the ensemble might not be able to beat the best classifier: you'll need to check this in your cross-validation. Be aware that any confidence or probability estimates may not be well-calibrated, so the predictions from the ensemble may not be particularly meaningful.
  
  There are more elaborate ways of ensembling classifiers. Random forest classifiers, for example, are just ensembles of decision trees; a technique called [bagging](https://en.wikipedia.org/wiki/Bootstrap_aggregating) is also used here to improve performance.
  
  The technique of using a second model to weight the predictions from an ensemble is known as *stacked generalisation*. It was introduced by Wolpert in 1991 [1], and you can find plenty of interesting examples of using the technique, e.g. on [Kaggle](https://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python).
  
  [1] David H. Wolpert. *Stacked generalization*. Neural Networks 5.2 (1992), pp. 241-259.
  
  
  ---
  
  ### Answer by username_2 on 2021-03-02. Score: 1
  
  It is a simple way to do it but it is not wrong.
  
  If you are getting probabilities for each model, then, you can average them. Then, you can do the classification.
  
  Also, you assign weights to each model based on a validation set and regressing the weights for each models prediction.
  
  
  ---
  
  