Diversity between classifiers in ensemble learning According to Wikipedia, Ensemble of models tends to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Using a variety of strong learning algorithms has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity. I have two questions please regarding this paragraph:

*

*Could you please explain the meaning of diversity with simple example?

*In ensemble classifier, do we combine (using for example, majority voting) weak classifiers or strong classifiers, or a combination of both weak and strong classifiers?


Your first question: This is probably intentionally formulated very vaguely on Wikipedia. But, as an example, you could think of Bayesian predictions as the weighted average of many models which differ continuously in their parameters. And often the weights are only large in a confined region, so the relevant models in this ensemble differ only by a small change in the parameters, which could qualify for small diversity in the models. A larger diversity would be obtained if you were to fit completely different models, e.g. a decision tree and an SVM.
Your second question: Following your citation of Wikipedia, a combination of strong classifiers would be preferable. But there are many examples where an ensemble of weak classifiers can obtain excellent results, like e.g. random forests or gradient boosting machines, provided the ensembles contain sufficiently many models. The performance is also very much dependent on how exactly you do the ensembling. Ordinary majority voting is a less sophisticated method, but, given enough models, might provide good results.
Most of the time the rule of thumb is: the more models, the better, no matter whether they are weak or strong.
