Diversity between classifiers in ensemble learning According to Wikipedia, Ensemble of models tends to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the models they combine. Using a variety of strong learning algorithms has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity. I have two questions please regarding this paragraph:

*

*Could you please explain the meaning of diversity with simple example?

*In ensemble classifier, do we combine (using for example, majority voting) weak classifiers or strong classifiers, or a combination of both weak and strong classifiers?



Ensemble of models tends to yield better results when there is a significant diversity among the models

Let me rephrase this: iff there is high variance between the submodels (or their predictions), ensemble methods help.
(This is very similar to what we routinely do e.g. with measurements iff the dominant factor for measurement uncertainty is variance: we then average multiple measurements, or go for the diagnosis of a panel of sensory experts, etc.)
The conclusion from this is that the submodels should be representative for the respective variance. (But it doesn't help to introduce arbitrary variance that isn't relevant for the model/data/application-combination)
Ensembles cannot help lowering the part of prediction error that is caused by bias.
Thus, the submodels of the ensemble should not have too much bias, but their variance is of less concern - since we plan to "average it away".
Now, a strong model may have achieved its good performance for a given (in particular: limited) data set by finding good bias-variance tradeoff for the single model prediction. This compromise may be sub-optimal for the ensemble submodels, in the sense that the ensemble may be able to achieve even better performace with (somewhat) weaker sub-models that reduce bias by accepting more variance compared to the strong model. Promoting variance in the sense of moving from such strong submodels to the corresponding weaker submodels does make sense. In other words, the optimal (sub)model complexity for a given application, data set, and training algorithm is different whether the model will be used individually or in an ensemble.
Another conclusion is: if you move from the optimal somewhat-weaker submodels to still weaker models with even more variance - that however isn't offset by lower bias - then that additional variance won't help the ensemble.

And lastly, if the strong model is strong by having both low bias and low variance, no ensemble is needed.
