Theoretical explanation of the superior performance of ensemble machine learning algorithms In my experience and I believe also in many others' experiences, ensemble models such as random forests and boosting give superior results compared to other models such as linear/logistic regression or single decision trees (I remember some experimental work which also confirms this but I could not find now). Of course there might be cases where linear regression is best, for example, when the underlying relationship is indeed linear. But ensemble models in general have better performance.
What is the explanation of this situation from a machine learning theory perspective? Are there some proofs which show that they have better generalization power? Any pointers to resources will also be appreciated.

We know that error in a certain model can be composited from bias and variance. A too complex model has low bias but large variance, while a too simple model has low variance but large bias, both leading to a high error but for two different reasons. As a result, two different ways to solve the problem come into people's mind (at leat to Breiman's mind), variance reduction for a complex model, or bias reduction for a simple model, which refers to bagging and boosting.
