Model Selection vs. Ensemble Learning Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"?
It would seem that model selection does indeed fit that definition. My thinking is that the model selection process considers a set of models and at every point in the sample space chooses the model from the model set which optimizes some criterion at that point. The result is thus a new model, defined piece-wise across the sample space according to the selection criterion.
Is this correct, or am I missing something here? I can't seem to find a definitive answer to this question in the literature, hence my post, but the paper "Estimation and Accuracy after Model Selection" by Bradley Efron seems to support this viewpoint. I've also heard of ensemble learning via a "bucket of models" (a description of which can be found on the "Ensemble Learning" wikipedia page) which seems related to what I am asking about, but I could just be misunderstanding that too.
Edit: Upon further reflection, it seems like frank's answer answers my question. By thinking of estimation as a specific kind of prediction problem (in the sense of using data to predict\guess what the parameter value is), it seems like model selection can be thought of as a specific kind of mixture of experts, which in turn is a specific kind of ensemble learning. Thus, model selection is a kind of ensemble learning.

Your description of "ensemble" sounds like what is usually referred to as "mixture of experts", which is a special type of ensemble.
In a mixture of experts, one applies one of a collection of models depending on some properties of the input. But the notion of "ensembles of models" includes many more techniques. E.g., a simple standard ensemble in regression would be to take the average of the output of all the member models of an ensemble. More complex ensembles use boosting methods like e.g. the gradient boosting machines. For an overview of possible ensemble techniques, see here.
