Making ensemble of K models trained during K-fold cross-validation Let's suppose we are doing K-fold cross-valiation to estimate the performance of a model with a given set of hyperparameters.
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([0, 0, 1, 1])
model = RandomForestClassifier()
rkf = ShuffleSplit(n_splits=5, random_state=42)
models_list = []
for train_index, test_index in rkf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]


    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    accs = accuracy_score(y_test, preds)
    models_list.append(model)

ensemble_model = VotingClassifier(models_list) # Ensemble the models into a VotingClassifier

If I'm not mistaken, the models trained on each fold are not combined into an ensemble to be used in production. Can we combine the models trained on the individual folds in an ensemble and use that in production?


Can we combine the models trained on the individual folds in an ensemble and use that in production?

Yes. If you need a reference we've been doing this in Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008). DOI: 10.1007/s00216-007-1818-6
However, for models that are already ensembles like randomForest, this isn't really substantially different from a randomForest with more trees.

I wasnt sure if theres is a mathematical reason for not doing it.

(For randomForest: see above)
There are conditions when it improves predictive ability and others where it doesn't:
If the surrogate models are stable in their predictions (you can easily and directly measure this by repeated k-fold CV), the ensemble will predict just as the single models and no better.
If the surrogate models are somewhat unstable, that instability (variance) is reduced by the ensemble prediction.
The ensemble prediction cannot help with bias in the predictions.
Side remark: a consequence of this is that I wouldn't expect hyperparameter settings optimized for single model prediction to be optimal for ensemble prediction.
(However, in practice you may find that they work well, because we often use hyperparameter tuning heuristics that err on the side of high complexity with too low bias and too high variance for single model prediction - which should be better for ensemble prediction)

The above was written with a fixed/single hyperparameter set for all models in the ensemble in mind.
If you look at building ensembles across e.g. a range of different complexities, have a look at Bayesian model averaging as well.
