Making ensemble of K models trained during K-fold cross-validation Let's suppose we are doing K-fold cross-valiation to estimate the performance of a model with a given set of hyperparameters.
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([0, 0, 1, 1])
model = RandomForestClassifier()
rkf = ShuffleSplit(n_splits=5, random_state=42)
models_list = []
for train_index, test_index in rkf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]


    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    accs = accuracy_score(y_test, preds)
    models_list.append(model)

ensemble_model = VotingClassifier(models_list) # Ensemble the models into a VotingClassifier

If I'm not mistaken, the models trained on each fold are not combined into an ensemble to be used in production. Can we combine the models trained on the individual folds in an ensemble and use that in production?

There's two versions of this:

*

*You trained a model (=just a single model) on K-folds to estimate performance. You don't want to re-train on all data (e.g. because that makes early stopping based on out-of-fold performance difficult). You can just average the predictions of all the models (e.g. average of predicted continuous values, average of probabilities for binary or multi-class prediction, but also voting ensemble and so on). However, note that there's not really any good reason to deviate from a simple average of some form and give more weight to one version of a model over another trained on different folds (you cannot really compare the out-of-fold performances). There's no particular reason this would be wrong, although I have heard a former Kaggle #1 mention that re-training on all data usually improves performance. Nevertheless, this is sometimes done e.g. for Kaggle. In practice, retraining on all data is also interesting to reduce complexity.

*You train multiple models (e.g. same model type, different hyperparameters, or different model classes) for each fold and use the out-of-fold predictions to decide how to combine their predictions (e.g. using some simple model to combine them, or just optimizing the weights in a weighted average) with hyperparameters tuned using the same CV scheme as before ("model stacking" or ensembling). Thereafter, you can - again - either combine for the models per fold or re-train on all data and then combine. All the same considerations as in point 1) apply, just with one extra layer of ensembling.

