Making ensemble of K models trained during K-fold cross-validation Let's suppose we are doing K-fold cross-valiation to estimate the performance of a model with a given set of hyperparameters.
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([0, 0, 1, 1])
model = RandomForestClassifier()
rkf = ShuffleSplit(n_splits=5, random_state=42)
models_list = []
for train_index, test_index in rkf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]


    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    accs = accuracy_score(y_test, preds)
    models_list.append(model)

ensemble_model = VotingClassifier(models_list) # Ensemble the models into a VotingClassifier

If I'm not mistaken, the models trained on each fold are not combined into an ensemble to be used in production. Can we combine the models trained on the individual folds in an ensemble and use that in production?

nnUNet is a successful procedure for automatic hyperparameter tuning and optimization of UNets (neural segmentation models), which applies the suggested method, https://arxiv.org/abs/1809.10486: "For the test cases we use the five networks obtained from our training set cross-validation as an ensemble to further increase the robustness of our models." I don't believe they claim it's optimal, but maybe the approach has more merit for deep neural nets than for random forest models?
