Hi ptrblck,
thanks for the useful link!
But I am still not sure if what I want to do is correct. I want to use k-fold CV for model-evaluation, I am not interested in any hyper-parameter selection. However, because I am using neural networks, I still need a validation set to decide when to stop training.
In the slides of the link you posted, this case is covered:
Dataset = [o, d, p, x]
Fold1: Train = [o, d, p], Test=[x]
Fold2: Train = [d, p, x], Test=[o]
Fold3: Train = [p, x, o], Test=[d]
Fold4: Train = [x, o, d], Test=[p]
from which I could estimate the performance of my model with fixed hyper-parameters as the average of the performance on the test sets across the different folds.
But no validation set is used here and I am wondering if I should keep the validation set fixed across the different folds (e.g. [p] as in my previous post) or if this should vary as well or also use some inner loop.
What do you think?