I noticed that when similar questions are asked the commonly given advice is to use SubsetRandomSampler or random.split. Isn’t it wrong, though?
I always thought that, during validation, the validation set should be the same through all the different runs, since, changing it every time wouldn’t allow getting a good estimate of the error given the hyperparameters.
I guess the solution is as simple as setting a manual seed before calling random.split and then setting it back to its initial value. I just wonder if my idea of how validation works is correct.