Hello @shaun,
the idea of using a validation set is that whatever you plan do to the test dataset would work for the validation set as well (really, you would use the test set’s DataLoader for the val_dataset).
For example, in the MNIST example’s (https://github.com/pytorch/examples/blob/master/mnist/main.py 15) test function, you can see how the function sums loss and correct guesses and then compute the average after the loop (for very large validation sets, you would need to look at overflow etc.).
As such, my suggestion would be to feed it to a dataloader that works similar to the test one (for the MNIST example, in fact, you could make test_loader a parameter to the test function and feed the val_loader. That would also do the model.eval() call mentioned earlier).
For “mass testing” I would expect that - like most examples I have seen - you would use minibatches as well if you have a sizeable validation set and aggregate the accuracy. If your validation set happens to fit in memory, you could pass batch_size = len(val_ds) to the validation DataLoader constructor.
Personally, I would prefer to keep the workflow with Dataset->DataLoader->Validation as that is scalable and looks like an efficient use of my time, but you should certainly just do whatever works for you.
Regarding the distinction between test and train: You would call model.eval() for validation and testing, that’s how the model knows.
One thing that changes is that the forward pass’s info will not be kept around to save memory. Many things will not depend on it (so people might leave it out) in terms of numerical results of the forward pass, but for example for models using dropout in the usual fashion (as opposed to Yarin Gal and collaborators 2), the model needs to know whether you are training or testing in the forward pass before it knows whether you use backward.
Hope this helps, even if it’s just my very limited take on something that ultimately boils down to style preferences and I cannot vouch for my expertise in that.
Best regards
Thomas