How to compute score on separate test set after k-fold cross validation on separate train set? I am aware there is quite a few similar questions but none answer was dealing with following situation:
I have a task with train dataset and test dataset provided. All previous approaches are measured on test dataset. I also want to measure my approach on test dataset.
So I perform 5 fold cross validation on training dataset and get five scores (s1, s2, s3, s4, s5) corresponding to five versions of approach (V1, V2, V3, V4, V5)
Now, the question is: what is "official", standard procedure to compute score for test set?
Possible answers I can see are:

*

*Pick the version V with best score s and use it to compute score on test dataset

*For each version V calculate score on test dataset and present mean value with standard deviation

... or is there another way without using n fold cross validation?

You want to test as few models on the test set as possible, to avoid overfitting by accident. So pick the version you believe in most, treat it as your one shot on this problem, and then use that. Then you have done yours, and there's no going back.
Think of the test set as something you can spend once on one model. It better be a model you believe in!
Because once you start going back from a spent test set, you will no longer be fitting against the general underlying mechanism, you will be fitting against the test set. And what fits the test set really well won't solve the general problem as well.
