Practical justification for not basing model selection on performance on test data? I've always been told not to use a model's performance on the test data to select a final model. I've read the responses to this question and others posted around the internet, but still have lingering doubts. While I can understand this theoretically in the context of "test data is unseen data and cannot influence decisions about model construction," I often find this challenging to wrap my head around in practice.
For example, suppose I have model A. I train it and get a validation score of 0.70. Then I have model B, which is trained on data that contains 2 new engineered features. My validation score for this model is 0.72.
I'm told to select model B based on its better validation score, so I do. When I evaluate on the test data, I get a test score of 0.67.
Out of curiosity, I check how model A performs on the test data. It scores 0.69.
What is the benefit of selecting model B in this case, when I know that model A performs better on the test data? If I were to deploy one of these models, should I really be sticking with B?


What is the benefit of selecting model B in this case

Assume both models' RMSE is 60 (appropriately scaled) units and that the sampling distribution for the RMSE is normal.  Let's assume you select a model by applying both on the test set selecting the model with the smallest error and then use the test performance as the out of sample performance estimate.
Ideally, the out of sample estimate is unbiased, meaning over the lifetime of the model and assuming no concept drift, this process has 0 bias for the true out of sample estimate.
Let's use R to see what would happen using your strategy
N = 10000
errs = rnorm(2*N, 60, 1)
errs = matrix(errs, ncol = 2)
selected_estimate = apply(errs,1,min)
hist(selected_estimate)


Look, estimate for out of sample performance is biased!  An estimate of this bias is mean(selected_estimate) - 60 which is about -0.6 when I run my code.  Is -0.6 big?  The answer doesn't matter, the point is that this should be 0 and we can see it isn't.
This is what you risk by selecting on the test set.  The test set is supposed to be an unbiased estimate of the performance once the model is selected.  By selecting a model on the test set, you've either risked biasing out of sample performance or have basically thrown away the ability to estimate the performance in an unbiased fashion. You've also thrown out data you could have ostensibly trained on because you've not really made use of the validation set; you've used the test set as a validation set.
