Practical justification for not basing model selection on performance on test data? I've always been told not to use a model's performance on the test data to select a final model. I've read the responses to this question and others posted around the internet, but still have lingering doubts. While I can understand this theoretically in the context of "test data is unseen data and cannot influence decisions about model construction," I often find this challenging to wrap my head around in practice.
For example, suppose I have model A. I train it and get a validation score of 0.70. Then I have model B, which is trained on data that contains 2 new engineered features. My validation score for this model is 0.72.
I'm told to select model B based on its better validation score, so I do. When I evaluate on the test data, I get a test score of 0.67.
Out of curiosity, I check how model A performs on the test data. It scores 0.69.
What is the benefit of selecting model B in this case, when I know that model A performs better on the test data? If I were to deploy one of these models, should I really be sticking with B?

The problem is one of over-fitting the model selection criterion, see my paper (with Mrs Marsupial)
G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (pdf)
Essentially if we make model choices by minimising a model selection criterion based on a finite amount of data, then that minimisation may involve changes that produce genuine improvements in generalisation performance, but it can also involve changes that are exploiting the particular sample of data and do not make performance better (and may make it worse).  This is analogous to over-fitting the training data, but at the next level up.
So one reason not to use the test data to select the model is that it will give an optimistically biased performance estimate.  However, it may be that you don't actually need that for your application.
If you have to make choices between models that have a lot of hyper-parameters (and hence a greater risk of over-fitting the model selection criterion on the validation set), then you may want to use the test set (which will then be optimistically biased), or a second validation set, for choosing the final model.  However, if the number of hyper-parameters is small, this is unlikely to be necessary and you can just use a single validation set.  See
J Wainer, G Cawley, Nested cross-validation when selecting classifiers is overzealous for most practical applications, Expert Systems with Applications 182, 115222 (pdf)
