Model selection in presence of overfitting - better test or closer train Suppose I have a tree-based model (Random Forest for the sake of the example) and I play with a regularization parameter (tree depth) to fight overfitting.
Eventually I can come up with two models - M1 and M2 with the following performance:
M1:
Train AUC: 0.98 +/- 0.01
Test AUC:  0.84 +/- 0.02
M2:
Train AUC: 0.85 +/- 0.02
Test AUC:  0.82 +/- 0.02
Theoretically M1 is extremely overfitted, which can be problematic, but practically - M1 also has a better performance on test.
Given that those are the only two options, which one would you choose and why?

Assuming that the test set AUC is truly independent of the training set then using M1 is fine.
It would be more worrying if you didn't have that $\pm$ sign because I would worry that this is "mostly luck" but sampling variation might eat these gains out. Ultimately we care about performance generalisation on unseen data, if the test AUC is truly reflecting that,  this is all the justification we need. Please see the thread on: Test accuracy higher than training. How to interpret?  too; it makes some excellent points on how to assess the correctness of the train-test split. For example, are we using the correct sampling unit? If there is an inherent spatial clustering in our data, maybe stratifying by cluster is more relevant.
