How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? I got good results on a radiation exposure prediction problem using SVM and DT where the ultimate goal is to predict the radiation dose an individual was exposed to using data about individual related to their health.
Overall, the feedback was positive but one comment was interesting to me: How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? How did you exclude this?
I took all the necessary precautions to make sure things like class imbalance, overfitting, etc. weren't an issue, so I'm certainly under the impression this is more related to the fundamentals of the field rather than an ML methodology question.
How would one begin to approach this question from an ML perspective where I feel the answer basically boils down to the fact that, after doing all necessary procedures that can be done to optimize model performance (regularization, cross validation, etc.), there is no such thing as "coming to the right classifications the wrong way"?
I feel like this is actually a question that needs a good answer for any technical communication to audiences perhaps not as conversant with ML in general (or who perhaps are but want to hear our thoughts nonetheless).

This is the purpose of the Validation Set.
Split your dataset in 3 : Train, Test and Validation. Never touch your Validation set again until the last phase.
Create your model using Train and Test, train your encoders, make your preprocesses, create your variables etc. Then create your model and tune it using Train and Test.
When you're satisfied and have the model you want, then apply your encoders, preprocesses on your Validation set, which acts as totally new and unknown Data from the model, the same type as you'd have on a real case.
Apply your model on those brand new data (that you didn't use as a reference to train your model, as you did with Test) to have an overlook on how your model will perform in real time with new data.
This can be shown easily to the audience : Ask them to give you the latest examples they have, remove the final answer from the data, and run your model on it. You'll have precise results on how your model works on fresh data. If they're still doubting, ask for a test phase in which you run your algorithm day to day and in real time, without replacing their current system for the moment, so you can compare results at the end of the test phase.
Another thing to check is being sure you don't use variables that you shouldn't have in real case, or that you shouldn't know without knowing the target. That's a classic case of having a good classification from a wrong way.
