How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? I got good results on a radiation exposure prediction problem using SVM and DT where the ultimate goal is to predict the radiation dose an individual was exposed to using data about individual related to their health.
Overall, the feedback was positive but one comment was interesting to me: How does one ensure Machine Learning doesn't come to correct classifications via the wrong ways? How did you exclude this?
I took all the necessary precautions to make sure things like class imbalance, overfitting, etc. weren't an issue, so I'm certainly under the impression this is more related to the fundamentals of the field rather than an ML methodology question.
How would one begin to approach this question from an ML perspective where I feel the answer basically boils down to the fact that, after doing all necessary procedures that can be done to optimize model performance (regularization, cross validation, etc.), there is no such thing as "coming to the right classifications the wrong way"?
I feel like this is actually a question that needs a good answer for any technical communication to audiences perhaps not as conversant with ML in general (or who perhaps are but want to hear our thoughts nonetheless).

One problem in ML is when it uses predictors we do not want it to use, like gender or ethnicity. And even if these are not fed into the model, we may still have predictors that do correlate with these factors, like ZIP codes correlating with ethnicity, or colleges correlating with gender.
Assuming that, say, gender does correlate with the outcome we are modeling, then even if we do not feed in genders, but only the college attended, and some colleges are traditionally gender-imbalanced, then we will overall get different classifications or predictions for men than for women.
This particular case can be found out by slicing you dataset by gender and checking the outcome predictions, while ignoring all other pieces of information.
Unfortunately, this is not simple to do, because the model is not using gender (which we didn't feed in). It's using the college attended, which in turn is correlated with the outcome. Does it even make sense to only slice the dataset in a single dimension like gender, while ignoring possible mediators or confounders, like the college? Is the problem that students from college A perform worse, and that men predominantly attend college A, or is the problem that men perform worse, and that they predominantly attend college A? And which predictor represents a "wrong" way for the model to come to predictions?
And then, of course, all this is mixed up with the question of whether the original problem is that the training data already exhibits the results of bias. Maybe male students from college A historically performed worse because there was always a hiring bias against students from college A. Or, conversely, a bias against men. There is no easy solution to this, because it is rarely possible to tease out the "real" effect of bias in the training sample from any true underlying differences.
Bottom line: there is no simple way to find out whether your model arrived at the "correct" predictions ("men perform worse") through "correct" ways (men indeed perform worse) or through "wrong" ways (men predominantly attend college A, and students from college A perform worse). In particular, there is no way you could test programmatically. Your best bet is likely to subject your model to various stress tests, and have a plan on how to react if you go into production and someone detects a flaw you didn't think of.
