Objective criteria for assumption violations that do not utilize p-values? Suppose we have a standard regression model and want to identify whether we have violated the assumptions of the model. Traditionally, we might utilize a significance test to determine whether (for example) the residuals are normally distributed (where the null model states that the residuals belong to a normal distribution).
I think there's consensus that using hypothesis tests to evaluate model assumptions isn't the best idea. These tests are sensitive to sample size (failing to detect major departures when N is small, and detecting even minor discrepancies when N is large). Personally, I think the biggest problem with these tests is they evaluate whether violated assumptions can be detected, not whether they are problematic.
When I evaluate assumptions, I exclusively use visuals. Unfortunately, visuals are more subjective (and they don't really tell you whether the observed violation is problematic). So, here's my question:
Are there any objective ways to evaluate statistical assumptions that do not rely on significance tests?
And, do these objective means of evaluating assumption evaluate whether the assumption violation matters, rather than whether it can be detected?
I'd love to see references to articles that I can study, if you have it. Thanks!
Edit
Some of the comments/answers mention that it depends on what one's research goals are. Yes, I absolutely agree and appreciate you all pointing this out. I suppose what I'm asking for is some literature that says, "If your goal is A, index B is appropriate. If your goal is C, criteria D is appropriate."

This is a good question, as it acknowledges that the issue is not whether model assumptions are fulfilled or not (they never are), but rather whether violations of the model assumption matter (in terms of misleading conclusions).
Unfortunately I tend to answer this question with "no". A problem is this: The statistical problem of interest is well defined within the framework of the assumed model, but if model assumptions are violated, there is no unique objective way to define what it actually means to say that conclusions are misled.
Here is an illustration. Let's say you are running a test about the mean of a normal distribution, but the underlying distribution is in fact skew. Now your sample size may be fairly large and there may not be any indication that second moments do "explode", so the Central Limit Theorem may justify normal theory to hold approximately (violation of normality could therefore be seen as not problematic). However, in most skew distributions mean, mode, and median are different, whereas in the normal distribution they are the same, meaning that even though the CLT authorises inference about the mean, it isn't clear whether in your specific application you should rather be interested in median or mode in order to summarise your distribution. Under the normal assumption this doesn't matter, but if the underlying distribution is in fact skew, it usually does, and the answer whether the normal assumption theory is fine or not depends on this.
Here is a paper I had my hand in on investigating the quality of formal misspecification tests for assumptions in testing, with connected considerations. Overall we're not as negative about this approach as some others, even though we agree that in a good number of situations it is not very good. But sometimes it's fine. Unfortunately this depends on a number of specifics of the situation.
M. I. Shamsudheen & C. Hennig: Should we test the model assumptions before running a model-based test?
https://arxiv.org/abs/1908.02218
Two maybe interesting papers that we cite:
Zimmerman DW (2011) A simple and effective decision rule for choosing a significance test to protect against non-normality. British Journal of Mathematical and Statistical Psychology 64:388-409 (Here an objective rule different from a formal assumptions test is proposed to choose between a t-test and a nonparametric test; namely to use the nonparametric test in case they have very different results.)
Spanos A (2018) Mis-specification testing in retrospect. Journal of Economic Surveys 32:541–577 (good and thoughtful survey paper even though I don't agree with everything)
I believe that developing decision rules between model-based and alternative procedures that have better performance than standard misspecification tests is a very promising research area (even though there will always have to be some "subjective" decision making, see above), but as far as I know there isn't much.
PS: Regarding the Zimmerman paper, I have seen a similar suggestion for linear regression somewhere, namely to run a least squares and a robust regression, and to use the least squares one if both are reasonably in line (of course one can define this "objectively"), and otherwise to use the robust fit. I don't remember exactly where that was anymore, somewhere in the robustness literature, and as far as I remember, it was defined but not much (or even nothing) was done to compare its quality to alternative approaches including running either least squares or robust fit all the time.
