When to check model assumptions Statistical methods are based on model assumptions. For example, an independent one-way ANOVA makes the following assumptions:

*

*Normally distributed residuals


*Homogeneity of variance


*Independence of observations
Whether or not these assumptions are met will influence the reliability of the independent one-way ANOVA’s results and the conclusions drawn from them (to varying degrees, depending on what assumptions are violated and how).
My question is: When should we check the assumptions of our model? Is it preferable to first check model assumptions or inspect model fit? How might that influence the interpretations and decisions we make thereafter, and why might this be preferable to the other approach?
In the case of general linear models, we first need to fit our model, otherwise we cannot test whether the residuals are normally distributed. But immediately after that we could either choose to check model assumptions or inspect model fit.
In particular I am interested in answers that speak to any of the following three approaches:

*

*The purpose of checking model assumptions is to decide whether the originally chosen test is appropriate for the data, so assumptions should be checked first. A different, more appropriate, test should be used if assumptions are violated, and conclusions should be drawn from this test. This approach is endorsed in textbooks (e.g., Dowdy et al., 2004), and is also the one I’ve encountered in statistics courses I’ve taken.


*The purpose of checking model assumptions is to assess the quality of the model we originally chose in light of our data, so assumptions should be checked second. Depending on the severity of violations, conclusions drawn from test results might be reigned in, or the model might be respecified. This seems to be the approach endorsed by Fisher (see Spanos, 2017).


*Model assumptions are often violated in the real world, so there’s no need to check them. Instead we should choose better default tests that are less constrained and stick with those (e.g., Declare et al., 2017, 2019). This is also a popular approach, for example, see Section 4 in the Spanos, 2017 paper cited above.
This preprint has some good discussion and examples comparing the performance of these approaches in different circumstances and with different tests. It concludes:

“In some setups either running a less constrained test or running the model-based test without preliminary testing have been found superior to the combined procedure involving preliminary [assumption checking to guide test selection].” However, “a sober look at the results reveals that the combined procedures are almost always competitive with at least one of the unconditional tests, and often with them both. It is clear, though, that recommendations need to depend on the specific problem, the specific tests involved. Results often also depend on in what way exactly model assumptions of the model-based test are violated, which is hard to know without some kind of data dependent reasoning.”

So the best performing approach depends on circumstances and the test used. However, if you were to pick one of the previously mentioned approaches as a general principle to follow when better information isn’t available, which would make the most preferable default?
Please base your answers on experience or evidence.

Some general observations: First, we tend to spend a lot of time using non-robust models when there are many robust options that have equal or greater statistical power.  Semiparametric (ordinal) regression for continuous Y is one class of model that is not used often enough.  Such models (e.g., the proportional odds model) do not depend on how Y is transformed (as long as the transformation is rank-preserving) and are not affected by outliers in Y (they have the same problems as parametric models with regard to outliers in X).  Routine use of semiparametric models would result in fewer assumptions in need of checking.
Second, goodness of fit needs to be judged against alternatives.  What if a more flexible, well-fitting model requires many more parameters and the net effect is overfitting that makes predictions unreliable and/or makes confidence intervals too wide?  Almost always, some kind of analysis is required, and the badness of fit of a proposed model should not rule the day, when the proposed alternative analysis is actually worse.
On the second point, it is often more useful to think of assessment of goodness of fit in terms of a contest between the proposed model and a more general model.  A fully worked out example is here.  In that example the proposed model is a proportional odds model in which the effect of a treatment on lowering the odds that $Y\geq j$ is assumed to be the same for all $j$ save the lowest value.  There are at least two alternative models: a partial proportional odds model that relaxes this assumption with respect to treatment, and a multinomial logistic model that relaxes the assumption with respect to all predictors.  I cast model checking as a contest between these models and use two metrics for comparison: AIC and bootstrap confidence intervals for differences in predicted probabilities of various $Y=j$ from pairs of models.  I also show formal likelihood ratio $\chi^2$ tests of goodness of fit by pairwise comparison of these three models.  I show that there is a cost of not assuming proportional odds.
The above setting is exactly analogous to comparing (1) a linear model with constant variance with (2) a linear model that allows the residual variance $\sigma^2$ to be a function of X.
For continuous Y a very cogent approach in my view is to hope for a simple linear model but to allow for departures from that.  A Bayesian model could put priors on the amount of variation of $\sigma^2$ and on the amount of non-normality of residuals.  By tilting the model towards normal residuals and constant $\sigma^2$ but allowing for departures from those as the sample size allows, one obtains a general solution that does not require binary model choices.  A simple example of this is the Bayesian $t$-test.
