When to check model assumptions Statistical methods are based on model assumptions. For example, an independent one-way ANOVA makes the following assumptions:

*

*Normally distributed residuals


*Homogeneity of variance


*Independence of observations
Whether or not these assumptions are met will influence the reliability of the independent one-way ANOVA’s results and the conclusions drawn from them (to varying degrees, depending on what assumptions are violated and how).
My question is: When should we check the assumptions of our model? Is it preferable to first check model assumptions or inspect model fit? How might that influence the interpretations and decisions we make thereafter, and why might this be preferable to the other approach?
In the case of general linear models, we first need to fit our model, otherwise we cannot test whether the residuals are normally distributed. But immediately after that we could either choose to check model assumptions or inspect model fit.
In particular I am interested in answers that speak to any of the following three approaches:

*

*The purpose of checking model assumptions is to decide whether the originally chosen test is appropriate for the data, so assumptions should be checked first. A different, more appropriate, test should be used if assumptions are violated, and conclusions should be drawn from this test. This approach is endorsed in textbooks (e.g., Dowdy et al., 2004), and is also the one I’ve encountered in statistics courses I’ve taken.


*The purpose of checking model assumptions is to assess the quality of the model we originally chose in light of our data, so assumptions should be checked second. Depending on the severity of violations, conclusions drawn from test results might be reigned in, or the model might be respecified. This seems to be the approach endorsed by Fisher (see Spanos, 2017).


*Model assumptions are often violated in the real world, so there’s no need to check them. Instead we should choose better default tests that are less constrained and stick with those (e.g., Declare et al., 2017, 2019). This is also a popular approach, for example, see Section 4 in the Spanos, 2017 paper cited above.
This preprint has some good discussion and examples comparing the performance of these approaches in different circumstances and with different tests. It concludes:

“In some setups either running a less constrained test or running the model-based test without preliminary testing have been found superior to the combined procedure involving preliminary [assumption checking to guide test selection].” However, “a sober look at the results reveals that the combined procedures are almost always competitive with at least one of the unconditional tests, and often with them both. It is clear, though, that recommendations need to depend on the specific problem, the specific tests involved. Results often also depend on in what way exactly model assumptions of the model-based test are violated, which is hard to know without some kind of data dependent reasoning.”

So the best performing approach depends on circumstances and the test used. However, if you were to pick one of the previously mentioned approaches as a general principle to follow when better information isn’t available, which would make the most preferable default?
Please base your answers on experience or evidence.

It's useful to distinguish different scenarios.
In the first scenario, strict type I error control, as well as pre-specification (otherwise the first is hard to imagine), are required. E.g. think of a randomized Phase III trial for a new drug. In this scenario, you would check your assumptions before you even run your trial using data that is as relevant as possible. Because it is well known that switching analysis methods based on some pre-tests (e.g. for normality*) leads to some type I error inflation (and other issues), you would not do an analysis strategy where you only check assumptions and determine your analysis method based on that (even if you could pre-specify the conditions and how it influences the analysis). Of course, certain small deviations can be looked at and judged to be somewhat irrelevant (e.g. there's a reason why things like blood pressure are usually analyzed using linear regression with a baseline value in the model, even though we know the model cannot be precisely true since blood pressure cannot be negative) and ignored, because many models are very robust to mild deviations. On the other hand, for other situations some appropriate re-medial actions are needed (e.g. we know it's much better to log-transform urine albumin-creatine ratios than to analyze untransformed values).
Many scientific investigations fall into this first category, even if they are not as tightly regulated as drug trials or overseen by regulatory bodies. However, very often we have a clear up-front plan and relevant similar data are available and allow us to check our assumptions to a sufficient degree that assumption checking on the actual data is usually more or less a waste of time.
The second setting is when things are completely new, we are trying out new measurement methods, trying to understand something about a new system (or how a new drug behaves in the human body) and so on. In this setting, there may just not be anything similar we can base reliable assumptions on. Here, we may have to be quite flexible and explore a good bit. However, then we should also not sell this as a definite confirmatory experiment. That's why some larger definite experiments have pilot parts to be able to make assumptions.
The third setting is the most difficult: When you don't know for sure whether you are in the first or the second setting, because there's some data/prior knowledge that may or may not be relevant and/or may be ambiguous and/or something might turn out to be very different (e.g. the first immuno-Oncology treatments with delayed treatment effects). However, it's very hard to say something general about these situations.
* Based on the discussion under another post, here's an illustration of the pre-test for normality using only residuals causing type I error inflation.
library(tidyverse)

set.seed(1234)
corrs <- map_dfr(1:1000000,
                     function(x){
                       sim_data <- tibble(y=rnorm(n=50), x=rbernoulli(n=50, p=0.5))
                       lmfit <- lm(y~1+x, data=sim_data)
                       wtres <- wilcox.test(sim_data$y[sim_data$x==0], sim_data$y[sim_data$x==1])
                       data.frame(test_statistic = summary(lmfit)$coeff["xTRUE", "t value"],
                              lm_pval = summary(lmfit)$coeff["xTRUE", "Pr(>|t|)"],
                                  wt_pval = wtres$p.value,
                              shapiro_wilk_pval = shapiro.test(lmfit$residuals)$p.value)
       })

corrs %>%
  mutate(conditional = ifelse(shapiro_wilk_pval<=0.1, wt_pval, lm_pval),
         signif1 = (lm_pval<=0.05),
         signif2 = (wt_pval<=0.05),
         signif3 = (conditional<=0.05)) %>%
  summarize(lm_lcl = qbeta(0.025, sum(signif1), 1+length(signif1)-sum(signif1)),
            lm_ucl = qbeta(0.975, 1+sum(signif1), length(signif1)-sum(signif1)),
            wilcox_lcl = qbeta(0.025, sum(signif2), 1+length(signif2)-sum(signif2)),
            wilcox_ucl = qbeta(0.975, 1+sum(signif2), length(signif2)-sum(signif2)),
            cond_lcl = qbeta(0.025, sum(signif3), 1+length(signif3)-sum(signif3)),
            cond_ucl = qbeta(0.975, 1+sum(signif3), length(signif3)-sum(signif3)))

This gives this result for tests with a nominal level of 0.05 showing a type I error inflation when doing a pre-test for normality.




Analysis method
Clopper-Pearson confidence interval for type I error rate




Linear model
0.0495 to 0.0503


two-sample Wilcoxon aka Mann-Whitney test
0.0482 to 0.0490


Conditional (switch to Wilcoxon if Shapiro-Wilk test of normality of residuals has p-value<=0.1
0.0512 to 0.0521



