When to check model assumptions Statistical methods are based on model assumptions. For example, an independent one-way ANOVA makes the following assumptions:

*

*Normally distributed residuals


*Homogeneity of variance


*Independence of observations
Whether or not these assumptions are met will influence the reliability of the independent one-way ANOVA’s results and the conclusions drawn from them (to varying degrees, depending on what assumptions are violated and how).
My question is: When should we check the assumptions of our model? Is it preferable to first check model assumptions or inspect model fit? How might that influence the interpretations and decisions we make thereafter, and why might this be preferable to the other approach?
In the case of general linear models, we first need to fit our model, otherwise we cannot test whether the residuals are normally distributed. But immediately after that we could either choose to check model assumptions or inspect model fit.
In particular I am interested in answers that speak to any of the following three approaches:

*

*The purpose of checking model assumptions is to decide whether the originally chosen test is appropriate for the data, so assumptions should be checked first. A different, more appropriate, test should be used if assumptions are violated, and conclusions should be drawn from this test. This approach is endorsed in textbooks (e.g., Dowdy et al., 2004), and is also the one I’ve encountered in statistics courses I’ve taken.


*The purpose of checking model assumptions is to assess the quality of the model we originally chose in light of our data, so assumptions should be checked second. Depending on the severity of violations, conclusions drawn from test results might be reigned in, or the model might be respecified. This seems to be the approach endorsed by Fisher (see Spanos, 2017).


*Model assumptions are often violated in the real world, so there’s no need to check them. Instead we should choose better default tests that are less constrained and stick with those (e.g., Declare et al., 2017, 2019). This is also a popular approach, for example, see Section 4 in the Spanos, 2017 paper cited above.
This preprint has some good discussion and examples comparing the performance of these approaches in different circumstances and with different tests. It concludes:

“In some setups either running a less constrained test or running the model-based test without preliminary testing have been found superior to the combined procedure involving preliminary [assumption checking to guide test selection].” However, “a sober look at the results reveals that the combined procedures are almost always competitive with at least one of the unconditional tests, and often with them both. It is clear, though, that recommendations need to depend on the specific problem, the specific tests involved. Results often also depend on in what way exactly model assumptions of the model-based test are violated, which is hard to know without some kind of data dependent reasoning.”

So the best performing approach depends on circumstances and the test used. However, if you were to pick one of the previously mentioned approaches as a general principle to follow when better information isn’t available, which would make the most preferable default?
Please base your answers on experience or evidence.

Some remarks:

*

*Model assumptions are never fulfilled in reality, and there is therefore no way to make sure or even check that they are fulfilled. The correct question is not whether model assumptions are fulfilled, but rather whether violations of the model assumptions can be suspected that mislead the interpretation of the results. Note that standard model misspecification tests are not always good at assessing this.


*Of course this depends on knowledge about what kinds of violations of the model assumptions have what kind of impact. Regarding standard ANOVA, extreme outliers are known to be harmful; skewness may make inference based on means questionable; difference in variances in the first place raises the question how to interpret the obviously existing differences between groups if differences in variance are much clearer and more characteristic of the situation than differences in means. Some literature suggests that what impact heteroscedasticity has and how much of a problem it is depends on the group sizes and whether bigger variance occurs in a smaller or larger group. Mild violations of these model assumptions are very often harmless in that characteristics of the test behaviour are not affected much. Also, with larger numbers of observations, the Central Limit Theorem will justify the application of normal theory for most non-normal distributions. Dependence between observations though can very often be harmful and should be modelled if it is clear enough to be detected.


*Note that dependence is as much a problem for any non-parametric, robust, or "model-free" alternatives to ANOVA as for standard ANOVA. Note also (example given in the preprint) that there are situations in which standard ANOVA is better than robust/non-parametric alternatives even if standard ANOVA model assumptions are violated whereas non-parametric assumptions hold (this may happen for example for lighter tailed distributions than the normal)! This is because parametric theory is often concerned about optimality (which is lost in theory if assumptions are not met, but the parametric method can still be rather good), whereas nonparametric and robust theory are often concerned about general minimum/worst case quality assurances, which means that these methods, even though legitimate,  may not be particularly good in some cases even where their assumptions are fulfilled. This isn't always the case: there are situations in which standard ANOVA breaks down and nonparametric/robust alternatives are much better, particularly with outliers. Robust methods may be slightly better than standard ANOVA in case of heteroscedasticity but nonparametric alternatives such as Kruskal-Wallis or permutation testing may not (I'm actually not sure, not even whether this has been investigated in detail; the last statement is rather my intuition). Permutation tests will also still be affected by outliers as long as their test statistic is non-robust! The message here is that nonparametric/robust/"model free" approaches are not a silver bullet.


*Technically (as also mentioned in our paper which is cited as "this preprint" in the question) applying misspecification tests, and then running an ANOVA on the same data conditionally on not having rejected model assumptions will invalidate the standard ANOVA-tests as correctly written in a comment by @rep_ho. This is due to the "misspecification paradox": see https://stats.stackexchange.com/a/592495/247165
Advice by @Björn is good to test model assumptions on independent (hopefully equally distributed) data where possible.
Note also that this problem does apply not only to formal misspecification testing, but also to looking at data visualisation (of the same data) and making decisions about whether to run the standard ANOVA dependent on these visualisations.


*However, given that model assumptions never hold anyway, violation of assumptions due to the misspecification paradox is usually preferable to leaving some bigger problems undetected because of not checking the model assumptions. There are even some situations in which one can show that misspecification testing will not affect the characteristics of the later ANOVA in case the assumed model held before misspecification testing, namely where the misspecification test is independent of the finally used statistic. This happens for example when running misspecification tests on only the residuals in the linear model, and then running a standard normality-based test only if normality is not rejected.


*Generally the formal impact of misspecification testing and looking at data will be small under the assumed model if fulfilled assumptions are rejected with low probability, i.e., testing at low level, or, when looking at the data, really only rejecting the model if violations are clear and strong.


*Now getting to recommendations what to do actually, I don't have a general recipe. When doing consultation, whatever I do or advise will depend on all available background information about the subject matter and aim of the study. This concerns for example the question whether the mean is an appropriate  good statistic to summarise group-wise results in case of skewness or heteroscedasticity. It also concerns potential consequences of type I and type II errors (outliers will not normally cause type I errors, but can increase the probability of type II errors big time), and more generally how "strongly" results will be interpreted.


*The first thing to do is always, if possible before having seen the data, to think about potentially harmful model assumption violations that could occur in the given situation. Are potential sources for dependence known? These should then be modelled or people should think hard about methods for data gathering where these problems don't occur. Dependence can be harmful, even in cases where it is hard or impossible to detect from the data! See https://arxiv.org/abs/2108.09227. Also if I know in advance that outliers can be expected to occur for this kind of data, I'd use something robust or nonparametric. On the other hand, there are situations in which the occurrence of outliers can be ruled out because the value range of outcomes is limited, and it can also with good reasons be expected that there is enough variation that not almost all observations concentrate on one side of the scale (in which case stray observations on the other side could still be seen as outliers).


*I'm a curious person and just the fact that a certain test or check rejects the model assumption wouldn't stop me from running a method that uses this assumption if I'd have used it otherwise. For a number of checks (based on residuals) this is necessary anyway. If I don't have any indication before seeing the data against standard ANOVA (see above), I'd run it. (Obviously "legal" and procedural concerns such as pre-registration have to be taken into account.)


*I'd also look at visual model diagnostics. I don't necessarily use formal misspecification tests because given the earlier discussion I know what kind of deviations from the model I look for, and misspecification tests don't know that. However, somebody with less experience and knowledge about these things will often do better running a formal misspecification test than not doing anything at all; also regularly running misspecification tests and comparing them with the visual impression gives us experience about what kind of apparent deviations from the model can still be compatible with random variation. So I run misspecification tests for informing my intuition rather than for making formal decisions about how to proceed. Anyway, such diagnostics may or may not prompt me to run a nonparametric or robust alternative on top of the standard ANOVA rather than "instead".


*Most importantly, I will also look at the raw data and boxplots. The question that I ask here is not in the first place "are the model assumptions violated?" (That's anyway the case.) Rather I ask: "Do I get the impression that the message from looking at the data about group differences is in line with what my tests say?" If yes, I can interpret the test results with some confidence (also possibly confirmed by the observation that standard ANOVA and an alternative approach convey the same message). If not I try to understand exactly how the data led the tests astray (or how my intuition went wrong - this may happen, too), and I may then explain both what I see and why one or more test results are not in line with it. (This can also concern meaningless significant results because of a so large sample that a meaninglessly small effect came out significant; or on the other hand insignificance even if a too small sample visually suggests differences between groups. Obviously one should also look at effect sizes and confidence intervals.)
Overall I'm probably closest to option 2 in the list in the question.
