Relevance of assumption of normality, ways to check and reading recommendations for non-statisticians I started reading around the topic of modern robust methods, consulted various statistics texts and did some research on the CV forum. I ended up being rather confused regarding the relevance of the assumption of normality.
While some authors state that even small deviations from normality can, under certain circumstances, cause major issues when using classical parametric tests, others argue the assumption of normality was not that important.
Recommendations on how to check (if at all) the assumption of normality vary broadly. For a non-statistician as myself, who uses textbooks as a basis to analyse research data, it is difficult to identify trustworthy resources and procedures that are well-accepted in the field.
Can anyone recommend a source with helpful and accessible information on current recommendations?
Edit
For some more background: My research is in the area of psychology, which - in my case - often requires t-tests, ANOVAS (often two-ways) or regressions. On a regular basis, I encounter issues with skewed distributions (e.g. in some groups, many people chose or obtained the/one of the highest values - which even makes sense content-wise, but still violates the assumption of normality). I found the skewed data to be difficult to "correct" in many cases (because the skew is so extreme or because groups differed in their skew, so the transformation didn't solve some of the issues or even caused new ones). This is why I started looking for alternatives and came across the modern robust methods (e.g. robust two-way ANOVA, WRS2 package for R).

You are right to be confused, as this is a confusing issue indeed.
I'm afraid it will be hard to find everything good to know in a single reference, but some may give that to you - I will not look around but rather tell you what I think.
(This answer is more about the background, how to think about model assumptions in general; I have written another answer with some more practical hints.)

*

*Models are idealisations and they never hold precisely in practice, so we will routinely apply methods to data for which the methods' model assumptions are more or less obviously violated.


*Having a model assumption for a certain method does not mean that the assumption has to be fulfilled for the method to make sense. It only means that if the model assumption is fulfilled, there is a theoretical guarantee that the method does what it's supposed to do (these guarantees are not always the same and can be stronger or weaker). The tricky issue here is that obviously there is then no theoretical guarantee in case that the model assumption is violated; and in fact it is both possible that our analysis is still fine, or that it is misleading, and it is hard to tell these two possibilities apart.


*Many statistics are asymptotically normally distributed even if the underlying data are not normal due to the Central Limit Theorem (CLT; which itself has assumptions that may be violated, but see above). This means that if your sample size is large, many deviations from normality are not problematic as results from assuming normality will hold approximately.


*There is a catch with item 3, which is that it depends on the unknown true underlying distribution how large a sample size is actually required so that results are satisfactorily normal (also it depends on what exactly you do because the CLT doesn't apply to everything; it does apply to the arithmetic mean though, on which many statistics are based).


*The key issue is not whether data are normal or not, and even not whether data are approximately normal, but rather whether normality is violated in ways that will mislead conclusions. In fact, some tiny violations of normality (a single gross outlier) may be harmful, whereas a distribution such as the uniform that looks very obviously non-normal will usually not cause problems for inference based on a normality assumption.


*Depending on what exactly you do, in most cases the following things are most problematic: Extreme outliers (either observations where something is wrong or essentially different from the others, or generally heavy distributional tails) and strong skewness. On the other hand, if there are no heavy tails, normality based inference is very often harmless, for example for discrete 5-point (or other) Likert scaled data (data that can only take values -2,-1,0,1,2 and are therefore pretty much guaranteed to not have outliers).


*In many situations other violations of model assumptions such as dependence are more critical than non-normality. In particular, the CLT requires independently identically distributed data or some alternative assumptions that are not much weaker. I have read once (I think in the Hampel et al. book on Robust Statistics) that many high quality astronomical data sets are heavy tailed, and some of them look more normal than they should due to long range dependence - and for the sake of analysis one would be better off with less normality and less dependence.
