One of the must-read papers in this area is a very short, very readable piece by Chris Achen called &quot;Let&#x27;s Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong.&quot; [0]<p>The basic idea is that if your data experiences non-linear coding errors due to some external process before it gets to you, then as related to the true causal relationship between your variables, you&#x27;re actually fitting a (possibly) non-linear transformation.<p>If you do this and still use the classical cook-book t-stat &#x2F; p-value stuff, you can run into big problems. As the paper shows, even for an extremely simple data set where there is merely a non-linear coding error, the coefficient can be very statistically significant (t-stat magnitude &gt; 2) and yet <i>it will have the wrong sign.</i><p>This is a really remarkable thing. By just jittering your data a bit, you can obtain a result that passes the usual, simplistic significance tests but for which the effect size <i>is negative when it should be positive</i> (or vice versa).<p>I remember when I worked in quant finance and we were using a terrible, shamefully bad automated framework for frequentist model fitting, the sense of dread was huge. If the data we used had been jittered by the data vendor, or if there was a slight bug in the code that did variable cleaning, outlier manipulation, or scoring, or any of a hundred other steps where tiny non-linearities could be introduced, then it would be entirely possible for us to see &quot;significant&quot; results that pointed in the opposite direction from the truth.<p>There are many other papers which point out the pitfalls of naive dump-it-all-on-the-right-hand-side regression models, but I think this one from Achen is unique in that it is extremely short, extremely simple to follow, and yet it is fully devastating to this entire technique of modeling.<p>[0] <a href="http:&#x2F;&#x2F;www.columbia.edu&#x2F;~gjw10&#x2F;achen04.pdf" rel="nofollow">http:&#x2F;&#x2F;www.columbia.edu&#x2F;~gjw10&#x2F;achen04.pdf</a>