Understanding Statistical Power and Significance Testing
I am an engineering manager in a large ecommerce company, overseeing machine learning and our in-house A&#x2F;B testing framework.<p>One of our big tenets in my organization is that null-hypothesis significance testing and any associated methodologies (random effects, frequentist experiment design, and various enhancements to fgls regression) are simply not applicable and not useful to answering questions of policy (any kind of policy, which subsumes pretty much all use cases of running tests).<p>We take a Bayesian approach from the ground up, put lots of research into weakly informative priors for all tests, develop meaningful posterior predictive checks and we state policy inference goals up front to understand what we are looking for (predictive accuracy? measurement of causal effect sizes? understanding risk in choosing between different options that have differently shaped posteriors?).<p>One solid paper is this:<p><a href="http:&#x2F;&#x2F;www.stat.columbia.edu&#x2F;~gelman&#x2F;research&#x2F;published&#x2F;retropower20.pdf" rel="nofollow">http:&#x2F;&#x2F;www.stat.columbia.edu&#x2F;~gelman&#x2F;research&#x2F;published&#x2F;retr...</a><p>which discusses “type m” (magnitude) and “type s” (sign) error probabilities in Bayesian analyses, and how that can provide some benefits and flexibility that NHST methods a cookie cutter power designs cannot.<p>Your mileage may vary, but my org has found this to be night and day better than frequentist approaches and we have no interest in going back to frequentist testing really for any use case.