The Low Base Rate Problem (2014)
To preface this comment, I generally really like Evan Miller&#x27;s writing on statistics, but this statement:<p>&gt; anyone who lacks a firm understanding of statistical power should not be designing or interpreting A&#x2F;B tests.<p>Completely ignores that you can approach A&#x2F;B testing from an entirely Bayesian perspective (which Miller has written about fairly well in the past).<p>I might be a bit biased in favor of Bayesian methods, but I&#x27;m honestly a bit surprised that many people still run A&#x2F;B tests using a frequentist frameworks such as this.<p>The biggest reason I would argue to dismiss the NHST (Null-hypothesis significance test) approaching is because <i>A&#x2F;B Tests are not really controlled experiments</i>, at least not in the same way clinical trials are. User behavior is always observational even if you have a control group. Whether you go NHST or Bayesian is a coin toss in a controlled environment, but when looking at user behavior on a web site it&#x27;s much better to think in Bayesian terms.<p>Other reasons why a Bayesian frame work should be used:<p>- Marketers care about &quot;Probability A is better than B&quot;, not &quot;failure to reject null hypothesis&quot;.<p>- If you&#x27;ve been running A&#x2F;B tests for years, you most certainly have a very good prior over the distribution of conversion rates. It&#x27;s literally wasting your time and money not to make use of these.<p>- If you&#x27;re measuring human behavior on a website, there are absolutely confounding variables at play and you want to model these explicitly. It&#x27;s much easier to do this when thinking in a Bayesian framework.<p>I&#x27;ve been designing and running A&#x2F;B test of all sorts of complexity at all sorts of companies for a very long time, and my experience has been that approaching the problem from a Bayesian perspective is by far the superior way to run your tests.