Disclaimer: I&#x27;m the director of data science at VWO, an Optimizely competitor.<p>In my view, the issue is not one-tail vs two-tail tests, or sequential vs one-look tests at all. The issue is a failure to quantify uncertainty.<p>Optimizely (last time I looked), our old reports, and most other tools, all give you improvement as a single number. Unfortunately that&#x27;s BS. It&#x27;s simply a lie to say &quot;Variation is 18% better than Control&quot; unless you had facebook levels oftraffic. An honest statement will quantify the uncertainty: &quot;Variation is between -4.5% and +36.4% better than Control&quot;.<p>When phrased this way, it&#x27;s hardly surprising that deploying this variation failed to achieve an 18% lift - 18% is just one possible value in a wide range of possible values.<p>The big problem with this is that customers (particularly agencies who are selling A&#x2F;B test results to clients) hate it. If we were VC funded, we might even have someone pushing us to tell customers the lie they want rather than the truth they need.<p>Note that to provide uncertainty bounds like this, one needs to use a Bayesian method (only us, AB Tasty and Qubit do this, unless I forgot about someone).<p>(Frequentist methods can provide confidence intervals, but these are NOT the same thing. Unfortunately p-values and confidence intervals are completely unsuitable for reporting to non-statisticians; they are completely misinterpreted by almost 100% of laypeople. <a href="http:&#x2F;&#x2F;myweb.brooklyn.liu.edu&#x2F;cortiz&#x2F;PDF%20Files&#x2F;Misinterpretations%20of%20Significance.pdf" rel="nofollow">http:&#x2F;&#x2F;myweb.brooklyn.liu.edu&#x2F;cortiz&#x2F;PDF%20Files&#x2F;Misinterpre...</a> <a href="http:&#x2F;&#x2F;www.ejwagenmakers.com&#x2F;inpress&#x2F;HoekstraEtAlPBR.pdf" rel="nofollow">http:&#x2F;&#x2F;www.ejwagenmakers.com&#x2F;inpress&#x2F;HoekstraEtAlPBR.pdf</a> )