<p><pre><code>    statistical significance is meaningless when split
    testing websites (you are guaranteed to archive
    significance if you run the test long enough
</code></pre>
Wait, what?  Not if you use the right test to evaluate your change.  This article isn&#x27;t showing that A&#x2F;B testing is bunk, it&#x27;s showing that Optimizely is using an inappropriate test.<p><pre><code>    Do an a&#x2F;b test with identical versions of you site.
    Spoiler: With 100% certainty, you&#x27;ll see both significant
    results and a sizable effect size difference.
</code></pre>
The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical.  Running your A vs A experiment and evaluating it correctly you&#x27;ll find results significant at the p &lt; N rate N% of the time.  And the longer you run it the lower your estimate of the effect size will be.