I think we agree. I should have elaborated more.<p><i>&gt; Wait, what? Not if you use the right test to evaluate your change.</i><p>Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning?<p>The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1.<p><i>&gt; The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical.</i><p>Yes, I know. But the magic lies in that <i>&quot;done properly&quot;</i>. Have you ever approached a tool in the field that would fail my proposed test? I haven&#x27;t.