<p><pre><code>    With a big enough N, the probability of reaching
    significance approaches 1.
</code></pre>
Well, not with an A vs A experiment.  Then the probability of significance at the p &lt; 0.05 level should stay 5%.<p>On the other hand, in an A vs B experiment where B is very slightly different from A you&#x27;re right that with a large enough population you become very likely to reach a p &lt; 0.05 significance level, just with an extremely tiny effect size.  The experiment has told you something about the difference between A and B, and you now have high certainty, but it&#x27;s just such a small difference that you probably don&#x27;t care.<p><pre><code>    Have you ever approached a tool in the field that
    would fail my proposed test?
</code></pre>
Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there&#x27;s a difference significant at the 0.05 level more than 5% of the time?  All the tools I&#x27;ve used for A&#x2F;B testing would do fine on this test.  This includes a custom tool at a computational advertising startup I worked at, Google&#x27;s internal experiment platform, and Google Analytics&#x27; Content Experiments.<p>(If I didn&#x27;t trust a tool to pass this test I&#x27;d just use it to get the counts for both sides of the experiment and do that stats on my own.)