&gt; since you know those groups are seeing the same thing and should be performing identically<p>That&#x27;s not how A&#x2F;B testing works.  95% confidence means you should expect a 5% false positive rate, i.e., you should expect the difference measured in an A&#x2F;A test to be statistically significant 5% of the time.  You&#x27;ll always measure <i>some</i> difference, since no two random samples will be 100% identical in every regard.<p>The procedure you and the parent propose is tantamount to selecting 1 out of every 20 test results and discounting it for no real reason.  It adds extra cost to your A&#x2F;B testing without producing more reliable results.<p>See also: <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;882&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;882&#x2F;</a><p>It&#x27;s a different matter if you&#x27;re running multiple A&#x2F;A-type tests over an extended period of time to ensure that the false positive rate is actually 5%, a kind of meta-statistical test.  As a sanity check this is sound, but vastly more expensive than what the OP is proposing (for example).  I&#x27;ve never seen anyone use A&#x2F;A, A&#x2F;A&#x2F;B, A&#x2F;A&#x2F;B&#x2F;B, etc. tests in this way.  Rather, I&#x27;ve only ever seen them used as you and the OP suggest: the two A buckets should be &quot;the same&quot; and if they aren&#x27;t, the results should be thrown out.