A/B testing gets misused to juice metrics in the short term
&quot;As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money.&quot;<p>This is a good touchstone to use for &quot;you&#x27;ve overoptimized your site, tone it back&quot;. I am also taken aback every time I&#x27;m on a site, I&#x27;ve got something in my shopping cart, I&#x27;m headed for the &quot;check out&quot; button, or I&#x27;m even <i>on the checkout page</i>, and some stupid interstitial pops up. Dude, I&#x27;m trying to enter my credit card information! Back off! Especially stupid for a &quot;sign up for our newsletter&quot; popup; we all know that unclicking the &quot;yes, we can email you every 17 seconds from now until the heat death of the universe with valuable offers from &#x27;our affiliates&#x27; which we define as &#x27;anyone we share a species with&#x27;!&quot; box on the checkout form is mandatory, and if we don&#x27;t see it immediately we&#x27;d best go hunting for it. You&#x27;ve already default populating the checkbox to &quot;yes&quot; on this <i>very screen</i>, get out of the way!<p>Less unbelievably stupid, but related, is when I&#x27;m examining product X and just after I scroll down a bit to read more you pop up something related to... well... anything other than product X! I&#x27;m signalling interest in product X as hard as I can, and you&#x27;ve AB tested that this is a great time to jangle your keys over there instead? Your AB testing is stupid and can&#x27;t possibly fail to be some stupid statistical fluke or other terrible error. What fisherman goes out on his boat, hooks a fish, and then rushes to throw another completely different lure out to the hooked fish and get them on that hook instead? This is another good touchstone for being &quot;overoptimized&quot;.