20 lines of code that beat A/B testing every time
This is thought-provoking, which is good.  However there are significant issues with the approach.<p>1. Real world performance varies over time.  For instance there are typically daily, weekly and monthly conversion rate fluctuations.  Not an issue for A/B testing, but a big issue for this approach if a random switch in direction happens at the same time that conversion fluctuations happen to head in a good direction.<p>2. (This is really a special case of #1 - but a very, very important special case.)  This approach creates long-lasting interaction effects between tests and independent changes.  That requires explanation.  Suppose you're running a test.  Version A is somewhat better.  But version B is temporarily looking slightly better when you make a significant improvement to your website (maybe you started another test that works).  Now you're adding a lot of good traffic to version B (good because of the other change) and very little of the new good traffic to version A.  This new version B traffic soundly beats your old version A traffic.  This correlation between time and website performance will continue until the old version A traffic is completely swamped by new version A traffic.  With only 5% of your traffic going to version A, this can easily take 100x as long as your test has been running - or more.  (Properly constructed A/B tests do not suffer this statistical anomaly.)<p>3. Code over time gets messy.  One of the most important characteristics of A/B testing is that you can delete the mess and move on.  With this approach you can't - it just hangs around adding to your technical debt.<p>4. Businesses are complex, and often have multiple measures they would like to balance.  For instance in a recent test, conversion to click was hurt, conversion to a person who clicked 5x was helped.  A/B testing let us notice that something weird was going on and think about what we really cared about.  This automated approach would make a decision and could have hidden a real problem.<p>5. Many tests perform differently on existing users and new users.  A/B testing with proper cohort analysis can let you tease this out and decide accordingly.  This approach doesn't give you that kind of sophistication.