How Coderwall grew over 50% with an A/B test
This is a very good basic A&#x2F;B test, and while it is certainly way better than nothing and something that a lot of websites should try, I think there are some important caveats that I&#x27;m thinking of that aren&#x27;t mentioned in the article:<p>1. How were the experiment groups chosen? You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn&#x27;t enough for a test like this.  It is only meaningful if you split up 20,000 pages that had similar traffic profiles and were getting enough aggregate traffic to be able to notice an increase.<p>2. SEO tests take a long time so if you are doing it, I&#x27;d recommend more than 2 variations.  You need to put the test up, wait for Google to index it, then wait a few weeks to see how the traffic changes.  So since your turn-around time is at least a few weeks and maybe longer, try 4 or 8 variations if your traffic can support it.<p>3. I prefer my A&#x2F;B tests to be a little more crazy, or at least try a crazy variation among more normal ones.  In my experience the biggest gainers (and also the biggest losers) are ideas that seem crazy.  Getting your feet wet with adding &quot;(Example)&quot; to the title is fine, but also try a crazy variation that is completely different and seeing how that does.  Give yourself a chance to be surprised by your audience so you can learn more about them.  And if the crazy variation loses by a lot, you have learned something important even if it isn&#x27;t a winner.<p>4. The facts are a little dubious here.  Traffic is up 50% over a timespan... there is no control.  Traffic generally goes up for growing websites even if you don&#x27;t do anything.  You should also show your work on the 14.8% increase and give us some error bars.  You say it is significant, how did you calculate this?  It does seem like the test was better, but it is also important to make valid claims about it.  Intellectual honesty when running A&#x2F;B tests is <i>really really</i> important [1].<p>1: <a href="http:&#x2F;&#x2F;danbirken.com&#x2F;ab&#x2F;testing&#x2F;2014&#x2F;04&#x2F;08&#x2F;pitfalls-of-bias-in-ab-testing.html" rel="nofollow">http:&#x2F;&#x2F;danbirken.com&#x2F;ab&#x2F;testing&#x2F;2014&#x2F;04&#x2F;08&#x2F;pitfalls-of-bias-...</a>