Testing 3 million hyperlinks, lessons learned
A variable called stuff? Seriously?<p>Shame as the rest of the article is quite good, but that really flags me that this is a little bit cowboy code.<p>Also interesting to read some sites are taking a 'white-list' approach to robots.txt, as he says this is resulting in people starting to ignore it.