Testing 3 million hyperlinks, lessons learned
A variable called stuff? Seriously?Shame as the rest of the article is quite good, but that really flags me that this is a little bit cowboy code.Also interesting to read some sites are taking a 'white-list' approach to robots.txt, as he says this is resulting in people starting to ignore it.