Don't get me started on 80legs. They took down our servers earlier this week. We couldn't even identify them based on IP address because of their regular-computer-using distributed system. If anything, 80Legs is the ultimate DDoS machine.<p>I mean, they state that their crawl average is 1 req/sec and can be modified if you contact them. The problem is that other people can modify the crawl rate (somebody set ours to 4/seq).<p>And remember, that's just the average. The distributed nature of their system means that it's ridiculously spikey. Furthermore they don't respect nofollow tags that we put in place to keep bots out of bad areas, so their crawlers got into very unoptimized areas of the site.<p>Overall though we turned it into a positive by ramping up our analysis tools, and creating a much stronger robots.txt. Still... I really don't like 80legs and bristle when people reference them as someone doing 'good'.