So.. we usually get comments like this on webmaster forums, and I usually just let them go, but since HN is "my" forum, I feel like saying something.<p>1. We identify ourselves as user-agent 008.  The proper way to track a bot is through user-agent, not IP address.<p>2. We never say that the average rate for all sites is 1 req/sec.  Many sites have higher rates.  We do a best guess on what load a site can handle based on a mix of Alexa/Quantcast/Compete stats.<p>3. We always respond to requests from webmasters to reduce the crawl rate.  Within 5 minutes of getting your message, we did this.  What other bot does this?  Oh yeah, <i>none</i> of them.<p>4. The nofollow tag is not a standard way to tell bots to not go to a link.  That is what robots.txt is for.  Nofollow was implemented for Google specifically and has caught on as a (not as common as you think) tag.  Robots.txt is the <i>right</i> way to talk to bots.<p>5. You know what happens when people don't use 80legs?  They figure out some way to get the information they want and soak up your bandwidth anyway.  A webmaster is fooling himself if he thinks he can stop people from crawling a site.  It's going to happen.  At least with 80legs the webmaster has a responsible company with real people to talk to so that the crawl can be <i>managed</i>.