The reason the request rate was at 4 was that at some point your site responded to 4 req/sec without problems.<p>User-agent is the accepted and <i>standardized</i> way to communicate with bots.  Getting angry at us for following robots.txt is like getting angry at someone for showing you her ID when she goes through airport security.<p>From our perspective, your house put up a sign that said "Hey everyone come in 4 at a time".