By the time we identified you as the problem the damage was already done. In fact, when combined with potential performance penalty that Google may apply (could be 5%, 10%, for one week, two weeks... who knows) the actual amount of damage in lost traffic/business to us could easily run into the thousands.<p>Are you going to reimburse us the damages? Obviously not.<p>I want you to realize that the timeliness of your response is completely irrelevant to the matter at hand.<p>Proper net etiquette would be to have a reasonable limit of an ACTUAL 1 req/sec, not an average. Furthermore, allow only the webmaster of the site to increase the rate.<p>Also, while nofollow sculpting is not the original intention of the metatag, MANY people use it as such, so respect that directive.<p>Also, you can have your system be aware when a site is slow to respond and adjust crawl rates/times as necessary.<p>Also, user-agent protections are not enough. Many webservers have built-in automatic IP-based attack detection/prevention, but as far as I know, most don't have user-agent based systems. The new virtual land is distributed and webservers have to react to that new reality, but the onus is on you to be aware that many people are still running legacy systems.<p>If you guys did that from the beginning there wouldn't be an issue. And I'm sure all those complaints you see on other webmaster forums would largely go away as well.<p>There are many things you can/could've done to prevent this problem. There's things we could have done as well, but placing the blame on us is akin to blaming a robbed family for not locking the door. Sure, said family should have locked the door, but that in no way excuses the robber of their actions.<p>I know it's a lot easier to ask for forgiveness than permission, but when you're performing the equivalent of a DDoS attack and causing immediate and direct financial damages to a company... that doesn't fly.