Question: Hello my server is getting hit by spiders from bingbot, google, yahoo, yandex, and some place in the UK all at the same time crashing it.

While I have no reason to think any one person did this, it strikes my odd that it happened all at the same time. 

Should I be worried? 

The only changes before this started was that I changed all the server passwords.

Answer: 
Bing, Google and Yahoo usually do not flood the the server with requests, but send far less than one request per second.

Please double check that you really get many requests from them within a very short time frame (e. g. within a second).

If your web application does have issues with handling occasional requests (for example on request per second), you can slow down Bing and Yahoo with the following entry in robots.txt:

```
Crawl-delay: 120

```

This will ask crawlers to wait at least 120 seconds between requests. For Google, you can define the delay in the [webmaster tools](http://www.google.com/webmasters/tools/).
I disagree that this is always normal behavior. I am under a similar DDOS attack form the search engines. We have ample resources and numerous redundant servers handling our site. All with DNS failover. And BOOM we got hit and knocked off the web. First I saw reports of failovers going up and down like crazy. 

Thought we had a bad AJAX or un-cached query page being hit. As I looked deeper into the logs and saw every bot, pretend bot and wanta be bot from both recognized and unknows, I knew it was an initiated attack. Why? Our robot.txt only allows googlebot, all else are excluded from our server fleet. 

So the robot.txt was being ignored, and we are being brought down. So if you ignore my robot.txt, my firewall will not ignore you. We firewalled (and still are) over 60 spiders so far. (all hitting at the same time, ignoring both the 360 delay and the disallow in our .txt). So I would not always think this is normal anymore.