The joys and sorrows of maintaining a personal website
> The enemy possesses every advantage. Personal website owners are nearly always just ordinary individuals. They often have glacially slow upload bandwidths, [...]. The enemy has seemingly unlimited funds, massive resources, acres of server farms,<p>One example of this is Googlebot "attacks". I have a personal, mostly static website that I've been serving from a home connection for over 25 years, upgrading the hardware and moving it with me as I relocated. Out of these 25 years, Google has been number #1 user of bandwidth, CPU time, and as a consequence the #1 generator of headaches.<p>During the mid 2000s I had numerous problems keeping the site online since the Google crawler would just effectively DDoS it, sending tens of concurrents requests completely ignoring the Crawl-Delay in robots.txt (and most of robots.txt, fwiw). Mind you, this was not the most powerful hardware you could find, since it had to be quite low power to be left 24h on, but it was perfectly able to serve the handful of connections per hour I expect out of my website.<p>Obviously your only alternative is to just ban the entire Google IP range and as a consequence disappear from the Internet. Nowadays the problem is less noticeable because my 1-2W server can handle the onslaught brought by Googlebot just fine, but they still ignore the Crawl-Delay.