Suddenly, Hacker News is not the first result for 'Hacker News'
Bottom line is that if you don't trust Google & use their tools (GWT) then you can end up in sticky situations like this one.<p>My experience with the Crawl Rate feature via GWT is that they do honour it pretty strictly, but for large sites Gbot can cause a lot of extra load even if pages are static.<p>A good CDN and stateless cache server will help but for sites as large as HN every request adds up!