Only Google is really allowed to crawl the web
The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling.<p>Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from.<p>Airlines are a good example.  If you search for flight status for a particular flight, Google presents that flight status in a box.  As an end user, that's great.  However, that sort of search used to (most times) lead to a visit to the airline web site.<p>The airline web site could then present things Google can't do.  Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it".<p>Google took those eyeballs away.  Okay, fine, that's their choice.  But they don't give anything back, which removes incentives from the actual source to do things better.<p>You see this recently with Wikipedia.  Google's widgets have been reducing traffic to Wikipedia pretty dramatically.  Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.<p>In short, I don't think the crawler is the problem. And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.