Only Google is really allowed to crawl the web
The bigger problem, to me, is not around crawling. It&#x27;s the asymmetrical power Google has after crawling.<p>Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from.<p>Airlines are a good example.  If you search for flight status for a particular flight, Google presents that flight status in a box.  As an end user, that&#x27;s great.  However, that sort of search used to (most times) lead to a visit to the airline web site.<p>The airline web site could then present things Google can&#x27;t do.  Like &quot;hey, we see you haven&#x27;t checked in yet&quot; or &quot;TSA wait times are longer than usual&quot; or &quot;We have a more-legroom seat upgrade if you want it&quot;.<p>Google took those eyeballs away.  Okay, fine, that&#x27;s their choice.  But they don&#x27;t give anything back, which removes incentives from the actual source to do things better.<p>You see this recently with Wikipedia.  Google&#x27;s widgets have been reducing traffic to Wikipedia pretty dramatically.  Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.<p>In short, I don&#x27;t think the crawler is the problem. And I don&#x27;t think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.