Only Google is really allowed to crawl the web
A somewhat offtopic question: why isn't there a lighter version of HTTPS that only signs messages, but doesn't encrypt them? This way it would be possible to cache the content and use the cache for indexing.