Thursday, October 29, 2009 at 5:50 PMWebmaster Level:
Intermediate Google uses numerous sources to find new webpages,
from links we find on the web to submitted URLs. We aim to
discover new pages quickly so that users can find new content in
Google search results soon after they go live. We recently
launched a feature that uses RSS and Atom feeds for the
discovery of new webpages. RSS/Atom feeds have been very popular
in recent years as a mechanism for content publication. They
allow readers to check for new content from publishers. Using
feeds for discovery allows us to get these new pages into our
index more quickly than traditional crawling methods. We may use
many potential sources to access updates from feeds including
Reader, notification services, or direct crawls of feeds. Going
forward, we might also explore mechanisms such as PubSubHubbub
to identify updated items. In order for us to use your RSS/Atom
feeds for discovery, it's important that crawling these files is
not disallowed by your robots.txt.