Search engine innovator Brian Pinkerton's WebCrawler was the
Web's first full-text retrieval search engine. Prior to this
development, search engine crawlers simply scanned the first
couple of hundred words (often less) from the top of your page
and moved on. (WebCrawler is now a metasearch engine.) At the
time, WebCrawler was starved of fresh content. Pinkerton was
programming his crawler to pillage Usenet (now GoogleGroups) at
a whopping 50 gigs a day. The purpose, of course, was to suck
out all the links in the Usenet feed for future crawling. But
little or nothing was dynamically delivered then, so crawling
for new content while maintaining the freshness of existing
index content was pretty straightforward. Today, that task is
much trickier. The Web continues to grow exponentially every
day, and, although there's tons of new content in good old flat
HTML, there's ten times more tucked away in millions and
millions of online databases that dynamically generate pages.
And accessing, indexing, and maintainin