document: Do not crawl in the DUST : different URLs with similar text We consider the problem of dust : Different URLs with Similar Text . Such duplicate URLs are prevalent in web sites , as web server software often uses aliases and redirections , translates URLs to some canonical form , and dynamically generates the same page from various different URL requests . We present a novel algorithm , DustBuster , for uncovering dust ; that is , for discovering rules for transforming a given URL to others that are likely to have similar content . DustBuster is able to detect dust effectively from previous crawl logs or web server logs , without examining page contents . Verifying these rules via sampling requires fetching few actual web pages . Search engines can benefit from this information to increase the effectiveness of crawling , reduce indexing overhead as well as improve the quality of popularity statistics such as PageRank .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O