document: Effective web-scale crawling through website analysis The web crawler space is often delimited into two general areas : full-web crawling and focused crawling . We present netSifter , a crawler system which integrates features from these two areas to provide an effective mechanism for web-scale crawling . netSifter utilizes a combination of page-level analytics and heuristics which are applied to a sample of web pages from a given website . These algorithms score individual web pages to determine the general utility of the overall website . In doing so , netSifter can formulate an in-depth opinion of a website ( and the entirety of its web pages ) with a relative minimum of work . netSifter is then able to bias the future efforts of its crawl towards higher quality websites , and away from the myriad of low quality websites and crawler traps that litter the World Wide Web .
doc_bio_tags: O O B O O O O O O O O O O O O O O O O B O O B O O O B O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O