document: A larger scale study of robots . txt A website can regulate search engine crawler access to its content using the robots exclusion protocol , specified in its robots . txt file . The rules in the protocol enable the site to allow or disallow part or all of its content to certain crawlers , resulting in a favorable or unfavorable bias towards some of them . A 2007 survey on the robots . txt usage of about 7,593 sites found some evidence of such biases , the news of which led to widespread discussions on the web . In this paper , we report on our survey of about 6 million sites . Our survey tries to correct the shortcomings of the previous survey and shows the lack of any significant preferences towards any particular search engine .
doc_bio_tags: O O O O O O O O O O O O B I B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O