document: Measuring the web crawler ethics Web crawlers are highly automated and seldom regulated manually . The diversity of crawler activities often leads to ethical problems such as spam and service attacks . In this research , quantitative models are proposed to measure the web crawler ethics based on their behaviors on web servers . We investigate and define rules to measure crawler ethics , referring to the extent to which web crawlers respect the regulations set forth in robots . txt configuration files . We propose a vector space model to represent crawler behavior and measure the ethics of web crawlers based on the behavior vectors . The results show that ethicality scores vary significantly among crawlers . Most commercial web crawlers ' behaviors are ethical . However , many commercial crawlers still consistently violate or misinterpret certain robots . txt rules . We also measure the ethics of big search engine crawlers in terms of return on investment . The results show that Google has a higher score than other search engines for a US website but has a lower score than Baidu for Chinese websites .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O