document: Design of a crawler with bounded bandwidth This paper presents an algorithm to bound the bandwidth of a Web crawler . The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads . The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth . The experimental results show the effectiveness of the proposed technique .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O