Does threading violate robots.txt?
I'm new to scraping and I recently realized that threading is probably the way to go to crawl a site quickly. Before I begin hacking that out though, I figured it would probably be intelligent to determine whether or not that will end up getting me throttled. So question is, if I rewrite my program to employ threads to crawl more quickly, will that violate most sites' robots.txt?
Does threading violate robots.txt?
===

I'm new to scraping and I recently realized that threading is probably the way to go to crawl a site quickly. Before I begin hacking that out though, I figured it would probably be intelligent to determine whether or not that will end up getting me throttled. So question is, if I rewrite my program to employ threads to crawl more quickly, will that violate most sites' robots.txt?