Avoiding Webscraping Throttling Using Python and Tor as a Proxy
While this is intellectually interesting, I'm troubled by the fact that the author seems not to have given the slightest thought that he's breaking the site's T&C, or of how much this abuse costs the service.<p>I'm particularly sensitive to this because I'm constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.