Avoiding Webscraping Throttling Using Python and Tor as a Proxy
While this is intellectually interesting, I'm troubled by the fact that the author seems not to have given the slightest thought that he's breaking the site's T&C, or of how much this abuse costs the service.I'm particularly sensitive to this because I'm constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.