No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the &quot;bad&quot; sites as in, really bad shit? The stuff that your ISP could use as evidence against you when in fact it was just your code running and it happened to come across one of these sorts of sites. How do you deal with all that? That is the only thing stopping me from experimenting around with web crawling.