Web Scraping in 2016
Keep in mind that companies have <i>sued</i> for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: <a href="http:&#x2F;&#x2F;www.informationweek.com&#x2F;software&#x2F;social&#x2F;linkedin-sues-after-scraping-of-user-data&#x2F;d&#x2F;d-id&#x2F;1113362" rel="nofollow">http:&#x2F;&#x2F;www.informationweek.com&#x2F;software&#x2F;social&#x2F;linkedin-sues...</a><p>OKCupid did a DMCA takedown for researchers releasing scraped data: <a href="https:&#x2F;&#x2F;www.engadget.com&#x2F;2016&#x2F;05&#x2F;17&#x2F;publicly-released-okcupid-profiles-taken-down-dmca-claim&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.engadget.com&#x2F;2016&#x2F;05&#x2F;17&#x2F;publicly-released-okcupi...</a><p>Since both of these incidents, I now only scrape if it&#x27;s a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.<p>(Disclosure: I have developed a Facebook Page Post Scraper [<a href="https:&#x2F;&#x2F;github.com&#x2F;minimaxir&#x2F;facebook-page-post-scraper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;minimaxir&#x2F;facebook-page-post-scraper</a>] which explicitly follows the permissions set by the Facebook API.)