Web Scraping in 2016
Keep in mind that companies have <i>sued</i> for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: <a href="http://www.informationweek.com/software/social/linkedin-sues-after-scraping-of-user-data/d/d-id/1113362" rel="nofollow">http://www.informationweek.com/software/social/linkedin-sues...</a><p>OKCupid did a DMCA takedown for researchers releasing scraped data: <a href="https://www.engadget.com/2016/05/17/publicly-released-okcupid-profiles-taken-down-dmca-claim/" rel="nofollow">https://www.engadget.com/2016/05/17/publicly-released-okcupi...</a><p>Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.<p>(Disclosure: I have developed a Facebook Page Post Scraper [<a href="https://github.com/minimaxir/facebook-page-post-scraper" rel="nofollow">https://github.com/minimaxir/facebook-page-post-scraper</a>] which explicitly follows the permissions set by the Facebook API.)