I&#x27;ve had a lot of success scraping websites with Capybara [1]. It&#x27;s intended for writing acceptance tests of web apps, but it works remarkably well for scraping websites. It&#x27;s written in Ruby, but the DSL it provides for interacting with web pages should be pretty understandable to anybody who&#x27;s programmed before. It also supports multiple browsers, which means you can tradeoff along these axes:<p>- Headless vs. Not
- JS support vs. Not<p>I put a repo together with a sample script [2] for scraping leads off of a website which I will not name, but whose name rhymes with &#x27;help&#x27;. It uses the PhantomJS browser for headless JS support. It also includes a Vagrantfile so you can avoid installing all the dependencies on your local machine.<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;jnicklas&#x2F;capybara" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jnicklas&#x2F;capybara</a><p>[2]: <a href="https:&#x2F;&#x2F;github.com&#x2F;toasterlovin&#x2F;scraping-yalp" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;toasterlovin&#x2F;scraping-yalp</a>