Web Scraping: Bypassing “403 Forbidden,” captchas, and more
I'm curious what others use to scrape modern (javascript based) web applications.<p>The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work.<p>For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved from the server.<p>This worked fine but required some effort to implement. Is there an open source solution which is as helpful as Scrapy but solves the issues provided by modern javascript websites/applications?<p>With tools like Chrome headless this should now be possible, right?