Running Scrapy on Dokku using a Digital Ocean server
I 'fixed' this issue by not using a Digital Ocean server. The website that I am trying to crawl, which is craigslist.org, just did not respond well to a DO server. It takes a long time to respond to a request. Other websites like Google or Amazon work just fine with DO.
My scraper works just fine on craigslist when using a VPS from another provider.
Not sure how to describe this but I am running a Scrapy spider on a Digital Ocean server ($5 server), the Scrapy project is deployed as a Dokku app.
However, it runs very slowly compared to the speed on my local computer and on a Heroku free tier dyno. On Dokku it crawls at a speed of 30 pages per minute while locally and on Heroku the speed is 200+ pages per minutes.
I do not know how to debug, analyze or where to start in order to fix the problem. Any help, clues or tips on how to solve this?