Scrapy: website is slowing down my requests
If the website is slowing down your requests, it's probably best to be more careful with the requests you're sending.
A couple of things you can try:

Set 'AUTOTHROTTLE_ENABLED' to True: when scrapy sees the server is responding slowly, it will adjust the crawling speed
Limit the number of concurrent requests yourself (by setting CONCURRENT_REQUESTS_PER_DOMAIN), and change the DOWNLOAD_DELAY to wait a bit between requests
Use proxies: most likely the target website is limiting you based on the IP. Using proxies will overcome this, but this will not help in case the target server is overloaded by your requests
I have written a spider in scrapy. Everytime i try to scrape a specific website, the spider is running quite well at the start. but when i goes down to like 500-600 request, the spider start to crawl very slowly.
I have checked memory and cpu settings but that is not the problem. Im quite sure that the website is slowing down my requests.
How do i adjust scrapy to scrape fast and consistently?