Scrapy extremely slow: probably bottleneck
Amazon, like other big websites, is using IA to detect the incoming requests. It means that if you are requesting a lot of traffic to the same "logical" order on the same website, they will detect you and try to ban you (even if you are using other proxies).
Try to separate and do it on different servers. Instead of using only one big server, use several small servers.
My recommendations:

Use proxies that are reliable (different providers)
Use random time to request every URL (just to confuse the detection algorithm and to avoid being identified as the same user using different IP)
Use different agents too (rotating)
Be nice to the site you are going to scrape. See robots.txt file to know more
If you are scraping a list of elements in Amazon (for instance, the top products sell in books category), don't scrape it sequentially. Pick up randomly to scrape that list of products in order to be unpredictable
Use a good structure to scrape (parallelizing). I suggest use:


Store the URLs to scrape in Redis
Use differents ScrapyD servers to run horizontal scraping
Use monitoring system to review logs, to review errors and manage the servers


I hope it helps!
I need help to find the bottleneck with my scrapy/python based scraper.
We are scraping products from Amazon (Italy at the moment) but we are struggling with overall requests throughput.
We are using backconnect rotating proxies: StormProxies (50 threads plan) + Proxyrotator (100 threads) + TOR but even 250+ available threads we can scrape only 1/2 URLs per second...
We are running it on OVH dedicated server, 8 core x 16GB ram, redis celery and docker as additional tools
I am an IT technician, the software is developed by my indian dev guy, if you need additional info or code just ask!
Thanks in advance