I'm crawling web pages to create a search engine and have been able to crawl close to 9300 pages in 1 hour using Scrapy. I'd like to know how much more can I improve and what value is considered as a 'good' crawling speed.
It really depends but you can always check your crawling benchmarks for your hardware by typing scrapy bench on your command line
What is a good crawling speed rate?