I have a lot of different sites I want to scrape using scrapy. I was wondering what is the best way of doing this?
Do you use a different "project" for each site you want to scrape, or do you use a different "spider", or neither?
Any input would be appreciated
different PROJECT for each site is a WORST idea .
different SPIDER for each site is a GOOD idea .
if you can adjust multiple sites in one SPIDER (based of there nature) is a BEST idea .
but again all depends on your Requirements.
Scrapy Python Crawler - Different Spider for each?