I need help defining an architecture for a tool that will scrape more than 1000 big websites daily for new updates.
I'm planning to use Scrapy for this project:

Giving that Scrapy needs a project for each website, how can I handle scraping 1000+ websites and storing it's data with Scrapy in just one project? I tried adding a project generator, but is this a good idea?
How can I tell if a website was updated with new content so I can scrape it again?

Thanks!
I will be interested to see what other answers come up for this. I have done some web crawling / scrapping with code that I have written myself using urllib to get the html then just searching the html for what I need, but not tried scrapy yet. 
I guess to see if there are differences you may just need to compare the previous and new html pages, but you would need to either work out what changes to ignore e.g. dates etc, or what specific changes you are looking for, unless there is an easier way to do this using scrapy. 
On the storage front you could either store the html data just in a file system or look into just writting it to a database as strings. Just a local database like SQLite should work fine for this, but there are many other options.
Finally, I would also advise you to check out the terms on the sites you are planning to scrape and also check for guidance in the robots.txt if included within the html as some sites give guidance on how frequently they are happy for web crawlers to use them etc.
Crawl and monitor +1000 websites