Scrapy: RSS control pub_date
I store all data in database as well, and calculate a hash value out of the data. That way you can look up the hash very quickly, and carry out de-dup operation on the fly.
I'm doing a RSS spider. How do you do for controlling the last crawl
date?
Right now what was I thinking is this:

Put in a control file the last pub_date that I have crawled. 
Then when the crawl starts, it checks the last pub_date against the
new pub_dates. If there are new items, then start crawling, if not, do
nothing.

How does everyone else resolve this?