title: How to remove expired items from database with Scrapy
body: <p>I am using spidering a video site that expires content frequently. I am considering using  <a href="http://scrapy.org/" rel="nofollow noreferrer">scrapy</a> to do my spidering, but am not sure how to delete expired items.</p>
  
  <p>Strategies to detect if an item is expired are:</p>
  
  <ol>
  <li>Spider the site's "delete.rss".</li>
  <li>Every few days, try reloading the contents page and making sure it still works.</li>
  <li>Spider every page of the site's content indexes, and remove the video if it's not found.</li>
  </ol>
  
  <p>Please let me know how to remove expired items in scrapy. I will be storing my scrapy items in a mysql DB via django. </p>
  
  <h3>2010-01-18 Update</h3>
  
  <p>I have found a solution that is working, but still may not be optimal. I am maintaining a "found_in_last_scan" flag on every video that I sync. When the spider starts, it sets all the flags to False. When it finishes, it deletes videos who still have the flag set to False. I did this by attaching to the <code>signals.spider_opened</code> and <code>signals.spider_closed</code>  Please confirm this is a valid strategy and there are no problems with it.</p>