Display the number of duplicate requests filtered in the post-crawler stats
You can maintain a list of urls that u have crawled , whenever you came across a url that is already in the list you can log it and increment a counter.
One of the Scrapy spiders (version 0.21) I'm running isn't pulling all the items I'm trying to scrape.
The stats show that 283 items were pulled, but I'm expecting well above 300 here. I suspect that some of the links on the site are duplicates, as the logs show the first duplicate request, but I'd like to know exactly how many duplicates were filtered so I'd have more conclusive proof. Preferably in the form of an additional stat at the end of the crawl.
I know that the latest version of Scrapy already does this, but I'm kinda stuck with 0.21 at the moment and I can't see any way to replicate that functionality with what I've got. There doesn't seem to be a signal emitted when a duplicate url is filtered, and DUPEFILTER_DEBUG doesn't seem to work either. 
Any ideas on how I can get what I need?