Scrapy - how to identify already scraped urls
I think jama22's answer is a little incomplete. 
In the snippet if self.FILTER_VISITED in x.meta:, you can see that you require FILTER_VISITED in your Request instance in order for that request to be ignored. This is to ensure that you can differentiate between links that you want to traverse and move around and item links that well, you don't want to see again.
Im using scrapy to crawl a news website on a daily basis. How do i  restrict scrapy from scraping already scraped URLs. Also is there any clear documentation or examples on SgmlLinkExtractor.