How to tell scrapy crawler to STOP following more links dynamically?
Scrapy's CrawlSpider has an internal _follow_links member variable which is not yet documented (experimental as for now)
setting self._follow_links = False will tell scrapy to stop following more links. But continue to finish up all the Request objects it has already created
Basically I have a regex rule for following pages
Each page has 50 links
When i hit a link that is too old (based on a pre-defined date-time)
I want to tell scrapy to stop following more pages, but NOT stop it entirely, it must continue to scrape the links it has already decided to scrape -> (complete all Request objects created). JUST that it must NOT follow any more links. So the program will eventually grind to a stop (when it's done scraping all the links)
Is there any way i can do this inside the spider?