Basically I have a regex rule for following pages
Each page has 50 links
When i hit a link that is too old (based on a pre-defined date-time)
I want to tell scrapy to stop following more pages, but NOT stop it entirely, it must continue to scrape the links it has already decided to scrape -> (complete all Request objects created). JUST that it must NOT follow any more links. So the program will eventually grind to a stop (when it's done scraping all the links)
Is there any way i can do this inside the spider?
Scrapy's CrawlSpider has an internal _follow_links member variable which is not yet documented (experimental as for now)
setting self._follow_links = False will tell scrapy to stop following more links. But continue to finish up all the Request objects it has already created
How to tell scrapy crawler to STOP following more links dynamically?