<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/11151000/45806794-84272480-bcf4-11e8-872c-7828c49657b9.png"><img src="https://user-images.githubusercontent.com/11151000/45806794-84272480-bcf4-11e8-872c-7828c49657b9.png" alt="screenshot 114" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/11151000/45806796-84272480-bcf4-11e8-90c8-0fa7a7c98bee.png"><img src="https://user-images.githubusercontent.com/11151000/45806796-84272480-bcf4-11e8-90c8-0fa7a7c98bee.png" alt="screenshot 115" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/11151000/45806798-84bfbb00-bcf4-11e8-9199-8f86e6e364ab.png"><img src="https://user-images.githubusercontent.com/11151000/45806798-84bfbb00-bcf4-11e8-9199-8f86e6e364ab.png" alt="screenshot 116" style="max-width: 100%;"></a></p>
<p dir="auto">I have noticed that spider_idle signal is not called when the last issued request is filtered out by dedupe filter.</p>
<p dir="auto">Example spider:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from scrapy import log, signals
from scrapy.http import Request
from scrapy.spider import BaseSpider


class IdleSpider(BaseSpider):
    name = 'idle_spider'

    def set_crawler(self, crawler):
        super(IdleSpider, self).set_crawler(crawler)
        self.crawler.signals.connect(self._spider_idle, signal=signals.spider_idle)

    def _spider_idle(self, spider):
        self.log('Enqueuing...', log.INFO)
        self.crawler.engine.crawl(Request('http://httpbin.org/'), self)

    def start_requests(self):
        return []

    def parse(self, response):
        self.log('Parsing...', log.INFO)
        pass"><pre class="notranslate"><code class="notranslate">from scrapy import log, signals
from scrapy.http import Request
from scrapy.spider import BaseSpider


class IdleSpider(BaseSpider):
    name = 'idle_spider'

    def set_crawler(self, crawler):
        super(IdleSpider, self).set_crawler(crawler)
        self.crawler.signals.connect(self._spider_idle, signal=signals.spider_idle)

    def _spider_idle(self, spider):
        self.log('Enqueuing...', log.INFO)
        self.crawler.engine.crawl(Request('http://httpbin.org/'), self)

    def start_requests(self):
        return []

    def parse(self, response):
        self.log('Parsing...', log.INFO)
        pass
</code></pre></div>
<p dir="auto">Message <em>"Enqueuing..."</em> is printed out exactly two times and <em>"Parsing..."</em> exactly once. After that <em>spider_idle</em> signal is sent no more. To me it seems like a bug, but if it is an intention, please reconsider it. It is rather unfortunate in the scenario, when _<em>spider_idle</em> is issuing new requests in batches and suddenly the last request was redirected to previously visited url (i.e. it wasn't even directly duplicate request).</p>
<p dir="auto">Of course the simple solution can be to use dont_filter on the issued requests, but it doesn't seem right to me.</p>