<h3 dir="auto">Description</h3>
<p dir="auto">Using <code class="notranslate">spider.crawler.engine.close_spide()</code> in pipeline will throw an exception</p>
<h3 dir="auto">Steps to Reproduce</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class IPVerification:
    @classmethod
    def from_settings(cls, settings):
        return cls(settings.get('REDIS_URL'))

    def __init__(self, redis_url):
        connect_pool = ConnectionPool.from_url(redis_url)
        self.redis = Redis(connection_pool=connect_pool, max_connections=128)
        self.count = 0

    def process_item(self, item, spider: Spider):
        ipv4 = item.get('ipv4')
        port = item.get('port')
        if ipv4 is None or port is None:
            return
        fingerprint = md5(ipv4.encode() + port.encode()).hexdigest()
        if self.redis.sadd('Item', fingerprint):
            return item
        if self.count &gt; 10:
            spider.crawler.engine.close_spider(spider, 'Multiple data repetitions')
        else:
            self.count += 1
        raise DropItem('Data duplication')"><pre class="notranslate"><code class="notranslate">class IPVerification:
    @classmethod
    def from_settings(cls, settings):
        return cls(settings.get('REDIS_URL'))

    def __init__(self, redis_url):
        connect_pool = ConnectionPool.from_url(redis_url)
        self.redis = Redis(connection_pool=connect_pool, max_connections=128)
        self.count = 0

    def process_item(self, item, spider: Spider):
        ipv4 = item.get('ipv4')
        port = item.get('port')
        if ipv4 is None or port is None:
            return
        fingerprint = md5(ipv4.encode() + port.encode()).hexdigest()
        if self.redis.sadd('Item', fingerprint):
            return item
        if self.count &gt; 10:
            spider.crawler.engine.close_spider(spider, 'Multiple data repetitions')
        else:
            self.count += 1
        raise DropItem('Data duplication')
</code></pre></div>
<p dir="auto"><strong>Reproduces how often</strong>: 100%</p>
<h3 dir="auto">Versions</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Scrapy       : 2.6.1
lxml         : 4.6.4.0
libxml2      : 2.9.5
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.2.0
Python       : 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
pyOpenSSL    : 22.0.0 (OpenSSL 1.1.1m  14 Dec 2021)
cryptography : 36.0.1
Platform     : Windows-10-10.0.22000-SP0
"><pre class="notranslate"><code class="notranslate">Scrapy       : 2.6.1
lxml         : 4.6.4.0
libxml2      : 2.9.5
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.2.0
Python       : 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
pyOpenSSL    : 22.0.0 (OpenSSL 1.1.1m  14 Dec 2021)
cryptography : 36.0.1
Platform     : Windows-10-10.0.22000-SP0

</code></pre></div>
<h3 dir="auto">Error</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" 'start_time': datetime.datetime(2022, 4, 6, 1, 52, 16, 182409)}
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Spider closed (Multiple data repetitions)
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Error while scheduling new request
Traceback (most recent call last):
  File &quot;D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\task.py&quot;, line 526, in _oneWorkUnit
    result = next(self._iterator)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\defer.py&quot;, line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File &quot;D:\Program Files\Python\Python310\lib\site-packages\scrapy\core\engine.py&quot;, line 187, in &lt;lambda&gt;
    d.addBoth(lambda _: self.slot.nextcall.schedule())
AttributeError: 'NoneType' object has no attribute 'nextcall'

Process finished with exit code 0"><pre class="notranslate"><code class="notranslate"> 'start_time': datetime.datetime(2022, 4, 6, 1, 52, 16, 182409)}
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Spider closed (Multiple data repetitions)
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Error while scheduling new request
Traceback (most recent call last):
  File "D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\task.py", line 526, in _oneWorkUnit
    result = next(self._iterator)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "D:\Program Files\Python\Python310\lib\site-packages\scrapy\core\engine.py", line 187, in &lt;lambda&gt;
    d.addBoth(lambda _: self.slot.nextcall.schedule())
AttributeError: 'NoneType' object has no attribute 'nextcall'

Process finished with exit code 0
</code></pre></div>
<p dir="auto">Sometimes pages depend on certain HTTP request headers sent, for rendering the expected result, and it's a manual and tedious job to find out which headers those are.</p>
<p dir="auto">So, here's an idea for automating this probing mechanism: create a new scrapy command <code class="notranslate">probe</code> which takes a url as argument and a text to look for.</p>
<p dir="auto">Scrapy then tries several combinations of HTTP headers (user-agent, accept, etc) and return a set that works (where works mean that the text passed is found).</p>
<p dir="auto">Here's a real world example to illustrate:</p>
<p dir="auto"><a href="http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm" rel="nofollow">http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm</a></p>
<p dir="auto">The page should contain a string <code class="notranslate">'var sFeatures'</code>, but that string is not returned with the default Scrapy HTTP request headers. So we run scrapy probe on it:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ scrapy probe http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm 'var sFeatures'
Found set of working headers:
{'Host': 'www.storage-cabinets-online.com', 'User-Agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid)
Firefox/3.6.10', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Accept-Encoding': 'gzip,deflate',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Keep-Alive': '115'}"><pre class="notranslate">$ scrapy probe http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm <span class="pl-s"><span class="pl-pds">'</span>var sFeatures<span class="pl-pds">'</span></span>
Found <span class="pl-c1">set</span> of working headers:
{<span class="pl-s"><span class="pl-pds">'</span>Host<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>www.storage-cabinets-online.com<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>User-Agent<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid)</span>
<span class="pl-s">Firefox/3.6.10<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Accept<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Accept-Language<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>en-us,en;q=0.5<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Accept-Encoding<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>gzip,deflate<span class="pl-pds">'</span></span>,
<span class="pl-s"><span class="pl-pds">'</span>Accept-Charset<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>ISO-8859-1,utf-8;q=0.7,*;q=0.7<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Keep-Alive<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>115<span class="pl-pds">'</span></span>}</pre></div>
<p dir="auto">The Scrapy probe command would try a different list of well known user-agents, along with Accept headers.</p>