<p dir="auto">I am running a spider with Scrapy but after it finishes crawling it can't seem to terminate. Log stats just recursively report that it is scraping 0 pages/minute. When I try to quit with Ctrl-C, it fails to shut down gracefully and I have to quit forcefully with Ctrl-C again. Any clue what is happening?</p>
<p dir="auto">After completing a scrape, I just get output like this:</p>
<blockquote>
<p dir="auto">2017-08-24 11:13:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:14:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:15:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:16:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:17:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:18:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:19:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:20:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-24 11:21:45 [scrapy.extensions.logstats] INFO: Crawled 60 pages (at 0 pages/min), scraped 54 items (at 0 items/min)</p>
</blockquote>
<p dir="auto">which continues indefinitely.</p>
<p dir="auto">My spider goes to a page that contains a list of links over multiple pages. It visits the first page, extracts the links (using the request meta trick to pass some information along while following the link), and then goes to the next page of links.</p>
<p dir="auto">A second parser extracts information from the individual pages.</p>
<p dir="auto">I don't see any error messages, and the job performs successfully; it just fails to end. This is a problem because I would like to use a script to call the job to run multiple times on different pages (same structure, different information), but the since the first job never finishes I can't ever get to the next set of pages to scrape.</p>
<p dir="auto">The <code class="notranslate">parse(self, response)</code> method yields two types of information.</p>
<ol dir="auto">
<li>
<p dir="auto">For each link on the page, visit the page to extract more information.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" request = scrapy.Request(item['url'], callback=self.parse_transcript)
 request.meta['item'] = item
 yield request"><pre class="notranslate"><code class="notranslate"> request = scrapy.Request(item['url'], callback=self.parse_transcript)
 request.meta['item'] = item
 yield request
</code></pre></div>
</li>
<li>
<p dir="auto">If there is another page of links, get link and increment page number by 1 using regex.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" while data['count'] &gt; 0:
     next_page = re.sub('(?&lt;=page=)(\d+)', lambda x: str(int(x.group(0)) + 1), response.url) 
     yield Request(next_page)"><pre class="notranslate"><code class="notranslate"> while data['count'] &gt; 0:
     next_page = re.sub('(?&lt;=page=)(\d+)', lambda x: str(int(x.group(0)) + 1), response.url) 
     yield Request(next_page)
</code></pre></div>
</li>
</ol>
<p dir="auto">I checked the engine status using the telnet extension. I'm not sure how to interpret this information though.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; est()
Execution engine status

time()-engine.start_time                        : 10746.1215799
engine.has_capacity()                           : False
len(engine.downloader.active)                   : 0
engine.scraper.is_idle()                        : False
engine.spider.name                              : transcripts
engine.spider_is_idle(engine.spider)            : False
engine.slot.closing                             : &lt;Deferred at 0x10d8fda28&gt;
len(engine.slot.inprogress)                     : 4
len(engine.slot.scheduler.dqs or [])            : 0
len(engine.slot.scheduler.mqs)                  : 0
len(engine.scraper.slot.queue)                  : 0
len(engine.scraper.slot.active)                 : 4
engine.scraper.slot.active_size                 : 31569
engine.scraper.slot.itemproc_size               : 0
engine.scraper.slot.needs_backout()             : False"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; est()
Execution engine status

time()-engine.start_time                        : 10746.1215799
engine.has_capacity()                           : False
len(engine.downloader.active)                   : 0
engine.scraper.is_idle()                        : False
engine.spider.name                              : transcripts
engine.spider_is_idle(engine.spider)            : False
engine.slot.closing                             : &lt;Deferred at 0x10d8fda28&gt;
len(engine.slot.inprogress)                     : 4
len(engine.slot.scheduler.dqs or [])            : 0
len(engine.slot.scheduler.mqs)                  : 0
len(engine.scraper.slot.queue)                  : 0
len(engine.scraper.slot.active)                 : 4
engine.scraper.slot.active_size                 : 31569
engine.scraper.slot.itemproc_size               : 0
engine.scraper.slot.needs_backout()             : False
</code></pre></div>
<p dir="auto">I tried raising an exception to close the spider after it reached the end of the links, but this prematurely stopped the spider from being able to visit all of the links that were scrapped. Further, the engine still appeared to hang after closing the spider.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="while data['count'] &gt; 0:
    next_page = re.sub('(?&lt;=page=)(\d+)', lambda x: str(int(x.group(0)) + 1), response.url)
    yield Request(next_page)
else:
    raise CloseSpider('End of transcript history has been reached.')"><pre class="notranslate"><code class="notranslate">while data['count'] &gt; 0:
    next_page = re.sub('(?&lt;=page=)(\d+)', lambda x: str(int(x.group(0)) + 1), response.url)
    yield Request(next_page)
else:
    raise CloseSpider('End of transcript history has been reached.')
</code></pre></div>
<p dir="auto">I also tried using the CLOSESPIDER_TIMEOUT extension, but to no avail. The spider appears to close properly, but the engine remains idling indefinitely.</p>
<blockquote>
<p dir="auto">2017-08-30 11:20:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 9 pages/min), scraped 42 items (at 9 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:23:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:24:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:25:44 [scrapy.core.engine] INFO: Closing spider (closespider_timeout)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:25:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:28:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:29:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:32:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">^C2017-08-30 11:33:31 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force</p>
</blockquote>
<blockquote>
<p dir="auto">2017-08-30 11:41:44 [scrapy.extensions.logstats] INFO: Crawled 48 pages (at 0 pages/min), scraped 42 items (at 0 items/min)</p>
</blockquote>
<blockquote>
<p dir="auto">^C2017-08-30 11:45:52 [scrapy.crawler] INFO: Received SIGINT twice, forcing unclean shutdown</p>
</blockquote>
<p dir="auto">Errors exists about the Twisted when installing scrapy in window by using pip install scrapy. Any one know how to fix it.</p>
<blockquote>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" copying src\twisted\internet\test\_yieldfromtests.py.3only -&gt; build\lib.win32-2.7\twisted\internet\test
    creating build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\chain.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\not-a-certificate -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing1.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2-duplicate.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\mail\test\rfc822.message -&gt; build\lib.win32-2.7\twisted\mail\test
    copying src\twisted\python\test\_deprecatetests.py.3only -&gt; build\lib.win32-2.7\twisted\python\test
    copying src\twisted\words\im\instancemessenger.glade -&gt; build\lib.win32-2.7\twisted\words\im
    copying src\twisted\words\xish\xpathparser.g -&gt; build\lib.win32-2.7\twisted\words\xish
    running build_ext
    building 'twisted.test.raiser' extension
    error: INCLUDE environment variable is empty

    ----------------------------------------
Command &quot;c:\python27\python.exe -u -c &quot;import setuptools, tokenize;__file__='c:\\users\\thomas~1\\appdata\\local\\temp\\pip-build-3oirm6\\Twisted\\setup.py';
f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))&quot;
 install --record c:\users\thomas~1\appdata\local\temp\pip-xvbgd2-record\install-record.txt --single-version-externally-managed --compile&quot; 
failed with error code 1 in c:\users\thomas~1\appdata\local\temp\pip-build-3oirm6\Twisted\"><pre class="notranslate"><code class="notranslate"> copying src\twisted\internet\test\_yieldfromtests.py.3only -&gt; build\lib.win32-2.7\twisted\internet\test
    creating build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\chain.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\not-a-certificate -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing1.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2-duplicate.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\internet\test\fake_CAs\thing2.pem -&gt; build\lib.win32-2.7\twisted\internet\test\fake_CAs
    copying src\twisted\mail\test\rfc822.message -&gt; build\lib.win32-2.7\twisted\mail\test
    copying src\twisted\python\test\_deprecatetests.py.3only -&gt; build\lib.win32-2.7\twisted\python\test
    copying src\twisted\words\im\instancemessenger.glade -&gt; build\lib.win32-2.7\twisted\words\im
    copying src\twisted\words\xish\xpathparser.g -&gt; build\lib.win32-2.7\twisted\words\xish
    running build_ext
    building 'twisted.test.raiser' extension
    error: INCLUDE environment variable is empty

    ----------------------------------------
Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\thomas~1\\appdata\\local\\temp\\pip-build-3oirm6\\Twisted\\setup.py';
f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))"
 install --record c:\users\thomas~1\appdata\local\temp\pip-xvbgd2-record\install-record.txt --single-version-externally-managed --compile" 
failed with error code 1 in c:\users\thomas~1\appdata\local\temp\pip-build-3oirm6\Twisted\
</code></pre></div>
</blockquote>