<p dir="auto">Hi,</p>
<p dir="auto">I'm trying to perform a broad crawl of the web with Scrapy in breadth-first order.</p>
<p dir="auto">The issue I'm running into is that after a few seconds of the crawl running, it seems to get stuck on just one or two domains instead of continuing down the list of seed URLs. I would expect it to either continue through the entire list of separate domains, and/or crawl multiple domains concurrently.</p>
<p dir="auto">Output of <code class="notranslate">scrapy version -v</code>:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Scrapy    : 1.3.2
lxml      : 3.4.0.0
libxml2   : 2.9.1
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 17.1.0
Python    : 2.7.9 (default, Jun 29 2016, 13:08:31) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-3.16.0-4-amd64-x86_64-with-debian-8.7"><pre class="notranslate">Scrapy    <span class="pl-c1">:</span> 1.3.2
lxml      <span class="pl-c1">:</span> 3.4.0.0
libxml2   <span class="pl-c1">:</span> 2.9.1
cssselect <span class="pl-c1">:</span> 1.0.1
parsel    <span class="pl-c1">:</span> 1.1.0
w3lib     <span class="pl-c1">:</span> 1.17.0
Twisted   <span class="pl-c1">:</span> 17.1.0
Python    <span class="pl-c1">:</span> 2.7.9 (default, Jun 29 2016, 13:08:31) - [GCC 4.9.2]
pyOpenSSL <span class="pl-c1">:</span> 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  <span class="pl-c1">:</span> Linux-3.16.0-4-amd64-x86_64-with-debian-8.7</pre></div>
<p dir="auto">Here is the main code for my spider:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# MySpider defines our Scrapy spider.
class MySpider(CrawlSpider):
    name = &quot;MyBot&quot;
    rules = (
        Rule(LinkExtractor(deny_domains=deny_domains), callback='parse_item', follow=True),
    )

    def start_requests(self):
        # Open the seed.csv file.
        seed_path = os.path.dirname(os.path.realpath(__file__)) + '/seed.csv'
        f = open(seed_path)
        csv_rdr = csv.reader(f)

        for row in csv_rdr:
            url = 'http://' + row[0].strip()
            yield Request(url=url, callback=self.parse)

    def parse_start_url(self, response):
        return self.parse_item(response)

    def parse_item(self, response):
        print '!!!!!!!!!!!!! Parsing: %s !!!!!!!!!!!!!' % response.url

        # Check the Content-Type.
        if is_content_type_ok(response.headers.getlist('Content-Type')):
            # Yield data here
            yield {}"><pre class="notranslate"><span class="pl-c"># MySpider defines our Scrapy spider.</span>
<span class="pl-k">class</span> <span class="pl-v">MySpider</span>(<span class="pl-v">CrawlSpider</span>):
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s">"MyBot"</span>
    <span class="pl-s1">rules</span> <span class="pl-c1">=</span> (
        <span class="pl-v">Rule</span>(<span class="pl-v">LinkExtractor</span>(<span class="pl-s1">deny_domains</span><span class="pl-c1">=</span><span class="pl-s1">deny_domains</span>), <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s">'parse_item'</span>, <span class="pl-s1">follow</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
    )

    <span class="pl-k">def</span> <span class="pl-en">start_requests</span>(<span class="pl-s1">self</span>):
        <span class="pl-c"># Open the seed.csv file.</span>
        <span class="pl-s1">seed_path</span> <span class="pl-c1">=</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">dirname</span>(<span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">realpath</span>(<span class="pl-s1">__file__</span>)) <span class="pl-c1">+</span> <span class="pl-s">'/seed.csv'</span>
        <span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-en">open</span>(<span class="pl-s1">seed_path</span>)
        <span class="pl-s1">csv_rdr</span> <span class="pl-c1">=</span> <span class="pl-s1">csv</span>.<span class="pl-en">reader</span>(<span class="pl-s1">f</span>)

        <span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-s1">csv_rdr</span>:
            <span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-s">'http://'</span> <span class="pl-c1">+</span> <span class="pl-s1">row</span>[<span class="pl-c1">0</span>].<span class="pl-en">strip</span>()
            <span class="pl-k">yield</span> <span class="pl-v">Request</span>(<span class="pl-s1">url</span><span class="pl-c1">=</span><span class="pl-s1">url</span>, <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">parse</span>)

    <span class="pl-k">def</span> <span class="pl-en">parse_start_url</span>(<span class="pl-s1">self</span>, <span class="pl-s1">response</span>):
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">parse_item</span>(<span class="pl-s1">response</span>)

    <span class="pl-k">def</span> <span class="pl-en">parse_item</span>(<span class="pl-s1">self</span>, <span class="pl-s1">response</span>):
        <span class="pl-k">print</span> <span class="pl-s">'!!!!!!!!!!!!! Parsing: %s !!!!!!!!!!!!!'</span> <span class="pl-c1">%</span> <span class="pl-s1">response</span>.<span class="pl-s1">url</span>

        <span class="pl-c"># Check the Content-Type.</span>
        <span class="pl-k">if</span> <span class="pl-en">is_content_type_ok</span>(<span class="pl-s1">response</span>.<span class="pl-s1">headers</span>.<span class="pl-en">getlist</span>(<span class="pl-s">'Content-Type'</span>)):
            <span class="pl-c"># Yield data here</span>
            <span class="pl-k">yield</span> {}</pre></div>
<p dir="auto">Here is my <code class="notranslate">settings.py</code> file:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="BOT_NAME = 'MyBot'

SPIDER_MODULES = ['crawler.spiders']
NEWSPIDER_MODULE = 'crawler.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'BotTest (+https://test.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Broad crawl settings.
CONCURRENT_REQUESTS = 128
REACTOR_THREADPOOL_MAXSIZE = 30
CONCURRENT_REQUESTS_PER_DOMAIN = 1
CONCURRENT_REQUESTS_PER_IP = 1
COOKIES_ENABLED = False
RETRY_ENABLED = False
DOWNLOAD_TIMEOUT = 3
LOG_LEVEL = 'INFO'
AJAXCRAWL_ENABLED = True
HTTPCACHE_ENABLED = False
LOGSTATS_INTERVAL = 10

# Disable auto throttling.
AUTOTHROTTLE_ENABLED = False

# Set delay.
RANDOMIZE_DOWNLOAD_DELAY = False
DOWNLOAD_DELAY = 2

# Set max download size to 2MB
DOWNLOAD_MAXSIZE = 2 * 1024 * 1024
DOWNLOAD_WARNSIZE = 0

# Set scheduler to BFO.
DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'"><pre class="notranslate"><span class="pl-v">BOT_NAME</span> <span class="pl-c1">=</span> <span class="pl-s">'MyBot'</span>

<span class="pl-v">SPIDER_MODULES</span> <span class="pl-c1">=</span> [<span class="pl-s">'crawler.spiders'</span>]
<span class="pl-v">NEWSPIDER_MODULE</span> <span class="pl-c1">=</span> <span class="pl-s">'crawler.spiders'</span>

<span class="pl-c"># Crawl responsibly by identifying yourself (and your website) on the user-agent</span>
<span class="pl-v">USER_AGENT</span> <span class="pl-c1">=</span> <span class="pl-s">'BotTest (+https://test.com)'</span>

<span class="pl-c"># Obey robots.txt rules</span>
<span class="pl-v">ROBOTSTXT_OBEY</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>

<span class="pl-c"># Broad crawl settings.</span>
<span class="pl-v">CONCURRENT_REQUESTS</span> <span class="pl-c1">=</span> <span class="pl-c1">128</span>
<span class="pl-v">REACTOR_THREADPOOL_MAXSIZE</span> <span class="pl-c1">=</span> <span class="pl-c1">30</span>
<span class="pl-v">CONCURRENT_REQUESTS_PER_DOMAIN</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>
<span class="pl-v">CONCURRENT_REQUESTS_PER_IP</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>
<span class="pl-v">COOKIES_ENABLED</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-v">RETRY_ENABLED</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-v">DOWNLOAD_TIMEOUT</span> <span class="pl-c1">=</span> <span class="pl-c1">3</span>
<span class="pl-v">LOG_LEVEL</span> <span class="pl-c1">=</span> <span class="pl-s">'INFO'</span>
<span class="pl-v">AJAXCRAWL_ENABLED</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
<span class="pl-v">HTTPCACHE_ENABLED</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-v">LOGSTATS_INTERVAL</span> <span class="pl-c1">=</span> <span class="pl-c1">10</span>

<span class="pl-c"># Disable auto throttling.</span>
<span class="pl-v">AUTOTHROTTLE_ENABLED</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>

<span class="pl-c"># Set delay.</span>
<span class="pl-v">RANDOMIZE_DOWNLOAD_DELAY</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-v">DOWNLOAD_DELAY</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span>

<span class="pl-c"># Set max download size to 2MB</span>
<span class="pl-v">DOWNLOAD_MAXSIZE</span> <span class="pl-c1">=</span> <span class="pl-c1">2</span> <span class="pl-c1">*</span> <span class="pl-c1">1024</span> <span class="pl-c1">*</span> <span class="pl-c1">1024</span>
<span class="pl-v">DOWNLOAD_WARNSIZE</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>

<span class="pl-c"># Set scheduler to BFO.</span>
<span class="pl-v">DEPTH_PRIORITY</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>
<span class="pl-v">SCHEDULER_DISK_QUEUE</span> <span class="pl-c1">=</span> <span class="pl-s">'scrapy.squeues.PickleFifoDiskQueue'</span>
<span class="pl-v">SCHEDULER_MEMORY_QUEUE</span> <span class="pl-c1">=</span> <span class="pl-s">'scrapy.squeues.FifoMemoryQueue'</span></pre></div>
<p dir="auto">And here is example output from a crawl:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$:~/crawler$ scrapy crawl MyBot -s JOBDIR=crawls/crawl-1 -o output.csv -t csv
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: MyBot)
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Overridden settings: {'FEED_URI': 'output.csv', 'AJAXCRAWL_ENABLED': True, 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'csv', 'DOWNLOAD_DELAY': 2, 'LOG_LEVEL': 'INFO', 'RETRY_ENABLED': False, 'CONCURRENT_REQUESTS_PER_IP': 1, 'DOWNLOAD_TIMEOUT': 3, 'LOGSTATS_INTERVAL': 10, 'DEPTH_PRIORITY': 1, 'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue', 'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue', 'CONCURRENT_REQUESTS': 128, 'RANDOMIZE_DOWNLOAD_DELAY': False, 'DOWNLOAD_WARNSIZE': 0, 'SPIDER_MODULES': ['crawler.spiders'], 'BOT_NAME': 'Bot', 'NEWSPIDER_MODULE': 'crawler.spiders', 'ROBOTSTXT_OBEY': True, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1, 'REACTOR_THREADPOOL_MAXSIZE': 30, 'DOWNLOAD_MAXSIZE': 2097152}
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.spiderstate.SpiderState']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-03-04 22:56:19 [scrapy.core.engine] INFO: Spider opened
2017-03-04 22:56:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://api.imgur.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.sina.com.cn/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://twitter.com/imgur !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.msn.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.ask.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.apple.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.bbc.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.adobe.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://wordpress.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.tumblr.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.reddit.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.yahoo.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.imdb.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.aliexpress.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.youtube.com/ !!!!!!!!!!!!!

... seconds later ...

!!!!!!!!!!!!! Parsing: https://www.paypal.com/us/home !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://stackoverflow.blog/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.amazon.co.jp/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/privacy !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/tags !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://login.live.com/login.srf?wa=wsignin1.0&amp;rpsnv=13&amp;ct=1488668190&amp;rver=6.4.6456.0&amp;wp=MBI_SSL_SHARED&amp;wreply=https:%2F%2Fmail.live.com%2Fdefault.aspx%3Frru%3Dinbox&amp;lc=1033&amp;id=64855&amp;mkt=en-US&amp;cbcxt=mai !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/upload !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/vidgif !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackexchange.com/sites !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/tour !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/memegen !!!!!!!!!!!!!
2017-03-04 22:56:39 [scrapy.extensions.logstats] INFO: Crawled 165 pages (at 90 pages/min), scraped 19 items (at 36 items/min)
!!!!!!!!!!!!! Parsing: http://meta.stackoverflow.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://imgur.com/signin?invokedBy=regularSignIn !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://imgur.com/register?invokedBy=regularSignIn !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/help !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/company/about !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/hot/viral !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackexchange.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/new/viral !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=interesting !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Funny !!!!!!!!!!!!!
2017-03-04 22:56:49 [scrapy.extensions.logstats] INFO: Crawled 175 pages (at 60 pages/min), scraped 25 items (at 36 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com/t/The_More_You_Know !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=featured !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Science_and_Tech !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=hot !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Gaming !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=week !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=month !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Eat_What_You_Want !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/42602342/java-unlimited-cryptography-extension-doesnt-work !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Aww !!!!!!!!!!!!!
2017-03-04 22:56:59 [scrapy.extensions.logstats] INFO: Crawled 185 pages (at 60 pages/min), scraped 42 items (at 102 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Inspiring !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/authentication !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/hdfs !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Awesome !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/kerberos !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Creativity !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/jce !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/The_Great_Outdoors !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Storytime !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/users/3593261/user3593261 !!!!!!!!!!!!!
2017-03-04 22:57:09 [scrapy.extensions.logstats] INFO: Crawled 195 pages (at 60 pages/min), scraped 48 items (at 36 items/min)"><pre class="notranslate">$:<span class="pl-k">~</span>/crawler$ scrapy crawl MyBot -s JOBDIR=crawls/crawl-1 -o output.csv -t csv
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: MyBot)
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Overridden settings: {<span class="pl-s"><span class="pl-pds">'</span>FEED_URI<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>output.csv<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>AJAXCRAWL_ENABLED<span class="pl-pds">'</span></span>: True, <span class="pl-s"><span class="pl-pds">'</span>COOKIES_ENABLED<span class="pl-pds">'</span></span>: False, <span class="pl-s"><span class="pl-pds">'</span>FEED_FORMAT<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>csv<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>DOWNLOAD_DELAY<span class="pl-pds">'</span></span>: 2, <span class="pl-s"><span class="pl-pds">'</span>LOG_LEVEL<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>INFO<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>RETRY_ENABLED<span class="pl-pds">'</span></span>: False, <span class="pl-s"><span class="pl-pds">'</span>CONCURRENT_REQUESTS_PER_IP<span class="pl-pds">'</span></span>: 1, <span class="pl-s"><span class="pl-pds">'</span>DOWNLOAD_TIMEOUT<span class="pl-pds">'</span></span>: 3, <span class="pl-s"><span class="pl-pds">'</span>LOGSTATS_INTERVAL<span class="pl-pds">'</span></span>: 10, <span class="pl-s"><span class="pl-pds">'</span>DEPTH_PRIORITY<span class="pl-pds">'</span></span>: 1, <span class="pl-s"><span class="pl-pds">'</span>SCHEDULER_MEMORY_QUEUE<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>scrapy.squeues.FifoMemoryQueue<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>SCHEDULER_DISK_QUEUE<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>scrapy.squeues.PickleFifoDiskQueue<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>CONCURRENT_REQUESTS<span class="pl-pds">'</span></span>: 128, <span class="pl-s"><span class="pl-pds">'</span>RANDOMIZE_DOWNLOAD_DELAY<span class="pl-pds">'</span></span>: False, <span class="pl-s"><span class="pl-pds">'</span>DOWNLOAD_WARNSIZE<span class="pl-pds">'</span></span>: 0, <span class="pl-s"><span class="pl-pds">'</span>SPIDER_MODULES<span class="pl-pds">'</span></span>: [<span class="pl-s"><span class="pl-pds">'</span>crawler.spiders<span class="pl-pds">'</span></span>], <span class="pl-s"><span class="pl-pds">'</span>BOT_NAME<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>Bot<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>NEWSPIDER_MODULE<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>crawler.spiders<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>ROBOTSTXT_OBEY<span class="pl-pds">'</span></span>: True, <span class="pl-s"><span class="pl-pds">'</span>CONCURRENT_REQUESTS_PER_DOMAIN<span class="pl-pds">'</span></span>: 1, <span class="pl-s"><span class="pl-pds">'</span>REACTOR_THREADPOOL_MAXSIZE<span class="pl-pds">'</span></span>: 30, <span class="pl-s"><span class="pl-pds">'</span>DOWNLOAD_MAXSIZE<span class="pl-pds">'</span></span>: 2097152}
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled extensions:
[<span class="pl-s"><span class="pl-pds">'</span>scrapy.extensions.feedexport.FeedExporter<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.extensions.logstats.LogStats<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.extensions.telnet.TelnetConsole<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.extensions.corestats.CoreStats<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.extensions.spiderstate.SpiderState<span class="pl-pds">'</span></span>]
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
[<span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.useragent.UserAgentMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.redirect.RedirectMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.downloadermiddlewares.stats.DownloaderStats<span class="pl-pds">'</span></span>]
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled spider middlewares:
[<span class="pl-s"><span class="pl-pds">'</span>scrapy.spidermiddlewares.httperror.HttpErrorMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.spidermiddlewares.offsite.OffsiteMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.spidermiddlewares.referer.RefererMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.spidermiddlewares.urllength.UrlLengthMiddleware<span class="pl-pds">'</span></span>,
 <span class="pl-s"><span class="pl-pds">'</span>scrapy.spidermiddlewares.depth.DepthMiddleware<span class="pl-pds">'</span></span>]
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-03-04 22:56:19 [scrapy.core.engine] INFO: Spider opened
2017-03-04 22:56:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://api.imgur.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.sina.com.cn/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://twitter.com/imgur <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.msn.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.ask.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.apple.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.bbc.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.adobe.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://wordpress.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.tumblr.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.reddit.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.yahoo.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://www.imdb.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.aliexpress.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.youtube.com/ <span class="pl-k">!!!!!!!!!!!!!</span>

... seconds later ...

<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.paypal.com/us/home <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://stackoverflow.blog/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://www.amazon.co.jp/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/privacy <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/tags <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://login.live.com/login.srf<span class="pl-k">?</span>wa=wsignin1.0<span class="pl-k">&amp;</span>rpsnv=13<span class="pl-k">&amp;</span>ct=1488668190<span class="pl-k">&amp;</span>rver=6.4.6456.0<span class="pl-k">&amp;</span>wp=MBI_SSL_SHARED<span class="pl-k">&amp;</span>wreply=https:%2F%2Fmail.live.com%2Fdefault.aspx%3Frru%3Dinbox<span class="pl-k">&amp;</span>lc=1033<span class="pl-k">&amp;</span>id=64855<span class="pl-k">&amp;</span>mkt=en-US<span class="pl-k">&amp;</span>cbcxt=mai <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/upload <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/vidgif <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackexchange.com/sites <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/tour <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/memegen <span class="pl-k">!!!!!!!!!!!!!</span>
2017-03-04 22:56:39 [scrapy.extensions.logstats] INFO: Crawled 165 pages (at 90 pages/min), scraped 19 items (at 36 items/min)
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://meta.stackoverflow.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://imgur.com/signin<span class="pl-k">?</span>invokedBy=regularSignIn <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: https://imgur.com/register<span class="pl-k">?</span>invokedBy=regularSignIn <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/help <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/company/about <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/hot/viral <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackexchange.com/ <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/new/viral <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/<span class="pl-k">?</span>tab=interesting <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Funny <span class="pl-k">!!!!!!!!!!!!!</span>
2017-03-04 22:56:49 [scrapy.extensions.logstats] INFO: Crawled 175 pages (at 60 pages/min), scraped 25 items (at 36 items/min)
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/The_More_You_Know <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/<span class="pl-k">?</span>tab=featured <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Science_and_Tech <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/<span class="pl-k">?</span>tab=hot <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Gaming <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/<span class="pl-k">?</span>tab=week <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/<span class="pl-k">?</span>tab=month <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Eat_What_You_Want <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/questions/42602342/java-unlimited-cryptography-extension-doesnt-work <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Aww <span class="pl-k">!!!!!!!!!!!!!</span>
2017-03-04 22:56:59 [scrapy.extensions.logstats] INFO: Crawled 185 pages (at 60 pages/min), scraped 42 items (at 102 items/min)
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Inspiring <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/questions/tagged/authentication <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/questions/tagged/hdfs <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Awesome <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/questions/tagged/kerberos <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Creativity <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/questions/tagged/jce <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/The_Great_Outdoors <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://imgur.com/t/Storytime <span class="pl-k">!!!!!!!!!!!!!</span>
<span class="pl-k">!!!!!!!!!!!!!</span> Parsing: http://stackoverflow.com/users/3593261/user3593261 <span class="pl-k">!!!!!!!!!!!!!</span>
2017-03-04 22:57:09 [scrapy.extensions.logstats] INFO: Crawled 195 pages (at 60 pages/min), scraped 48 items (at 36 items/min)</pre></div>
<p dir="auto">Thanks, let me know if you need any other information!</p>
<p dir="auto">We need to add a deployment section to the documentation covering scrapyd-deploy.</p>