<p dir="auto">(Just logging the issue -- I haven't investigated if it's only about <code class="notranslate">custom_settings</code>)</p>
<p dir="auto">To reproduce, start a project using the basic template (which create a <code class="notranslate">settings.py</code> with <code class="notranslate">ROBOTSTXT_OBEY=True</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ scrapy version -v
Scrapy    : 1.1.2
lxml      : 3.6.3.0
libxml2   : 2.9.4
Twisted   : 16.3.2
Python    : 3.5.1+ (default, Mar 30 2016, 22:46:26) - [GCC 5.3.1 20160330]
pyOpenSSL : 16.0.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-4.4.0-34-generic-x86_64-with-Ubuntu-16.04-xenial

$ scrapy startproject overriddensettings
You can start your first spider with:
    cd overriddensettings
    scrapy genspider example example.com
$ cd overriddensettings/
/overriddensettings$ scrapy genspider example example.com
Created spider 'example' using template 'basic' in module:
  overriddensettings.spiders.example
$ cat overriddensettings/settings.py |grep ROBOTS
ROBOTSTXT_OBEY = True"><pre class="notranslate"><code class="notranslate">$ scrapy version -v
Scrapy    : 1.1.2
lxml      : 3.6.3.0
libxml2   : 2.9.4
Twisted   : 16.3.2
Python    : 3.5.1+ (default, Mar 30 2016, 22:46:26) - [GCC 5.3.1 20160330]
pyOpenSSL : 16.0.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-4.4.0-34-generic-x86_64-with-Ubuntu-16.04-xenial

$ scrapy startproject overriddensettings
You can start your first spider with:
    cd overriddensettings
    scrapy genspider example example.com
$ cd overriddensettings/
/overriddensettings$ scrapy genspider example example.com
Created spider 'example' using template 'basic' in module:
  overriddensettings.spiders.example
$ cat overriddensettings/settings.py |grep ROBOTS
ROBOTSTXT_OBEY = True
</code></pre></div>
<p dir="auto">Then, set <code class="notranslate">ROBOTSTXT_OBEY</code> to <code class="notranslate">False</code> in the spider's custom settings</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat overriddensettings/spiders/example.py 
# -*- coding: utf-8 -*-
import scrapy


class ExampleSpider(scrapy.Spider):
    name = &quot;example&quot;
    allowed_domains = [&quot;example.com&quot;]
    start_urls = (
        'http://www.example.com/',
    )
    custom_settings = {'ROBOTSTXT_OBEY': False}
    def parse(self, response):
        pass"><pre class="notranslate"><code class="notranslate">$ cat overriddensettings/spiders/example.py 
# -*- coding: utf-8 -*-
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = (
        'http://www.example.com/',
    )
    custom_settings = {'ROBOTSTXT_OBEY': False}
    def parse(self, response):
        pass
</code></pre></div>
<p dir="auto">The console logs show <code class="notranslate">'ROBOTSTXT_OBEY': True</code> in overridden settings, while it's in fact <code class="notranslate">False</code> in this test spider</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ scrapy crawl example
2016-08-26 14:49:43 [scrapy] INFO: Scrapy 1.1.2 started (bot: overriddensettings)
2016-08-26 14:49:43 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'overriddensettings', 'SPIDER_MODULES': ['overriddensettings.spiders'], 'NEWSPIDER_MODULE': 'overriddensettings.spiders', 'ROBOTSTXT_OBEY': True}
2016-08-26 14:49:43 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-08-26 14:49:43 [scrapy] INFO: Enabled item pipelines:
[]
2016-08-26 14:49:43 [scrapy] INFO: Spider opened
2016-08-26 14:49:43 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-08-26 14:49:43 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.example.com/&gt; (referer: None)
2016-08-26 14:49:44 [scrapy] INFO: Closing spider (finished)
2016-08-26 14:49:44 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 213,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 952,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 8, 26, 12, 49, 44, 49575),
 'log_count/DEBUG': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 8, 26, 12, 49, 43, 655150)}
"><pre class="notranslate"><code class="notranslate">$ scrapy crawl example
2016-08-26 14:49:43 [scrapy] INFO: Scrapy 1.1.2 started (bot: overriddensettings)
2016-08-26 14:49:43 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'overriddensettings', 'SPIDER_MODULES': ['overriddensettings.spiders'], 'NEWSPIDER_MODULE': 'overriddensettings.spiders', 'ROBOTSTXT_OBEY': True}
2016-08-26 14:49:43 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-08-26 14:49:43 [scrapy] INFO: Enabled item pipelines:
[]
2016-08-26 14:49:43 [scrapy] INFO: Spider opened
2016-08-26 14:49:43 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-08-26 14:49:43 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.example.com/&gt; (referer: None)
2016-08-26 14:49:44 [scrapy] INFO: Closing spider (finished)
2016-08-26 14:49:44 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 213,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 952,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 8, 26, 12, 49, 44, 49575),
 'log_count/DEBUG': 1,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 8, 26, 12, 49, 43, 655150)}

</code></pre></div>
<p dir="auto">How do I nest loaders into each parse function so that it returns everything from each method in a unique format? When I try to run the following, I get consistent duplicates. I thought the loader would retain the values from the xpath until the next parse method, and then return everything into the output processor but it seems to only grab the first item and create duplicates for the rest.</p>
<p dir="auto">Here's the scraper:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import scrapy
from scrapy.item import Field
from scrapy.crawler import CrawlerProcess
from itemloaders.processors import TakeFirst
from scrapy.loader import ItemLoader
import pandas as pd
from collections import defaultdict

class GsmItem(scrapy.Item):
    brandname = Field(output_processor = TakeFirst())
    devicecount = Field(output_processor = TakeFirst())
    mobiles = Field(output_processor = TakeFirst())
    stuff = Field(output_processor = TakeFirst())

class GsmSpider(scrapy.Spider):
    name = 'gsm'
    
    def start_requests(self):
        yield scrapy.Request(
            url = &quot;https://www.gsmarena.com/makers.php3&quot;,
            callback = self.parse
        )
    def parse(self, response):
        data = response.xpath(&quot;//div[@class='st-text']//tr&quot;)
        for devices in data:
            loader = ItemLoader(GsmItem(), selector = devices)
            loader.add_xpath('brandname', &quot;//div[@class='st-text']//tr//a//text()&quot;)
            loader.add_xpath('devicecount', &quot;//div[@class='st-text']//tr//a//span//text()&quot;)
            #yield loader.load_item()
            product_page = devices.xpath(&quot;//div[@class='st-text']//tr//a//@href&quot;).get()
            yield response.follow(
                product_page,
                callback = self.product_parse,
                cb_kwargs = {
                    'loader':loader
                }
            )
    def product_parse(self, response, loader):
        container = response.xpath(&quot;//div[@class='makers']//li&quot;)
        for mobiles in container:
            loader.add_value(&quot;mobiles&quot;, mobiles.xpath(&quot;.//a//strong//text()&quot;).getall())
            yield loader.load_item()

        next_page = response.xpath(&quot;//a[@title='Next page']//@href&quot;).get()
        if next_page:
            yield response.follow(
                response.urljoin(next_page),
                callback = self.parse_specs,
                cb_kwargs = {
                    'loader':loader
                }
            )
    def parse_specs(self, response, loader):
        loader.add_value(&quot;stuff&quot;,response.xpath(&quot;//th[normalize-space()='Launch']&quot;).get())
        yield loader.load_item()

process = CrawlerProcess(
        settings={
            &quot;CONCURRENT_REQUESTS&quot;: 64,
            &quot;FEED_URI&quot;:'test_scraper.jl',
            &quot;FEED_FORMAT&quot;:'jsonlines',
        }
    )
process.crawl(GsmSpider)
process.start()"><pre class="notranslate"><code class="notranslate">import scrapy
from scrapy.item import Field
from scrapy.crawler import CrawlerProcess
from itemloaders.processors import TakeFirst
from scrapy.loader import ItemLoader
import pandas as pd
from collections import defaultdict

class GsmItem(scrapy.Item):
    brandname = Field(output_processor = TakeFirst())
    devicecount = Field(output_processor = TakeFirst())
    mobiles = Field(output_processor = TakeFirst())
    stuff = Field(output_processor = TakeFirst())

class GsmSpider(scrapy.Spider):
    name = 'gsm'
    
    def start_requests(self):
        yield scrapy.Request(
            url = "https://www.gsmarena.com/makers.php3",
            callback = self.parse
        )
    def parse(self, response):
        data = response.xpath("//div[@class='st-text']//tr")
        for devices in data:
            loader = ItemLoader(GsmItem(), selector = devices)
            loader.add_xpath('brandname', "//div[@class='st-text']//tr//a//text()")
            loader.add_xpath('devicecount', "//div[@class='st-text']//tr//a//span//text()")
            #yield loader.load_item()
            product_page = devices.xpath("//div[@class='st-text']//tr//a//@href").get()
            yield response.follow(
                product_page,
                callback = self.product_parse,
                cb_kwargs = {
                    'loader':loader
                }
            )
    def product_parse(self, response, loader):
        container = response.xpath("//div[@class='makers']//li")
        for mobiles in container:
            loader.add_value("mobiles", mobiles.xpath(".//a//strong//text()").getall())
            yield loader.load_item()

        next_page = response.xpath("//a[@title='Next page']//@href").get()
        if next_page:
            yield response.follow(
                response.urljoin(next_page),
                callback = self.parse_specs,
                cb_kwargs = {
                    'loader':loader
                }
            )
    def parse_specs(self, response, loader):
        loader.add_value("stuff",response.xpath("//th[normalize-space()='Launch']").get())
        yield loader.load_item()

process = CrawlerProcess(
        settings={
            "CONCURRENT_REQUESTS": 64,
            "FEED_URI":'test_scraper.jl',
            "FEED_FORMAT":'jsonlines',
        }
    )
process.crawl(GsmSpider)
process.start()
</code></pre></div>