<p dir="auto">Hi,</p>
<p dir="auto">I am using Scrapy version 1.1.2. It is somenthing weird, because depending on the Rules configuration, the callback function is called or not...</p>
<p dir="auto">My rules:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="rules = (
        Rule(LinkExtractor(allow=('.*balsamo'))), # get category page
        Rule(LinkExtractor(allow=(&quot;.*\/\d+.*&quot;),restrict_xpaths=('//*[@id=&quot;center_column&quot;]/div[3]')), callback='category'), # get product pages links in the category page
)"><pre class="notranslate"><code class="notranslate">rules = (
        Rule(LinkExtractor(allow=('.*balsamo'))), # get category page
        Rule(LinkExtractor(allow=(".*\/\d+.*"),restrict_xpaths=('//*[@id="center_column"]/div[3]')), callback='category'), # get product pages links in the category page
)
</code></pre></div>
<p dir="auto">If I remove the restrict_xpaths filter, the callback function is called, but when I use restrict_xpaths, the callback function is never called, even when the ouput shows the right crawled pages as "crawled". This is the output resulting from the above rules.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/robots.txt (referer: None)
2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es (referer: None)
2016-09-28 19:00:58 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/75-balsamo (referer: http://www.example.es)
2016-09-28 19:00:58 [scrapy] DEBUG: Filtered duplicate request: GET http://www.example.es/75-balsamo - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/233-relajante-balsamo-aromatico-dream-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/459-balsamo-de-tigre-rojo-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/235-refrescante-bio-balsamo-hierbas-classic-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/234-tranquilizador-balsamo-aromatico-balance-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/460-balsamo-de-tigre-blanco-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] INFO: Closing spider (finished)"><pre class="notranslate"><code class="notranslate">2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/robots.txt (referer: None)
2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es (referer: None)
2016-09-28 19:00:58 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/75-balsamo (referer: http://www.example.es)
2016-09-28 19:00:58 [scrapy] DEBUG: Filtered duplicate request: GET http://www.example.es/75-balsamo - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/233-relajante-balsamo-aromatico-dream-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/459-balsamo-de-tigre-rojo-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/235-refrescante-bio-balsamo-hierbas-classic-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/234-tranquilizador-balsamo-aromatico-balance-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/460-balsamo-de-tigre-blanco-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] INFO: Closing spider (finished)
</code></pre></div>
<p dir="auto">I totally dig this idea, but I also think this needs to be well thought-out or it could spell more trouble than it solves and be hard to deprecate and fix later.</p>
<p dir="auto">My thoughts on the current proposal:</p>
<p dir="auto"><code class="notranslate">[addons]</code> in <code class="notranslate">scrapy.cfg</code> would not provide any way to pass configuration settings. So we still need to modify settings.py for, say, the database connection parameters.<br>
That doesn't look very portable.</p>
<p dir="auto">Also, if two different addons expose or use the same settings, it gets ugly.<br>
Say we have different addons, one with a MongoPipeline and one with a MongoQueue, which by chance both use the <code class="notranslate">MONGO_COLLECTION</code> setting name.</p>
<p dir="auto">This is obviously already a problem, but I think with add-ons, people would more expect a "plug-and-play" system which "just works", and be less careful with checking individual addon-components for name clashes or dependency issues.</p>
<p dir="auto">Unless addons should be expected to be configured prior to use, or only as a bundle for hardcoded configuration settings, I think we need another layer deep to have addons and their settings in the same place.<br>
Maybe we can use a <code class="notranslate">[addon_(name)]</code> (or simply <code class="notranslate">[(addon_name)]</code>) ini setting which can then take multiple parameters.<br>
These settings should be namespaced to the addon.<br>
Example:</p>
<div class="highlight highlight-source-ini notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[httpcache] # (looking in python path, no settings)
#enabled = True # (does ini-style require a parameter to recognize the section?)
[mongodb_pipeline]
path = /path/to/mongodb_pipeline.py # (explicit path)
host = 'localhost'
port = 27017"><pre class="notranslate"><span class="pl-en">[httpcache]</span> <span class="pl-c"><span class="pl-c">#</span> (looking in python path, no settings)</span>
<span class="pl-c"><span class="pl-c">#</span>enabled = True # (does ini-style require a parameter to recognize the section?)</span>
<span class="pl-en">[mongodb_pipeline]</span>
<span class="pl-k">path</span> = /path/to/mongodb_pipeline.py <span class="pl-c"><span class="pl-c">#</span> (explicit path)</span>
<span class="pl-k">host</span> = <span class="pl-s"><span class="pl-pds">'</span>localhost<span class="pl-pds">'</span></span>
<span class="pl-k">port</span> = 27017</pre></div>
<p dir="auto">These settings would then expand to <code class="notranslate">(addon_name)_(setting)</code>, e.g. <code class="notranslate">mongodb_pipeline_port</code> or <code class="notranslate">MONGODB_PIPELINE_PORT</code> to prevent namespace clashes between addons.</p>
<p dir="auto">(Would this copy settings on scrapyd deployment?)<br>
Alternatively, maybe all the addon defining and configuration could be in settings.py, but I think it should be in a single place.</p>