<p dir="auto">Pages which have duplicate values in their query string are treated as different pages:</p>
<ul dir="auto">
<li><a href="http://www.example.com/?q=" rel="nofollow">http://www.example.com/?q=</a></li>
<li><a href="http://www.example.com/?q=&amp;q=" rel="nofollow">http://www.example.com/?q=&amp;q=</a></li>
<li><a href="http://www.example.com/?q=&amp;q=&amp;q=" rel="nofollow">http://www.example.com/?q=&amp;q=&amp;q=</a></li>
<li>...</li>
</ul>
<p dir="auto">If the first page has a link to the second and the second a link to the third, <code class="notranslate">scrapy</code> enters an infinite loop, requesting each page in succession. I've put a simple page <a href="http://fcoelho.alwaysdata.net/scrapy/page.php" rel="nofollow">here</a> to test the issue. Using a CrawlSpider with a rule like <code class="notranslate">Rule(SgmlLinkExtractor(), callback='parse_page', follow=True)</code> will cause this infinite recursion. Here is a sample spider:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

class LoopingSpider(CrawlSpider):
    name = &quot;loop&quot;
    start_urls = [&quot;http://fcoelho.alwaysdata.net/scrapy/page.php&quot;]
    allowed_domains = [&quot;fcoelho.alwaysdata.net&quot;]
    rules = (
        Rule(SgmlLinkExtractor(), callback='parse_page', follow=True),
    )

    def parse_page(self, response):
        print &quot;Page: %s&quot; % response.url
        return []"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">spiders</span> <span class="pl-k">import</span> <span class="pl-v">CrawlSpider</span>, <span class="pl-v">Rule</span>
<span class="pl-k">from</span> <span class="pl-s1">scrapy</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">linkextractors</span>.<span class="pl-s1">sgml</span> <span class="pl-k">import</span> <span class="pl-v">SgmlLinkExtractor</span>

<span class="pl-k">class</span> <span class="pl-v">LoopingSpider</span>(<span class="pl-v">CrawlSpider</span>):
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s">"loop"</span>
    <span class="pl-s1">start_urls</span> <span class="pl-c1">=</span> [<span class="pl-s">"http://fcoelho.alwaysdata.net/scrapy/page.php"</span>]
    <span class="pl-s1">allowed_domains</span> <span class="pl-c1">=</span> [<span class="pl-s">"fcoelho.alwaysdata.net"</span>]
    <span class="pl-s1">rules</span> <span class="pl-c1">=</span> (
        <span class="pl-v">Rule</span>(<span class="pl-v">SgmlLinkExtractor</span>(), <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s">'parse_page'</span>, <span class="pl-s1">follow</span><span class="pl-c1">=</span><span class="pl-c1">True</span>),
    )

    <span class="pl-k">def</span> <span class="pl-en">parse_page</span>(<span class="pl-s1">self</span>, <span class="pl-s1">response</span>):
        <span class="pl-k">print</span> <span class="pl-s">"Page: %s"</span> <span class="pl-c1">%</span> <span class="pl-s1">response</span>.<span class="pl-s1">url</span>
        <span class="pl-k">return</span> []</pre></div>
<p dir="auto">It seems like in the end the problem "arises" because <code class="notranslate">scrapy.utils.url.canonicalize_url</code> only sorts the query string keys, but don't remove duplicates. As far as I can tell, duplicates are "wrong", and shouldn't be counted as normal web page behavior. Would it be sane to remove arguments from the query string if their key is repeated?</p>
<h2 dir="auto">Summary</h2>
<p dir="auto">If the file mentioned in <code class="notranslate">scrapy genspider</code> already exists, then genspider should refuse to generate the the file.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">As it stands, existing code can be blown away if this command runs twice.</p>
<h2 dir="auto">Describe alternatives you've considered</h2>
<p dir="auto">Prompting the user for overwriting existing spider.</p>