<p dir="auto">By default, the link extractor calls canonicalize_url on the collected links.<br>
The following is not what browsers do:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="canonicalize_url('http://example.com/index.php?/a/=/o/')
'http://example.com/index.php?%2Fa%2F=%2Fo%2F'  # encoding forward slashes
canonicalize_url('http://example.com/index.php?a')
'http://example.com/index.php?a='  # appending = on empty arguments"><pre class="notranslate"><span class="pl-en">canonicalize_url</span>(<span class="pl-s">'http://example.com/index.php?/a/=/o/'</span>)
<span class="pl-s">'http://example.com/index.php?%2Fa%2F=%2Fo%2F'</span>  <span class="pl-c"># encoding forward slashes</span>
<span class="pl-en">canonicalize_url</span>(<span class="pl-s">'http://example.com/index.php?a'</span>)
<span class="pl-s">'http://example.com/index.php?a='</span>  <span class="pl-c"># appending = on empty arguments</span></pre></div>
<p dir="auto">I doubt this is a problem in canonicalize_url<br>
because it's not meant to mimic browsers in the first place, is it?</p>
<p dir="auto">However this is a problem for the link extractor<br>
because it can potentially end up extracting urls<br>
that are wrong from the server's perspective.<br>
In this example, the server doesn't recognise the extractor's url, only the browser's:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# http://forum.laptop.bg/index.php?/discover/
LinkExtractor(restrict_xpaths=('//a[contains(@href, &quot;/topic&quot;)]',)).extract_links(response)[0].url
# Extractor: http://forum.laptop.bg/index.php?%2Ftopic%2F57339-%D0%BB%D0%B0%D0%BF%D1%82%D0%BE%D0%BF-asus-w90vp%2F=&amp;comment=221153&amp;do=findComment
# Browser:   http://forum.laptop.bg/index.php?/topic/57339-%D0%BB%D0%B0%D0%BF%D1%82%D0%BE%D0%BF-asus-w90vp/&amp;do=findComment&amp;comment=221153"><pre class="notranslate"><span class="pl-c"># http://forum.laptop.bg/index.php?/discover/</span>
<span class="pl-v">LinkExtractor</span>(<span class="pl-s1">restrict_xpaths</span><span class="pl-c1">=</span>(<span class="pl-s">'//a[contains(@href, "/topic")]'</span>,)).<span class="pl-en">extract_links</span>(<span class="pl-s1">response</span>)[<span class="pl-c1">0</span>].<span class="pl-s1">url</span>
<span class="pl-c"># Extractor: http://forum.laptop.bg/index.php?%2Ftopic%2F57339-%D0%BB%D0%B0%D0%BF%D1%82%D0%BE%D0%BF-asus-w90vp%2F=&amp;comment=221153&amp;do=findComment</span>
<span class="pl-c"># Browser:   http://forum.laptop.bg/index.php?/topic/57339-%D0%BB%D0%B0%D0%BF%D1%82%D0%BE%D0%BF-asus-w90vp/&amp;do=findComment&amp;comment=221153</span></pre></div>
<p dir="auto">Was this a design decision or a bug?</p>
<p dir="auto">Cannot override the default <code class="notranslate">Connection: close</code> header of scrapy requests. I want to scrape a site for which I need to send <code class="notranslate">Connection: keep-alive</code> headers and remove the default <code class="notranslate">Connection: close</code> header.</p>
<p dir="auto">Tried using custom settings, custom headers and default headers to change <code class="notranslate">Connection: close</code> to <code class="notranslate">Connection: keep-alive</code> but it instead merges and sends two <code class="notranslate">Connection</code> headers.</p>
<p dir="auto">It works in scrapy shell but not when used in a spider.</p>
<p dir="auto">I used Fiddler to check the headers scrapy was sending. Please use Fiddler or any other packet sniffing tool to check the headers as it works fine in scrapy shell.</p>
<p dir="auto">Below is the different things I tried:<br>
Using custom settings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="custom_settings = {
        'DEFAULT_REQUEST_HEADERS': {
            'Connection': 'keep-alive'
        }
    }"><pre class="notranslate"><code class="notranslate">custom_settings = {
        'DEFAULT_REQUEST_HEADERS': {
            'Connection': 'keep-alive'
        }
    }
</code></pre></div>
<p dir="auto">Using custom headers:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="headers = {&quot;Connection&quot;: &quot;keep-alive&quot;}

for url in urls:
        yield scrapy.Request(url=url, callback=self.parse, headers=headers)"><pre class="notranslate"><code class="notranslate">headers = {"Connection": "keep-alive"}

for url in urls:
        yield scrapy.Request(url=url, callback=self.parse, headers=headers)
</code></pre></div>
<p dir="auto">Changing default header settings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DEFAULT_REQUEST_HEADERS = {
  'Connection': 'keep-alive',
}"><pre class="notranslate"><code class="notranslate">DEFAULT_REQUEST_HEADERS = {
  'Connection': 'keep-alive',
}
</code></pre></div>