<p dir="auto">In version 14.2, if the duplicate filter should detect whether the URL its ignoring has been sent to a Spider previously. If not, it should not silently ignore the duplicate URL. A recent Request I did had a 302 redirect to an identical URL, before redirecting to a unique URL. It took half an hour before I realized that the scheduler was ignoring the first redirect. There were no debug indications or exceptions raised. I set dont_filter to True, which fixed the situation, as expected.</p>
<p dir="auto">Scrapy should either raise a quiet exception of some sort or realize that a duplicate URL had not previously been sent to a spider instead of silently ignoring duplicates.</p>
<p dir="auto">P.S.: I don't know why the site sent me to a duplicate URL, but it had modified cookies in the response.</p>
<h3 dir="auto">Description</h3>
<p dir="auto">In Scrapy 2.6, an ItemLoader instantiated with a base item keeps a reference to a single same item and instead of loading new items, it repeats the values added, therefore, producing duplicate items.</p>
<h3 dir="auto">Steps to Reproduce</h3>
<p dir="auto">Run the following script:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from scrapy.loader import ItemLoader

base_item = {}
for i in range(5):
    il = ItemLoader(item=base_item)
    il.add_value('b', i)
    new_item = il.load_item()
    print(new_item)"><pre class="notranslate"><code class="notranslate">from scrapy.loader import ItemLoader

base_item = {}
for i in range(5):
    il = ItemLoader(item=base_item)
    il.add_value('b', i)
    new_item = il.load_item()
    print(new_item)
</code></pre></div>
<p dir="auto">Output when ran in <code class="notranslate">scrapinghub-stack-scrapy:2.6</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{'b': [0]}
{'b': [0, 1]}
{'b': [0, 1, 2]}
{'b': [0, 1, 2, 3]}
{'b': [0, 1, 2, 3, 4]}"><pre class="notranslate"><code class="notranslate">{'b': [0]}
{'b': [0, 1]}
{'b': [0, 1, 2]}
{'b': [0, 1, 2, 3]}
{'b': [0, 1, 2, 3, 4]}
</code></pre></div>
<p dir="auto">Output when ran in <code class="notranslate">scrapinghub-stack-scrapy:1.6</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{'b': [0]}
{'b': [1]}
{'b': [2]}
{'b': [3]}
{'b': [4]}"><pre class="notranslate"><code class="notranslate">{'b': [0]}
{'b': [1]}
{'b': [2]}
{'b': [3]}
{'b': [4]}
</code></pre></div>
<p dir="auto"><strong>Expected behavior:</strong> The output should be the same as when ran in older Scrapy stack: <code class="notranslate">scrapinghub-stack-scrapy:1.6</code></p>
<p dir="auto"><strong>Possible resolution:</strong> Deep copying the <code class="notranslate">base_item</code> works as intended:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from scrapy.loader import ItemLoader
from copy import deepcopy

base_item = {}
for i in range(5):
    il = ItemLoader(item=deepcopy(base_item))
    il.add_value('b', i)
    new_item = il.load_item()
    print(new_item)"><pre class="notranslate"><code class="notranslate">from scrapy.loader import ItemLoader
from copy import deepcopy

base_item = {}
for i in range(5):
    il = ItemLoader(item=deepcopy(base_item))
    il.add_value('b', i)
    new_item = il.load_item()
    print(new_item)
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{'b': [0]}
{'b': [1]}
{'b': [2]}
{'b': [3]}
{'b': [4]}"><pre class="notranslate"><code class="notranslate">{'b': [0]}
{'b': [1]}
{'b': [2]}
{'b': [3]}
{'b': [4]}
</code></pre></div>