<p dir="auto">Hi,</p>
<p dir="auto">I am using SplashFormRequest.from_response() method to submit a form multiple times with different parameters. The formdata is passed on correctly it seems, but eventually when the request is executed, only the first set of parameters are used for the subsequent requests too. I ran a test on google to submit 3 different parameters on the search form and get the same issue. I am pasting the relevant piece of code and part of the output log from the crawl. Any idea why this might be happening? Thanks.</p>
<p dir="auto">This is part of the parse method.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="            for formd in formparams:
              print('===== Submitting FORM ===== ' + json.dumps(formd))
              yield SplashFormRequest.from_response(
                response,  
                formdata=formd,
                method=GET,
                callback=self.parse,
                meta=meta, **kwargs)"><pre class="notranslate"><code class="notranslate">            for formd in formparams:
              print('===== Submitting FORM ===== ' + json.dumps(formd))
              yield SplashFormRequest.from_response(
                response,  
                formdata=formd,
                method=GET,
                callback=self.parse,
                meta=meta, **kwargs)
</code></pre></div>
<p dir="auto">Output log</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2017-10-02 23:38:02 [scrapy.core.engine] INFO: Spider opened
2017-10-02 23:38:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
REQUEST === &lt;GET http://google.com&gt;
REQUEST === &lt;GET http://www.google.com via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:09 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET http://www.google.com via http://127.0.0.1:8050/execute&gt; (referer: None)
2017-10-02 23:38:10 [scrapy.core.scraper] DEBUG: Scraped from &lt;200 https://www.google.com/?gws_rd=ssl&gt;
&lt;CDRItem: _id: '78EA5573E093F9030B1F945E84B59B51D405C360BB4C421ED93A66976195BC2C', url: 'https://www.google.com/?gws_rd=ssl', timestamp_crawl: '2017-10-03T06:38:10.971013Z'&gt;
===== Submitting FORM ===== {&quot;q&quot;: &quot;happy&quot;}
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy&gt;
===== Submitting FORM ===== {&quot;q&quot;: &quot;sad&quot;}
2017-10-02 23:38:10 [py.warnings] WARNING: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy_splash/middleware.py:184: UserWarning: Unexpected request.meta['splash']['_replaced_args']
  warnings.warn(&quot;Unexpected request.meta['splash']['_replaced_args']&quot;)
  
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=sad&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:10 [scrapy.dupefilters] DEBUG: Filtered duplicate request: &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
===== Submitting FORM ===== {&quot;q&quot;: &quot;neutral&quot;}
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=neutral&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:11 [scrapy.dupefilters] DEBUG: Filtered duplicate request: &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:22 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt; (referer: None)
2017-10-02 23:38:22 [scrapy.core.scraper] DEBUG: Scraped from &lt;200 https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy&gt;
&lt;CDRItem: _id: 'B31AF63FD579B31F9A18BE78F30F845C21DD5F7DE36838FE87CED836E5783BEB', url: 'https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy', timestamp_crawl: '2017-10-03T06:38:22.662764Z'&gt;
2017-10-02 23:38:22 [scrapy.core.engine] INFO: Closing spider (finished)"><pre class="notranslate"><code class="notranslate">2017-10-02 23:38:02 [scrapy.core.engine] INFO: Spider opened
2017-10-02 23:38:02 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
REQUEST === &lt;GET http://google.com&gt;
REQUEST === &lt;GET http://www.google.com via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:09 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET http://www.google.com via http://127.0.0.1:8050/execute&gt; (referer: None)
2017-10-02 23:38:10 [scrapy.core.scraper] DEBUG: Scraped from &lt;200 https://www.google.com/?gws_rd=ssl&gt;
&lt;CDRItem: _id: '78EA5573E093F9030B1F945E84B59B51D405C360BB4C421ED93A66976195BC2C', url: 'https://www.google.com/?gws_rd=ssl', timestamp_crawl: '2017-10-03T06:38:10.971013Z'&gt;
===== Submitting FORM ===== {"q": "happy"}
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy&gt;
===== Submitting FORM ===== {"q": "sad"}
2017-10-02 23:38:10 [py.warnings] WARNING: /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/scrapy_splash/middleware.py:184: UserWarning: Unexpected request.meta['splash']['_replaced_args']
  warnings.warn("Unexpected request.meta['splash']['_replaced_args']")
  
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=sad&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:10 [scrapy.dupefilters] DEBUG: Filtered duplicate request: &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
===== Submitting FORM ===== {"q": "neutral"}
REQUEST === &lt;GET http://google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=neutral&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:11 [scrapy.dupefilters] DEBUG: Filtered duplicate request: &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
REQUEST === &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt;
2017-10-02 23:38:22 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy via http://127.0.0.1:8050/execute&gt; (referer: None)
2017-10-02 23:38:22 [scrapy.core.scraper] DEBUG: Scraped from &lt;200 https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy&gt;
&lt;CDRItem: _id: 'B31AF63FD579B31F9A18BE78F30F845C21DD5F7DE36838FE87CED836E5783BEB', url: 'https://www.google.com/search?source=hp&amp;oq=&amp;gs_l=&amp;btnG=Search&amp;q=happy', timestamp_crawl: '2017-10-03T06:38:22.662764Z'&gt;
2017-10-02 23:38:22 [scrapy.core.engine] INFO: Closing spider (finished)
</code></pre></div>
<p dir="auto">It is very easy to have a subtle bug when using a custom duplicates filter that changes how request fingerprint is calculated.</p>
<ul dir="auto">
<li>Duplicate filter checks request fingerprint and makes Scheduler drop the request if it is a duplicate.</li>
<li>Cache storage checks request fingerprint and fetches response from cache if it is a duplicate.</li>
<li>If fingerprint algorithms differ we're in trouble.</li>
</ul>
<p dir="auto">The problem is that there is no way to override request fingerprint globally; to make Scrapy always take something extra in account (an http header, a meta option) user must override duplicates filter and all cache storages that are in use.</p>
<p dir="auto">Ideas about how to fix it:</p>
<ol dir="auto">
<li>Use duplicates filter <code class="notranslate">request_fingerprint</code> method in cache storage if this method is available;</li>
<li>create a special Request.meta key that <code class="notranslate">request_fingerprint</code> function will take into account;</li>
<li>create a special Request.meta key that will allow to provide a pre-calculated fingerprint;</li>
<li>add a settings.py option to override request fingerprint function globally.</li>
</ol>