<p dir="auto">Hello. First I would like to say it is not a bug report, but rather an expression of my inability to find answers by myself.</p>
<p dir="auto">IMPORTANT<br>
I would like to see the set of  correct settings that lowers memory consumption to minimum while retaining speed at a high enough level with using proxy/dropped requests and scale at around 1-10M pages and 1000-2000 requests/minute. Now it goes up to 4Gb (can be not correct)</p>
<p dir="auto">NOT IMPORTANT</p>
<ol dir="auto">
<li>Is it possible to use the same cache db file while running two spiders with the same name? (I tried and half of scraped data has been lost. (not 100% sure but still))</li>
<li>Is it possible to automatically turn off windows embedded antivirus?(speed issue)</li>
<li>Is it possible to run one spider and fetch cache from 2 sets of previously created files?</li>
</ol>
<p dir="auto">I've tried to use a AUTOTHROTTLE_TARGET_CONCURRENCY but it is don't increase speed without increasing CONCURRENT_REQUESTS.</p>
<p dir="auto">DEPTH_PRIORITY = -1<br>
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleLifoDiskQueue'<br>
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.LifoMemoryQueue'<br>
(will it have any effect without -s JOBDIR setting?)<br>
RETRY_PRIORITY_ADJUST = 10<br>
DUPEFILTER_CLASS = 'scrapy.dupefilters.BaseDupeFilter'</p>
<p dir="auto">HTTPCACHE_ENABLED = True<br>
It seems that httpcache and dedupfilter increases memory consumption. (even for a simple 1-2 depth site.) Is it true?</p>
<p dir="auto">Should I just read the source?</p>
<p dir="auto">There are situations where websites return 200 responses but the content is not available due to bans or temporal issues which can be fixed by retrying requests.</p>
<p dir="auto">There should be an easier way to retry requests inside spider callbacks, which should ideally reuse the code in Retry downloader middleware.</p>
<p dir="auto">I see two approaches for this.</p>
<ol dir="auto">
<li>
<p dir="auto">Introduce new exception called RetryRequest which can be raised inside a spider callback to indicate a retry. I personally prefer this but the implementation of this is a little untidy due to this bug <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="9634974" data-permission-text="Title is private" data-url="https://github.com/scrapy/scrapy/issues/220" data-hovercard-type="issue" data-hovercard-url="/scrapy/scrapy/issues/220/hovercard" href="https://github.com/scrapy/scrapy/issues/220">#220</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from scrapy.exceptions import RetryRequest

def parse(self, response):
    if response.xpath('//title[text()=&quot;Content not found&quot;]'):
        raise RetryRequest('Missing content')"><pre class="notranslate"><code class="notranslate">from scrapy.exceptions import RetryRequest

def parse(self, response):
    if response.xpath('//title[text()="Content not found"]'):
        raise RetryRequest('Missing content')
</code></pre></div>
</li>
<li>
<p dir="auto">Introduce a new class RetryRequest which wraps a request that needs to be retried. A RetryRequest can be yielded from a spider callback to indicate a retry</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from scrapy.http import RetryRequest

def parse(self, response):
    if response.xpath('//title[text()=&quot;Content not found&quot;]'):
        yield RetryRequest(response.request, reason='Missing content')"><pre class="notranslate"><code class="notranslate">from scrapy.http import RetryRequest

def parse(self, response):
    if response.xpath('//title[text()="Content not found"]'):
        yield RetryRequest(response.request, reason='Missing content')
</code></pre></div>
</li>
</ol>
<p dir="auto">Will be sending two PRs for the two approaches. Happy to hear about any other alternatives too.</p>