<p dir="auto">This is scrapy's default <code class="notranslate">Dupefilter</code> class method <code class="notranslate">request_seen</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class RFPDupeFilter(BaseDupeFilter):

    def request_seen(self, request):
        fp = self.request_fingerprint(request)
        if fp in self.fingerprints:
            return True
        self.fingerprints.add(fp)
        if self.file:
            self.file.write(fp + os.linesep)"><pre class="notranslate"><code class="notranslate">class RFPDupeFilter(BaseDupeFilter):

    def request_seen(self, request):
        fp = self.request_fingerprint(request)
        if fp in self.fingerprints:
            return True
        self.fingerprints.add(fp)
        if self.file:
            self.file.write(fp + os.linesep)
</code></pre></div>
<p dir="auto">While implementing a custom dupefilter. i cannot retrieve the <code class="notranslate">spider</code> object from this class unlike other scrapy middleware</p>
<p dir="auto">Is there any way i can know which <code class="notranslate">spider</code> object this is? so i can customize it via a spider on spider basis?</p>
<p dir="auto">Also i cannot just implement a middleware which reads urls and puts it into a list &amp; checks duplicates instead of a custom dupefilter. This is because i need to pause/resume crawls and need scrapy to store the request  fingerprint by default using the <code class="notranslate">JOBDIR</code> setting</p>
<h2 dir="auto">Summary</h2>
<p dir="auto">Let us use DI framework in the pipelines and spiders. My personal favourite: <a href="https://python-dependency-injector.ets-labs.org/introduction/di_in_python.html" rel="nofollow">https://python-dependency-injector.ets-labs.org/introduction/di_in_python.html</a></p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">We have a lot of common code we want to reuse in our scraping framework. Using DI to keep them tidy is nice to have.</p>
<h2 dir="auto">Describe alternatives you've considered</h2>
<p dir="auto">Duplicate all the code and constructions. But only idiots likes duplicated codes or make global variables.</p>