<p dir="auto">Hi all,</p>
<p dir="auto">My name is Martin, a master candidate in information system at Carnegie Mellon. I have worked as an analyst and product manager before, but I would like to become more involved in the development space and contribute more to open source projects.</p>
<p dir="auto">I have personally used scrapy a lot, using it both for personal projects as well as for work. I would be extremely excited to contribute to this project through GoSC.</p>
<p dir="auto">I find the request fingerprints project very interesting, as I have personally faced issues with this before. For example, when I scraped Instagram, the website uses a "session_id" to check on the frequency of requests. The default behavior works in this case, but if I want to see how different users sees the same URL (for example, the recommendation page), it would be hard with the current framework.</p>
<p dir="auto">I checked the source code and found the fingerprint is implemented here.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="_fingerprint_cache = weakref.WeakKeyDictionary()
def request_fingerprint(request, include_headers=None):
    if include_headers:
        include_headers = tuple(to_bytes(h.lower())
                                 for h in sorted(include_headers))
    cache = _fingerprint_cache.setdefault(request, {})
    if include_headers not in cache:
        fp =  ()
        fp.update(to_bytes(request.method))
        fp.update(to_bytes(canonicalize_url(request.url)))
        fp.update(request.body or b'')
        if include_headers:
            for hdr in include_headers:
                if hdr in request.headers:
                    fp.update(hdr)
                    for v in request.headers.getlist(hdr):
                        fp.update(v)
        cache[include_headers] = fp.hexdigest()
    return cache[include_headers]"><pre class="notranslate"><code class="notranslate">_fingerprint_cache = weakref.WeakKeyDictionary()
def request_fingerprint(request, include_headers=None):
    if include_headers:
        include_headers = tuple(to_bytes(h.lower())
                                 for h in sorted(include_headers))
    cache = _fingerprint_cache.setdefault(request, {})
    if include_headers not in cache:
        fp =  ()
        fp.update(to_bytes(request.method))
        fp.update(to_bytes(canonicalize_url(request.url)))
        fp.update(request.body or b'')
        if include_headers:
            for hdr in include_headers:
                if hdr in request.headers:
                    fp.update(hdr)
                    for v in request.headers.getlist(hdr):
                        fp.update(v)
        cache[include_headers] = fp.hexdigest()
    return cache[include_headers]
</code></pre></div>
<p dir="auto">The function is used in the dupefilter class, but the optional parameter "include_header" is never used anywhere in the project. The documentation (see below) says that you can customize this by overwriting the request_fingerprint function. However, this will be out of scope for most users who did the pip install, and I think users should be provided more convinient ways to adjust this setting.</p>
<blockquote>
<p dir="auto">The default (RFPDupeFilter) filters based on request fingerprint using the scrapy.utils.request.request_fingerprint function. In order to change the way duplicates are checked you could subclass RFPDupeFilter and override its request_fingerprint method. This method should accept scrapy Request object and return its fingerprint (a string).</p>
</blockquote>
<p dir="auto">On the idea page, it says</p>
<blockquote>
<p dir="auto">Scrapy uses a Request fingerprinting scheme for de-duplicating requests and for caching. Currently, the fingerprinting algorithm cannot be modified by Scrapy users.</p>
</blockquote>
<p dir="auto">No use cases were mentioned here. The use case I can think of is to provide user simple ways to treat certain headers as part of the fingerprint (for example, language settings or session_id for some sites). Under the current implementation, even if the user chooses to enable the "include_headers" parameter, he/she cannot specify the part of headers to be included, meaning things like rotating User-Agent will cause the program to crawl the same page multiple times.</p>
<p dir="auto">Those are just my two cents on this issue. If anyone can provide more use cases on this, it can really help me think this through.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Gallaecio/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Gallaecio">@Gallaecio</a> If you can provide some comment on this it would be great :)</p>
<p dir="auto">Thanks in advance,<br>
Martin</p>
<p dir="auto"><code class="notranslate">import logging</code><br>
<code class="notranslate">import coloredlogs</code></p>
<p dir="auto"><code class="notranslate">logging.captureWarnings(True)</code><br>
<code class="notranslate">LOGGER = logging.getLogger(__name__)</code><br>
<code class="notranslate">coloredlogs.install(level='INFO')</code></p>
<p dir="auto"><strong>Initialization is done this way and translates into a duplication of logs in the output. Any solution to keep only the colored logs?</strong></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/23501993/161999047-5f173f40-04d0-4cf1-b029-18788ccf8877.png"><img src="https://user-images.githubusercontent.com/23501993/161999047-5f173f40-04d0-4cf1-b029-18788ccf8877.png" alt="image" style="max-width: 100%;"></a></p>