<p dir="auto">if allowed_domains occurred in url(in path or query) it also crawl that url.<br>
e.g:<br>
allowed_domains =["www.rankwatch.com"]</p>
<p dir="auto">when i run LinkExtractor it also crawl :<br>
{"url": "<a href="https://www.facebook.com/login.php?skip_api_login=1&amp;api_key=864596490301578&amp;signed_next=1&amp;next=https%3A%2F%2Fwww.facebook.com%2Fv2.4%2Fdialog%2Foauth%3Fredirect_uri%3Dhttps%253A%252F%252Fwww.rankwatch.com%252Flearning%252Fuser%252Fsimple-fb-connect%252Freturn%26state%3Daa68bcc193612233a0511106f69f9203%26scope%3Demail%26client_id%3D864596490301578%26ret%3Dlogin%26sdk%3Dphp-sdk-4.0.23%26logger_id%3Df7320741-707c-2f7e-1947-390b18a93e99&amp;cancel_url=https%3A%2F%2Fwww.rankwatch.com%2Flearning%2Fuser%2Fsimple-fb-connect%2Freturn%3Ferror%3Daccess_denied%26error_code%3D200%26error_description%3DPermissions%2Berror%26error_reason%3Duser_denied%26state%3Daa68bcc193612233a0511106f69f9203%23_%3D_&amp;display=page&amp;locale=en_GB&amp;logger_id=f7320741-707c-2f7e-1947-390b18a93e99%22%7D" rel="nofollow">https://www.facebook.com/login.php?skip_api_login=1&amp;api_key=864596490301578&amp;signed_next=1&amp;next=https%3A%2F%2Fwww.facebook.com%2Fv2.4%2Fdialog%2Foauth%3Fredirect_uri%3Dhttps%253A%252F%252Fwww.rankwatch.com%252Flearning%252Fuser%252Fsimple-fb-connect%252Freturn%26state%3Daa68bcc193612233a0511106f69f9203%26scope%3Demail%26client_id%3D864596490301578%26ret%3Dlogin%26sdk%3Dphp-sdk-4.0.23%26logger_id%3Df7320741-707c-2f7e-1947-390b18a93e99&amp;cancel_url=https%3A%2F%2Fwww.rankwatch.com%2Flearning%2Fuser%2Fsimple-fb-connect%2Freturn%3Ferror%3Daccess_denied%26error_code%3D200%26error_description%3DPermissions%2Berror%26error_reason%3Duser_denied%26state%3Daa68bcc193612233a0511106f69f9203%23_%3D_&amp;display=page&amp;locale=en_GB&amp;logger_id=f7320741-707c-2f7e-1947-390b18a93e99"}</a>,</p>
<p dir="auto">Hello,</p>
<p dir="auto">To avoid crawling duplicated URLs, I store all crawled URLs in database, and then check if one URL requested is already in DB or not.</p>
<p dir="auto">Here is my filter class:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class CustomFilter(RFPDupeFilter):
    &quot;&quot;&quot;A dupe filter that considers specific ids in the url&quot;&quot;&quot;
    db = DBManager(settings[ 'MONGODB_VISITED_URLS' ])

    def request_seen(self, request):
        if self.db.exist(&quot;url&quot;, request.url):
            return True
        else:
            visitedUrl = VisitedURL()
            visitedUrl['url'] = request.url
            visitedUrl['orgDupURL'] = None
            self.db.insert(visitedUrl)
            return False"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-v">CustomFilter</span>(<span class="pl-v">RFPDupeFilter</span>):
    <span class="pl-s">"""A dupe filter that considers specific ids in the url"""</span>
    <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-v">DBManager</span>(<span class="pl-s1">settings</span>[ <span class="pl-s">'MONGODB_VISITED_URLS'</span> ])

    <span class="pl-k">def</span> <span class="pl-en">request_seen</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>):
        <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">db</span>.<span class="pl-en">exist</span>(<span class="pl-s">"url"</span>, <span class="pl-s1">request</span>.<span class="pl-s1">url</span>):
            <span class="pl-k">return</span> <span class="pl-c1">True</span>
        <span class="pl-k">else</span>:
            <span class="pl-s1">visitedUrl</span> <span class="pl-c1">=</span> <span class="pl-v">VisitedURL</span>()
            <span class="pl-s1">visitedUrl</span>[<span class="pl-s">'url'</span>] <span class="pl-c1">=</span> <span class="pl-s1">request</span>.<span class="pl-s1">url</span>
            <span class="pl-s1">visitedUrl</span>[<span class="pl-s">'orgDupURL'</span>] <span class="pl-c1">=</span> <span class="pl-c1">None</span>
            <span class="pl-s1">self</span>.<span class="pl-s1">db</span>.<span class="pl-en">insert</span>(<span class="pl-s1">visitedUrl</span>)
            <span class="pl-k">return</span> <span class="pl-c1">False</span></pre></div>
<p dir="auto">However I found request_seen method gets called 3 times for one page crawling, here are stack trace for the 3 calls in order:</p>
<p dir="auto">------------------------------ 1<br>
request_seen [filter.py:18]<br>
enqueue_request [scheduler.py:51]<br>
schedule [engine.py:189]<br>
crawl [engine.py:183]<br>
_process_spidermw_output [scraper.py:183]<br>
 [defer.py:63]<br>
_oneWorkUnit [task.py:491]<br>
_tick [task.py:645]<br>
runUntilCurrent [base.py:825]<br>
mainLoop [base.py:1203]<br>
run [base.py:1194]<br>
start [crawler.py:251]<br>
 [start.py:15]<br>
run [pydevd.py:931]<br>
 [pydevd.py:1524]</p>
<p dir="auto">------------------------------ 2<br>
request_seen [filter.py:18]<br>
enqueue_request [scheduler.py:51]<br>
schedule [engine.py:189]<br>
crawl [engine.py:183]<br>
_handle_downloader_output [engine.py:155]<br>
_runCallbacks [defer.py:588]<br>
_startRunCallbacks [defer.py:501]<br>
callback [defer.py:393]<br>
_runCallbacks [defer.py:588]<br>
_startRunCallbacks [defer.py:501]<br>
callback [defer.py:393]<br>
connectionLost [http11.py:320]<br>
_bodyDataFinished_CONNECTED [_newclient.py:1161]<br>
dispatcher [_newclient.py:916]<br>
connectionLost [_newclient.py:537]<br>
_disconnectParser [_newclient.py:1513]<br>
_finishResponse_WAITING [_newclient.py:1487]<br>
dispatcher [_newclient.py:916]<br>
_finished [_newclient.py:440]<br>
dataReceived [http.py:1478]<br>
rawDataReceived [_newclient.py:299]<br>
dataReceived [basic.py:578]<br>
dataReceived [_newclient.py:385]<br>
dataReceived [_newclient.py:1533]<br>
dataReceived [endpoints.py:102]<br>
_dataReceived [tcp.py:215]<br>
doRead [tcp.py:209]<br>
_doReadOrWrite [posixbase.py:597]<br>
callWithContext [context.py:81]<br>
callWithContext [context.py:118]<br>
callWithContext [log.py:84]<br>
callWithLogger [log.py:101]<br>
doPoll [epollreactor.py:396]<br>
mainLoop [base.py:1206]<br>
run [base.py:1194]<br>
start [crawler.py:251]<br>
 [start.py:15]<br>
run [pydevd.py:931]<br>
 [pydevd.py:1524]</p>
<p dir="auto">------------------------------ 3<br>
request_seen [filter.py:18]<br>
enqueue_request [scheduler.py:51]<br>
schedule [engine.py:189]<br>
crawl [engine.py:183]<br>
_handle_downloader_output [engine.py:155]<br>
_runCallbacks [defer.py:588]<br>
_startRunCallbacks [defer.py:501]<br>
callback [defer.py:393]<br>
_runCallbacks [defer.py:588]<br>
_startRunCallbacks [defer.py:501]<br>
callback [defer.py:393]<br>
connectionLost [http11.py:320]<br>
_bodyDataFinished_CONNECTED [_newclient.py:1161]<br>
dispatcher [_newclient.py:916]<br>
connectionLost [_newclient.py:537]<br>
_disconnectParser [_newclient.py:1513]<br>
_finishResponse_WAITING [_newclient.py:1487]<br>
dispatcher [_newclient.py:916]<br>
_finished [_newclient.py:440]<br>
dataReceived [http.py:1478]<br>
rawDataReceived [_newclient.py:299]<br>
dataReceived [basic.py:578]<br>
dataReceived [_newclient.py:385]<br>
dataReceived [_newclient.py:1533]<br>
dataReceived [endpoints.py:102]<br>
dataReceived [policies.py:120]<br>
_flushReceiveBIO [tls.py:392]<br>
dataReceived [tls.py:422]<br>
_dataReceived [tcp.py:215]<br>
doRead [tcp.py:209]<br>
_doReadOrWrite [posixbase.py:597]<br>
callWithContext [context.py:81]<br>
callWithContext [context.py:118]<br>
callWithContext [log.py:84]<br>
callWithLogger [log.py:101]<br>
doPoll [epollreactor.py:396]<br>
mainLoop [base.py:1206]<br>
run [base.py:1194]<br>
start [crawler.py:251]<br>
 [start.py:15]<br>
run [pydevd.py:931]<br>
 [pydevd.py:1524]</p>
<p dir="auto">I thought that this method should be called only once for one page, why it gets called 3 times?<br>
this brought me a problem, seems the DB access gets a pb, the result are not reflected, in the end I have 3 record in DB.</p>
<p dir="auto">I will look into the pb, but I would still need to understand why this method gets 3 calls, did I make something wrong?</p>
<p dir="auto">thanks and best regards</p>