<p dir="auto">I'm doing</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    def start_requests(self):
        while 1:
            words = read_a_list_wanna_crawl()
            ips = get_a_ip_list()
            if words.count() &gt; 0:
                for _, __ in zip(words, ips):
                    print('do while')
                    yield scrapy.Request(processed_url, self.html_parse, meta={'proxy': ip, ...})"><pre class="notranslate"><code class="notranslate">    def start_requests(self):
        while 1:
            words = read_a_list_wanna_crawl()
            ips = get_a_ip_list()
            if words.count() &gt; 0:
                for _, __ in zip(words, ips):
                    print('do while')
                    yield scrapy.Request(processed_url, self.html_parse, meta={'proxy': ip, ...})
</code></pre></div>
<p dir="auto">but when len(zip(words, ips)) == 1, scrapy print do while forever(Infinite loop) and never download any requests.<br>
but if len(zip(words, ips)) &gt; 1, scrapy will not go in to infinite loop.</p>
<p dir="auto">is this a bug? can scrapy handle this?</p>
<p dir="auto">ps: (another way to solve this)<br>
Is it able to create a fake scrapy.Request() that don't do request but do the callback to finish this kind control flow in scrapy?</p>
<p dir="auto">Hey, I'm wondering what would be the correct way of suspending the crawler in a scenario like this:</p>
<ul dir="auto">
<li>Crawl a table page per page</li>
<li>Stop once an item on the page has already been processed</li>
</ul>
<p dir="auto">Let's say I have 5 pages defined in the start_urls, these pages urls look something like:</p>
<ul dir="auto">
<li>target.com/list-0</li>
<li>target.com/list-50</li>
<li>target.com/list-100</li>
<li>target.com/list-150</li>
<li>target.com/list-200</li>
</ul>
<p dir="auto">The items on the pages change over time, the number means "0 to 49 most recent", "50 to 99 most recent", and so on.</p>
<p dir="auto">Assuming the last processed item is on page 100, should I check for duplicates in the item Pipeline and stop the crawler there? To avoid the older page's items from being processed and not wasting the target's resources.</p>