title: How to Efficiently Scrape page results from a paginated list of links returned by a website's search functionality
body: <p>I am trying to scrape the pages of an online retailer through their search functionality. As an example, when a request is sent to the url www.example.com/search the page returns a paginated list of links to all of the website's products. Every product page is more or less the same and as such relatively simple to scrape. The problem is that there are roughly 40,000 product page links returned and I want to scrape all of them initially loading up a database and then scheduling a scraper to run daily to add any new products. I want to know what a good approach would be to efficiently scrape those 40,000 product pages using scrapy. Right now my code is along the lines of:</p>
  
  <pre><code>ExampleSpider(scrapy.Spider):
      next_page = 1
      last_page = 100
      start_urls = ['example.com/search?page={}'.format(next_page)]
  
      parse(self, response):
          yield scrapy.Request(response.url, callback=self.follow_product_links)
          yield scrapy.Request(response.url, callback=self.follow_pagination_links)
  
      follow_product_links(self,response):
          for href in selector_that_gets_all_the_product_links:
              yield response.follow(href, callback=self.parse_product)
  
      parse_product(self,response):
          # Scrape the product page and yield an item
          # details are not relevant to my problem
  
      follow_pagination_links(self,response):
          self.next_page += 1
          if self.next_page &lt; self.last_page:
              url = 'example.com/search?page={}'.format(self.next_page)
              scrapy.Request(url, callback=self.parse)
  </code></pre>
  
  <p>This isn't runnable code, but it should give you the idea of what I am trying to do. I know Scrapy is asynchronous which should certainly help, but are there any better techniques I could be using? As an aside I was hoping the website exposed a public API for querying their database, but sadly they do not. </p>