title: Scrapy won't call scraper function
body: <p>I have written a scraper using Scrapy and I have a weird yet simple problem. </p>
  
  <p>I login using u/p to scrape the data but sometimes the site redirects me to /login.asp page with redir= query parameter containing the url I was about to scrape. so I added re_login_if_needed() function and I call it as the first statement of the parse() callback. The idea is to check if response.url has a redirect URL so I can re-login to the site and continue scrapping with parse() as before.</p>
  
  <p>The problem is that somehow re_login_if_needed() function is never executed. Any DEBUG PRINT statement I put in there, is never printed out.</p>
  
  <p>How could that be?</p>
  
  <p>In by class I have:</p>
  
  <pre><code>import scrapy
  
  
  class QuotesSpider(scrapy.Spider):
      name = "quotes"
  
      def start_requests(self):
          urls = [
              'http://quotes.toscrape.com/page/1/',
              'http://quotes.toscrape.com/page/2/',
          ]
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
  
          self.re_login_if_needed(response)
  
          page = response.url.split("/")[-2]
          filename = 'quotes-%s.html' % page
          with open(filename, 'wb') as f:
              f.write(response.body)
          self.log('Saved file %s' % filename)
  
      def re_login_if_needed(self, response):
          # check if response.url contains redirect code, i.e: "/login.asp?redir="
          # and relogin ...
  </code></pre>