title: Scrapy python following pagination
body: <p>I'm having problem following pagination of this website: <a href="http://gamesurf.tiscali.it/ps4/recensioni.html" rel="nofollow noreferrer">http://gamesurf.tiscali.it/ps4/recensioni.html</a></p>
  
  <p>My spider part of code :</p>
  
  <pre><code>for pag in response.css('li.square-nav'):
      next = pag.css('li.square-nav &gt; a &gt; span::text').extract_first()
      if next=='»':
          next_page_url = pag.css('a::attr(href)').extract_first()
          if next_page_url:
              next_page_url = response.urljoin(next_page_url)
              yield scrapy.Request(url=next_page_url, callback=self.parse)
  </code></pre>
  
  <p>If i run my spider on windows terminal it works on all pages of the website, but when i deploy to scrapinghub and run from the button in the dashboard, spider scrape only the first page of the website.
  Between log messages there is a warning: </p>
  
  <pre><code>[py.warnings] /app/__main__.egg/reccy/spiders/reccygsall.py:21: 
  UnicodeWarning: Unicode equal comparison failed to convert both arguments to 
  Unicode - interpreting them as being unequal.
  </code></pre>
  
  <p>Line 21 is this:</p>
  
  <pre><code>if next=='»':
  </code></pre>
  
  <p>I have checked problem is not caused by robot.txt.
  How can i fix this?
  Thanks</p>
  
  <p>Here the entire spider:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  
  
  class QuotesSpider(scrapy.Spider):
      name = 'reccygsall'
      allowed_domains = ['gamesurf.tiscali.it']
      start_urls = ['http://gamesurf.tiscali.it/ps4/recensioni.html']
  
  def parse(self, response):
      for quote in response.css("div.boxn1"):
          item = {
              'title':  quote.css('div.content.fulllayer &gt; h3 &gt; a::text').extract_first(),
              'text':  quote.css('div.content.fulllayer &gt; h3 &gt; a::attr(href)').extract_first(),
          }
          yield item
  
  
      for pag in response.css('li.square-nav'):
          next = pag.css('li.square-nav &gt; a &gt; span::text').extract_first()
          if next=='»':
              next_page_url = pag.css('a::attr(href)').extract_first()
              if next_page_url:
                  next_page_url = response.urljoin(next_page_url)
                  yield scrapy.Request(url=next_page_url, callback=self.parse)
  </code></pre>