title: How to avoid redirection of the webcrawler to the mobile edition?
body: <p>I subclassed a <code>CrawlSpider</code> and want to extract data from website.
  However, I always get redirected to the site's mobile version. I tried to change
  the <code>USER_AGENT</code> variable in scrapy's settings to <code>Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1</code>, but still get redirected.</p>
  
  <p><strong>Is there another way to signal another client and avoid redirection?</strong></p>