title: Scrapy gives URLError: <urlopen error timed out>
body: <p>So I have a scrapy program I am trying to get off the ground but I can't get my code to execute it always comes out with the error below. </p>
  
  <p>I can still visit the site using the <code>scrapy shell</code> command so I know the Url's and stuff all work.</p>
  
  <p>Here is my code</p>
  
  <pre><code>from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  from Malscraper.items import MalItem
  
  class MalSpider(CrawlSpider):
    name = 'Mal'
    allowed_domains = ['www.website.net']
    start_urls = ['http://www.website.net/stuff.php?']
    rules = [
      Rule(LinkExtractor(
          allow=['//*[@id="content"]/div[2]/div[2]/div/span/a[1]']),
          callback='parse_item',
          follow=True)
    ]
  
    def parse_item(self, response):
      mal_list = response.xpath('//*[@id="content"]/div[2]/table/tr/td[2]/')
  
      for mal in mal_list:
        item = MalItem()
        item['name'] = mal.xpath('a[1]/strong/text()').extract_first()
        item['link'] = mal.xpath('a[1]/@href').extract_first()
  
        yield item
  </code></pre>
  
  <p>Edit: Here is the trace.</p>
  
  <pre><code>Traceback (most recent call last):
    File "C:\Users\2015\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
      r = opener.open(req, timeout=timeout)
    File "C:\Users\2015\Anaconda\lib\urllib2.py", line 431, in open
      response = self._open(req, data)
    File "C:\Users\2015\Anaconda\lib\urllib2.py", line 449, in _open
      '_open', req)
    File "C:\Users\2015\Anaconda\lib\urllib2.py", line 409, in _call_chain
      result = func(*args)
    File "C:\Users\2015\Anaconda\lib\urllib2.py", line 1227, in http_open
      return self.do_open(httplib.HTTPConnection, req)
    File "C:\Users\2015\Anaconda\lib\urllib2.py", line 1197, in do_open
      raise URLError(err)
  URLError: &lt;urlopen error timed out&gt;
  </code></pre>
  
  <p>Edit2:</p>
  
  <p>So with the scrapy <code>shell command</code> I am able to manipulate my responses but I just noticed that the same exact error comes up again when visiting the site</p>
  
  <p>Edit3: </p>
  
  <p>I am now finding that the error shows up on EVERY website I use the <code>shell command</code> with, but I am able to manipulate the response still.</p>
  
  <p>Edit4: 
  So how do I verify I am atleast receiving a response from Scrapy when running the <code>crawl command</code>?
  Now I don't know if its my code that is the reason my logs turns up empty or the error ?</p>
  
  <p>Here is my settings.py</p>
  
  <pre><code>BOT_NAME = 'Malscraper'
  
  SPIDER_MODULES = ['Malscraper.spiders']
  NEWSPIDER_MODULE = 'Malscraper.spiders'
  FEED_URI = 'logs/%(name)s/%(time)s.csv'
  FEED_FORMAT = 'csv'
  </code></pre>