title: Trying to crawl all links of a webpage with scrapy. But I cannot output the links on a page
body: <p>My first question here :)</p>
  
  <p>I was trying to crawl my schools website for all possible webpages there are. But I cannot get the links into a text file. I have the right permissions, so that  is not the problem.</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from scrapy.item import Item
  from scrapy.spider import BaseSpider
  
  class hsleidenSpider(CrawlSpider):
          name = "hsleiden1"
          allowed_domains = ["hsleiden.nl"]
          start_urls = ["http://hsleiden.nl"]
  
          # allow=() is used to match all links
          rules = [
          Rule(SgmlLinkExtractor(allow=()), follow=True),
          Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
          ]
  
          def parse_item(self, response):
                  x = HtmlXPathSelector(response)
  
                  filename = "hsleiden-output.txt"
                  open(filename, 'ab').write(response.url)
  </code></pre>
  
  <p>So I am only scanning on the hsleiden.nl page. And I would like to have the response.url into the textfile hsleiden-output.txt.</p>
  
  <p>Is there any way to do this right?</p>