title: Scrapy ends after first result
body: <p>I've been looking around and cant find the answer I'm looking for. I got my crawler (scrapy) to return the results close to what i'm looking for. So What I'm trying to do now is get it to pull the multiple results from the page. Currently it pulls the first one and stops. If I take off the extract_first() then it pulls all the data and groups them. So looking for one of 2 answers that would work.</p>
  
  <p>1) continue crawling results and not ending
  2) ungroup each item onto a new line of results</p>
  
  <p>Here is my code:</p>
  
  <pre><code>    import scrapy
  from scrapy.selector import Selector
  from urlparse import urlparse
  from urlparse import urljoin
  from scrapy import Request
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.selector import HtmlXPathSelector
  #from scrappy.http import HtmlResponse
  
  class MySpider(CrawlSpider):
      name = "ziprecruiter"
  
      def start_requests(self):
          allowed_domains = ["https://www.ziprecruiter.com/"]     
          urls = [
              'https://www.ziprecruiter.com/candidate/search?search=operations+manager&amp;location=San+Francisco%2C+CA'
              ]
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
          for houses in response.xpath('/html/body'):
              yield {
  
                  'Job_title:' : houses.xpath('.//span[@class="just_job_title"]//text()[1]').extract_first(),
                  'Company:' : houses.xpath('.//a[@class="t_org_link name"]//text()[1]').extract_first(),
                  'Location:' : houses.xpath('.//a[@class="t_location_link location"]//text()[1]').extract_first(),
                  'FT/PT:' : houses.xpath('.//span[@class="data_item"]//text()[1]').extract_first(),
  
                  'Link' : houses.xpath('/html/body/main/div/section/div/div[2]/div/div[2]/div[1]/article[4]/div[1]/button[1]/text()').extract_first(),
                  'Link' : houses.xpath('.//a/@href[1]').extract_first(),
                  'pay' : houses.xpath('./section[@class="perks_item"]/span[@class="data_item"]//text()[1]').extract_first()
  
                  }
  </code></pre>
  
  <p>Thank you in advance!</p>
  
  <p>EDIT::
  After more research I redefined the container to crawl in and that gives me all the right answers. Now my question is how do I get each item on the page instead of only the first result... it just doesn't loop. Heres my code:</p>
  
  <pre><code>    import scrapy
  from scrapy.selector import Selector
  from urlparse import urlparse
  from urlparse import urljoin
  from scrapy import Request
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.selector import HtmlXPathSelector
  #from scrappy.http import HtmlResponse
  
  class MySpider(CrawlSpider):
      name = "ziprecruiter"
  
      def start_requests(self):
          allowed_domains = ["https://www.ziprecruiter.com/"]     
          urls = [
              'https://www.ziprecruiter.com/candidate/search?search=operations+manager&amp;location=San+Francisco%2C+CA'
              ]
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
          for houses in response.xpath('/html/body/main/div/section/div/div[2]/div/div[2]/div[1]/article[1]/div[2]'):
              yield {
  
                  'Job_title:' : houses.xpath('.//span[@class="just_job_title"]//text()').extract(),
                  'Company:' : houses.xpath('.//a[@class="t_org_link name"]//text()').extract(),
                  'Location:' : houses.xpath('.//a[@class="t_location_link location"]//text()').extract(),
                  'FT/PT:' : houses.xpath('.//span[@class="data_item"]//text()').extract(),
                  'Link' : houses.xpath('.//a/@href').extract(),
                  'pay' : houses.xpath('./section[@class="perks_item"]/span[@class="data_item"]//text()').extract()
  
                  }
  </code></pre>