title: Scrapy Google Search
body: <p>I am trying to <strong>scrape</strong> google search and people also search links.</p>
  <p>Example when you go on google and you search for &quot;Christopher Nolan&quot;. Google also produces a &quot;people also search for&quot;  which includes images of people related to the our search. In this case our People also search produces (Christian bale, Emma Thomas, Zack Synder etc). I am interested in scraping this data.</p>
  <p>I am using <code>scrapy</code> framework and wrote a simple <strong>scraper</strong> but it returns an empty CSV data file. Below is code I have so far your help is appreciated. Hope everything is clear in what i want to achieve. I used Xpath helper (google app) to help find the Xpath.</p>
  <p>My code:</p>
  <pre><code># PyGSSpider(spidder folder)
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import Selector
  from PyGoogleSearch.items import PyGSItem
  import sys
  
  class PyGSSpider(CrawlSpider):
      name = &quot;google&quot;
      allowed_domains = [&quot;www.google.com&quot;]
      start_urls = [&quot;https://www.google.com/#q=christopher+nolan&quot;]
  
      #Extracts Christopher Nolan link     
      rules = [
          Rule(SgmlLinkExtractor(allow=(&quot;https://www.google.com/search?q=christpher+noaln&amp;oq=christpher+noaln&amp;aqs&quot;)), follow=True),
          Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
      ]
     
      #Parse function for extracting the people also search link.
      def parse_item(self,response):
          self.log('Hi, this is an item page! %s' % response.url)
          sel=Selector(response)
          item=PyGSItem()
          item['peoplealsosearchfor'] = sel.xpath('//div[@id=&quot;cnt&quot;]/@href').extract()
         
          return item
  </code></pre>
  <p>items.py:</p>
  <pre><code>from scrapy.item import Item, Field
  
  class PyGSItem(Item):
      peoplealsosearchfor = Field()
  </code></pre>