title: Trouble running a parser created using scrapy with selenium
body: <p>I've written a scraper in Python scrapy in combination with selenium to scrape some <code>titles</code> from a website. The <code>css selectors</code> defined within my scraper is flawless. I wish my scraper to keep on clicking on the next page and parse the information embedded in each page. It is doing fine for the first page but when it comes to play the role for selenium part the scraper keeps clicking on the same link over and over again.</p>
  
  <p>As this is my first time to work with selenium along with scrapy, I don't have any idea to move on successfully. Any fix will be highly appreciated.</p>
  
  <p>If I try like this then it works smoothly (there is nothing wrong with selectors):</p>
  
  <pre><code>class IncomeTaxSpider(scrapy.Spider):
      name = "taxspider"
  
      start_urls = [
          'https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx',
      ]
  
      def __init__(self):
          self.driver = webdriver.Chrome()
          self.wait = WebDriverWait(self.driver, 10)
  
      def parse(self,response):
          self.driver.get(response.url)
  
          while True:
              for elem in self.wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"h1.faqsno-heading"))):
                  name = elem.find_element_by_css_selector("div[id^='arrowex']").text
                  print(name)
  
              try:
                  self.wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[id$='_imgbtnNext']"))).click()
                  self.wait.until(EC.staleness_of(elem))
              except TimeoutException:break
  </code></pre>
  
  <p>But my intention is to make my script run this way:</p>
  
  <pre><code>import scrapy
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.common.exceptions import TimeoutException
  
  class IncomeTaxSpider(scrapy.Spider):
      name = "taxspider"
  
      start_urls = [
          'https://www.incometaxindia.gov.in/Pages/utilities/exempted-institutions.aspx',
      ]
  
      def __init__(self):
          self.driver = webdriver.Chrome()
          self.wait = WebDriverWait(self.driver, 10)
  
      def click_nextpage(self,link):
          self.driver.get(link)
          elem = self.wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[id^='arrowex']")))
  
          #It keeeps clicking on the same link over and over again
  
          self.wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input[id$='_imgbtnNext']"))).click()  
          self.wait.until(EC.staleness_of(elem))
  
  
      def parse(self,response):
          while True:
              for item in response.css("h1.faqsno-heading"):
                  name = item.css("div[id^='arrowex']::text").extract_first()
                  yield {"Name": name}
  
              try:
                  self.click_nextpage(response.url) #initiate the method to do the clicking
              except TimeoutException:break
  </code></pre>
  
  <p>These are the titles visible on that landing page (to let you know what I'm after):</p>
  
  <pre><code>INDIA INCLUSION FOUNDATION
  INDIAN WILDLIFE CONSERVATION TRUST
  VATSALYA URBAN AND RURAL DEVELOPMENT TRUST
  </code></pre>
  
  <p><strong><em>I'm not willing to get the data from that site, so any alternative approach other than what I've tried above is useless to me. My only intention is to have any solution related to the way I tried in my second approach.</em></strong></p>