title: Scrapy Selenium: Why pagination is not working for scrapy-selenium?
body: <p>I am trying to get data using scrapy-selenium but there is some issue with the pagination. I have tried my level best to use different selectors and methods but nothing changes. It can only able to scrape the 1st page. I have also checked the other solutions but still, I am unable to make it work. Looking forward to experts' advice.</p>
  <p><em><strong>Source: <a href="https://www.gumtree.com/property-for-sale/london" rel="nofollow noreferrer">https://www.gumtree.com/property-for-sale/london</a></strong></em></p>
  <pre><code>import scrapy
  from urllib.parse import urljoin
  from scrapy_selenium import SeleniumRequest
  from selenium import webdriver
  from selenium.webdriver.chrome.options import Options
  from shutil import which
  from selenium.webdriver.common.by import By
  import time
  from selenium.webdriver.chrome.service import Service
  from webdriver_manager.chrome import ChromeDriverManager
  
  
  class Basic2Spider(scrapy.Spider):
      name = 'basic2'
      
      def start_requests(self):
          driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
          driver.set_window_size(1920, 1080)
          
          driver.get(&quot;https://www.gumtree.com/property-for-sale/london&quot;)
          time.sleep(2)
  
          property_xpath = driver.find_elements(By.XPATH, &quot;(//article[@class='listing-maxi']/a)[position()&gt;=2 and position()&lt;30]&quot;)
  
          for detail in property_xpath:
              href= detail.get_attribute('href')
              time.sleep(2)
              yield SeleniumRequest(
              url = href,
              )
              
          driver.quit()
          return super().start_requests()
  
      def parse(self, response):
          yield {
              'Title': response.xpath(&quot;//div[@class='css-w50tn5 e1pt9h6u11']/h1/text()&quot;).get(),
              'Price': response.xpath(&quot;//h3[@itemprop='price']/text()&quot;).get(),
              'Add Posted': response.xpath(&quot;//*[@id='content']/div[1]/div/main/div[5]/section/div[1]/dl[1]/dd/text()&quot;).get(),
              'Links': response.url
              }
  
          next_page = response.xpath(&quot;//li[@class='pagination-currentpage']/following-sibling::li[1]/a/text()&quot;).get()
          if next_page:
              abs_url = f'https://www.gumtree.com/property-for-sale/london/page{next_page}'
              yield SeleniumRequest(
                  url= abs_url,
                  wait_time=5,
                  callback=self.parse
          )
  </code></pre>