title: Google search next pages using selenium
body: <p>I'm trying to automate the clicking of the next page in google search, after I must have gone into the links in the 1st and 2nd search page.</p>
  
  <p>I've so far been able to do the following:</p>
  
  <ol>
  <li>Spin up the chrome browser</li>
  <li>Go to the Google webpage</li>
  <li>Type in the search words</li>
  <li>Click on the search icon</li>
  <li>Go into the links on the 1st and 2nd google page</li>
  </ol>
  
  <p>See my code below:</p>
  
  <pre><code>from time import sleep
  from selenium import webdriver
  from parsel import Selector
  from selenium.webdriver.common.keys import Keys
  
  #path to the chromedriver
  driver = webdriver.Chrome('/Users\my_path/chromedriver')
  
  driver.get('https://www.gooogle.com')
  
  #locate search form by name
  search_query = driver.find_element_by_name('q')
  
  #Input search words
  search_query.send_keys('X-Men')
  
  #Simulate return key
  search_query.send_keys(Keys.RETURN)
  
  Xmen_urls = driver.find_elements_by_class_name('iUh30')
  
  for page in range(0,3):
      Xmen_urls = [url.text for url in Xmen_urls]
  
      #loop to iterate through all links in the google search query
      for Xmen_url in Xmen_urls:
           driver.get(Xmen_url)
           sel = Selector(text = driver.page_source)
  
      #Go back to google search
      driver.get('https://www.gooogle.com') 
  
      #locate search form by name
      search_query = driver.find_element_by_name('q')
  
      #Input search words
      search_query.send_keys('X-Men')
  
      #Simulate return key
      search_query.send_keys(Keys.RETURN)
  
      #find next page icon in Google search
      Next_Google_page = driver.find_element_by_link_text("Next").click()
  
      page += 1
  </code></pre>
  
  <p>When I'm done collecting the links on the '2nd' search page, how do I tell the algorithm to start from the '2nd' search page and not the 1st search page (this will enable me go into >2 pages).</p>
  
  <p>I know it's a 'for loop' and syntax re-arranging I'm missing somewhere but my brain is frozen at this point.</p>
  
  <p>I saw this page: <a href="https://stackoverflow.com/questions/31040973/how-to-click-the-next-link-in-google-search-results">How to click the next link in google search results?</a> but it only helps if I'm not navigating away from the google search page</p>
  
  <p>What am I doing wrong?</p>