title: Selenium/python - Clicking all elements which are in a particular div
body: <p>I am learning to scrape using selenium and was trying one of the SAP pages. </p>
  
  <p>The link is : 
  <a href="https://help.sap.com/viewer/2e167338c1b24da9b2a94e68efd79c42/4.2.4/en-US/" rel="nofollow noreferrer">https://help.sap.com/viewer/2e167338c1b24da9b2a94e68efd79c42/4.2.4/en-US/</a></p>
  
  <p>What I intend to do is get all the leaf/last level urls for this page present on the table of contents panel. Unfortunately, the links are not in html and only if we click on each side button on the left panel, will it expand, also, each button click can have more nested buttons! I need to click all these and get the underlying data.</p>
  
  <p>I starting by identifying the xpath and tried clicking them, but it gives the following error : </p>
  
  <pre><code>import time
  driver = webdriver.Chrome("/Users/tarun/Downloads/chromedriver")
  
  driver.get("https://help.sap.com/viewer/2e167338c1b24da9b2a94e68efd79c42/4.2.4/en-US/")
  time.sleep(20) #tried this
  htmlSource = driver.page_source
  
  buttonsSidePanel = driver.find_elements_by_xpath('//*[@id="toc"]/treecontrol/div/div[*]/div[1]/div[1]')
  #also tried - //*[@id="toc"]/treecontrol/div/div[*]/div[1]/div[1]/span[1]
  for x in buttonsSidePanel:
      x.click()
      time.sleep(1)
  
  
  driver.close()
  </code></pre>
  
  <p>The error : <code>ElementNotVisibleException: Message: element not visible</code></p>
  
  <p>I have a few questions in this regard (apologies if some of them are naive)
   1. )The elements are visible in the html when I do inspect element! Am unable to figure out what is it that am doing wrong?</p>
  
  <ol start="2">
  <li>Also, how do I recursively click the underlying buttons if a button click shows more buttons, or is there a way by which I can click all buttons at once and then scrape the data? I looked at a few stackoverflow questions, but couldn't resolve my task</li>
  </ol>
  
  <p>3.)After clicking the button, now how do we scrape the newly generated content?</p>